Bacalhau Project Report - Nov 21, 2022
Bacalhau Project Report - Nov 21, 2022
Improved testing & lots of planning for the future of the project.
Hi folks,
We spent the last week mainly improving testing and planning our next moves. Classic post-conference cleanup mode 😅 Also, we put out Swagger docs for our API, got uploads to Filecoin turned back on, and are onboarding new team members, yay!
Improved testing/CI 🍏🟢💚✅
This is important so that Bacalhau is an excellent place for developers to work. CI should be prompt and a reliable signal of whether a change is good. Writing tests should be as painless as possible.
We now have an improved Scenario-based testing framework. You can now define a test workload simply as a Go function.
We are now tracking test coverage.
We reverted the change to use WithDirectPeers in GossipSub, instead of using a manual reconnect loop. Surprise surprise, this made a whole lot of our test flakiness go away! We have an open issue to track further understanding and improvements of GossipSub.
Tons of other CI flakes and slowness issues have been resolved. CI is now stable! You can see in the CircleCI stats how it got bad around Nov 1, when we introduced WithDirectPeers, and now it’s OK again, also thanks to all the other work on stability:
We split out “unit” tests from “integration” tests (terms used loosely), so now
make testonly runs the unit-style fast tests locally. We are also working on making it so that the unit-style tests run 100% offline, and only depend on golang (not Docker!), which means they should also run offline so that Dave can finally run the tests while he’s hacking at 30,000 feet 🛫You can now enable “auto-merge” on a PR so that GitHub closes it automatically for you when the tests pass. This is actually a huge quality-of-life win, you can push a one-line change and then forget about it, then later notice on Slack that it got auto-merged. Nice! 👌
OpenAPI & SDKs 💁👌🎍😍
We now have Swagger docs for our API.
We’re also close to publishing auto-generated SDKs for the API in … maybe … every human programming language ever created?
Estuary is back on 📈⚡🔑🦡
Publishing to Estuary is back on by default due to a nice change to publish first to IPFS ourselves and then push to Estuary in the background. This means all results from Bacalhau jobs eventually make it into Filecoin for safe storage!
Planning! 📖👷🏼🗓️
We have done a metric shitload of planning for projects through the end of this year and next year. The full plan and roadmap are up here, but here are the highlights.
Deterministic WASM: Make it possible to run WASM directly on the Bacalhau network, rather than only Docker jobs.
Job Pipelines as DAGs: Enable users to chain Bacalhau jobs together. Eventually, with lineage and the promise of reproducible (re-runnable) pipelines. Be compatible with existing systems to drive adoption.
FIL+: Create an incentive "bridge" for Service Providers to become Compute Providers and participate in the Bacalhau network by rewarding them with extra FIL+ for "blessed" results of compute jobs. The Bacalhau team is to be the initial moderators. The integration will be with the Evergreen team initially.
Incentive Protocol & Simulator: Develop the verification protocol that will allow Bacalhau and related projects to support incentives in the future.
Examples & Docs: Make the docs and examples excellent and increase the number of examples we have to drive user adoption.
Resilience & Scale: Re-architect the communication primitives in the network to be vastly more scalable, and develop architectures that will enable high availability and truly production-ready deployments.
Invocation Spec: In collaboration with the IPVM community (Fission, Warpfork + other folks), we will rapidly standardize the invocation spec, aka job spec used across Bacalhau and 3+ other projects so that they become interoperable. This is also important so that the serialized invocation spec hashes to the same value for use in the Octostore project.
Networking: Enable limited network access for Bacalhau jobs in a way that (a) doesn’t impinge on determinism/reproducibility and (b) doesn’t make the Bacalhau network a giant free botnet for DDoS attacks.
Formal Verification: Formal verification of network protocols can automatically find exploits against them using a model checker. It exhaustively models the “most general adversary” and checks all possible network traffic flows. Explore and apply relevant projects as an alternative approach to hardening the protocol.
Game Theory: Mathematically model the Bacalhau network and understand how to develop DSIC incentive mechanisms for the protocol. The goal in this field is to design rules so that strategic behavior by participants leads to a desirable outcome.
Trusted Execution Environments: Leverage AMD TEE hardware to enable private execution of compute without trusting the host computer.
Octostore (Hash linked everything): The first big engineering project of the CoD working group came out of the first two hours of the joint session on the 2nd day of the CoD summit (platform track). Driven by Juan's suggestion to focus on making everything hash linked, the goal here is to create a global database of proposed results (with some confidence/reputation) of running certain content-addressed programs on certain content-addressed input data. For example, the system might report that the "University of Bristol" and "Caltech" both got result R1 when running program C1 on input data D1, even if one computation happened on the Golem network and the other on the Bacalhau network.
Encryption: Allow users to distribute private keys to trusted compute nodes and then distribute jobs on the network. Only compute nodes that have private keys can bid on and complete the work.
Phew, we’re going to be busy next year 😅 Check out the Notion doc for more details and roughly when we expect the sub-milestones of each project to land.
What's next? ⏭️
Improved API (e.g., Job object model in Submit endpoint) and publish OpenAPI docs.
Grafana publishes dashboards.
Improve docs around how to request resources.
Improved Ethereum example.
FIL+ design doc.
Websockets.
Dashboard.
Invocation Spec.
Networking design doc.
Further improved testing.
Questions/comments? Let us know!
Thanks for reading!
Your Humble Bacalhau Team
