Bacalhau Project Report - Jan 25, 2023
Bacalhau Project Report - Jan 25, 2023
bprotocol, BOINC, node labels, stable diffusion endpoint, simplifying installation & centralized logging
A DevOps heavy week, with some feature work and DevRel goodness too.
bprotocol deployed & reverted ⏪
We deployed the new bprotocol code in v0.3.17, and at first it looked great!
But then pretty quickly it did not.
So, we had to revert it. It’s not clear yet why it worked in testing but pretty rapidly failed in production, but we are investigating now with a staging cluster we’re putting under load!
We were excited to roll out this major change because it includes:
Updated API with Python SDK
Networking support
bprotocol, of course, which upgrades our scheduling to be point-to-point rather than broadcast on GossipSub for huge performance gains
Hopefully we’ll get it out soon! 🤞
BOINC progress 👽🧓🏻
We’re integrating with BOINC so that we can run a BOINC client on the Bacalhau network. This has been made much simpler by the fact that jobs can now (in the now-reverted v0.3.17, at least!) make external networking calls (rate limited and with an allowlist), and we have allow-listed network traffic to two projects:
which generates a database of canonical forms (CF) of diagonal Latin squares (DLS) of order 10 having orthogonal diagonal Latin squares (ODLS), and
search for weak astrophysical signals from spinning neutron stars (often called pulsars) using data from the LIGO gravitational-wave detectors, the MeerKAT radio telescope, the Fermi gamma-ray satellite, as well as archival data from the Arecibo radio telescope.
We chose latinsquares because the jobs are fairly quick, which makes testing easier, and einsteinathome because the jobs are will generate more data for us to store in Filecoin via Estuary.
We now have this working!
And as soon as we manage to get a new v0.3.18 released & deployed, we’ll be actively running jobs from both of these projects (and maybe more) on our production network continuously 🎉
Stable Diffusion Endpoint 👀
Our excellent DevRel team is working on a demo of using Bacalhau and FVM to mint NFTs from text prompts with AI, like:
One of the key moving parts to this was the ability for the frontend to directly call into Bacalhau (for now, we’ll have the ability for Solidity code in FEVM to drive Bacalhau directly before too long…) but for this we needed a CORS enabled API endpoint which we decided to put on the dashboard API server.
Shouldn’t be too difficult, right? Just an API that calls into Bacalhau to run a job and wait for the result? Well, it turned out to be rather an adventure because the Bacalhau API changed between v0.3.16 and v0.3.17, including the way requests were signed, so we built the `stablediffusion` endpoint for the new code, waited until v0.3.17 was deployed, but then by the time we could test it, the whole network had gone down 🙀
Rapidly rolling back production to v0.3.16 (with necessary terraform patches) rebuilding the stablediffusion API endpoint against the previous API interface, rolling back some dashboard changes, and merging it all together was a game of git whackamole, but we got there, and the endpoint got deployed 🎉
And then the frontend hooked up the Bacalhau generated image to an FVM minted NFT! 🎉🎉🎉
Cheers! Great teamwork here across timezones (UK/Australia/West Coast)! Join the demo here:
Centralized logging 📝
We now centrally log from our production servers to Grafana Cloud’s Loki service. This means that figuring out what went wrong in production just got a bit easier.
Node labels 🏷️
You can now label your nodes when you create them and target jobs to specific nodes. This will be useful when for example only certain nodes have a certain data source on them and we one day support being able to ingress local data from a node (e.g. stream logs etc).
Simplified Setup ⚙️
Currently in the main branch (not in a release yet) is the ability to simply run `bacalhau serve` without pointing it to a separate IPFS server and have it run an in-process IPFS server. This will make it easier to deploy Bacalhau for Compute Providers and also in embedded IoT devices etc.
What’s next
Clean up mess left over from getting the stablediffusion endpoint finished in a rush
Get bprotocol deployed again!
Simplify deploying private Bacalhau clusters
Project Frog 🐸
Production ready Airflow DAG integration
OpenLineage integration
Questions/comments? Let us know!
Thanks for reading!
Your Humble Bacalhau Team