Bacalhau

Share this post

Bacalhau Project Report - Jan 25, 2023

blog.bacalhau.org

Discover more from Bacalhau

Compute Over Data
Continue reading
Sign in
Project Report

Bacalhau Project Report - Jan 25, 2023

bprotocol, BOINC, node labels, stable diffusion endpoint, simplifying installation & centralized logging

Luke Marsden
Jan 25, 2023
2
Share this post

Bacalhau Project Report - Jan 25, 2023

blog.bacalhau.org
Share

A DevOps heavy week, with some feature work and DevRel goodness too.

bprotocol deployed & reverted ⏪

What if the world had a reset button | Art parody, Class ...

We deployed the new bprotocol code in v0.3.17, and at first it looked great!

But then pretty quickly it did not.

Uh Oh GIFs | Tenor

So, we had to revert it. It’s not clear yet why it worked in testing but pretty rapidly failed in production, but we are investigating now with a staging cluster we’re putting under load!

We were excited to roll out this major change because it includes:

  • Updated API with Python SDK

  • Networking support

  • bprotocol, of course, which upgrades our scheduling to be point-to-point rather than broadcast on GossipSub for huge performance gains

Hopefully we’ll get it out soon! 🤞

BOINC progress 👽🧓🏻

We’re integrating with BOINC so that we can run a BOINC client on the Bacalhau network. This has been made much simpler by the fact that jobs can now (in the now-reverted v0.3.17, at least!) make external networking calls (rate limited and with an allowlist), and we have allow-listed network traffic to two projects:

latinsquares:

which generates a database of canonical forms (CF) of diagonal Latin squares (DLS) of order 10 having orthogonal diagonal Latin squares (ODLS), and

einsteinathome:

search for weak astrophysical signals from spinning neutron stars (often called pulsars) using data from the LIGO gravitational-wave detectors, the MeerKAT radio telescope, the Fermi gamma-ray satellite, as well as archival data from the Arecibo radio telescope.

We chose latinsquares because the jobs are fairly quick, which makes testing easier, and einsteinathome because the jobs are will generate more data for us to store in Filecoin via Estuary.

We now have this working!

And as soon as we manage to get a new v0.3.18 released & deployed, we’ll be actively running jobs from both of these projects (and maybe more) on our production network continuously 🎉

Stable Diffusion Endpoint 👀

Our excellent DevRel team is working on a demo of using Bacalhau and FVM to mint NFTs from text prompts with AI, like:

One of the key moving parts to this was the ability for the frontend to directly call into Bacalhau (for now, we’ll have the ability for Solidity code in FEVM to drive Bacalhau directly before too long…) but for this we needed a CORS enabled API endpoint which we decided to put on the dashboard API server.

Shouldn’t be too difficult, right? Just an API that calls into Bacalhau to run a job and wait for the result? Well, it turned out to be rather an adventure because the Bacalhau API changed between v0.3.16 and v0.3.17, including the way requests were signed, so we built the `stablediffusion` endpoint for the new code, waited until v0.3.17 was deployed, but then by the time we could test it, the whole network had gone down 🙀

Rapidly rolling back production to v0.3.16 (with necessary terraform patches) rebuilding the stablediffusion API endpoint against the previous API interface, rolling back some dashboard changes, and merging it all together was a game of git whackamole, but we got there, and the endpoint got deployed 🎉

And then the frontend hooked up the Bacalhau generated image to an FVM minted NFT! 🎉🎉🎉

Cheers! Great teamwork here across timezones (UK/Australia/West Coast)! Join the demo here:

Twitter avatar for @BacalhauProject
The Bacalhau Project @BacalhauProject
@ETHGlobal S𝗽𝗮𝗰𝗲 🛸𝗪𝗮𝗿𝗽 Hackathon Workshop: Bridging Compute over data & Compute over state ⚒️ Building #AI Art NFTs on #FEVM with @BacalhauProject #OSS Stable Diffusion 👩‍💻 A workshop with @DeveloperAlly ⏰ Today, 1/25, 5:00pm ET 🖱️Join in: youtube.com/watch?v=qvSxmd…
Twitter avatar for @DeveloperAlly
Ally Haire|developerally.eth|alisonwonderland.lens @DeveloperAlly
Today, I decided in all my wisdom that this should be the first ever @bacalhauproject AI-generated #NFT minted on #FVM Hyperspace. I feel like I made the right choice 🦄 If you want to know more about how to make a project like this... tune in tomorrow! https://t.co/UsEOMELoV7 https://t.co/dVANzSLPO9
10:52 AM ∙ Jan 25, 2023

Centralized logging 📝

We now centrally log from our production servers to Grafana Cloud’s Loki service. This means that figuring out what went wrong in production just got a bit easier.

Node labels 🏷️

You can now label your nodes when you create them and target jobs to specific nodes. This will be useful when for example only certain nodes have a certain data source on them and we one day support being able to ingress local data from a node (e.g. stream logs etc).

Simplified Setup ⚙️

Currently in the main branch (not in a release yet) is the ability to simply run `bacalhau serve` without pointing it to a separate IPFS server and have it run an in-process IPFS server. This will make it easier to deploy Bacalhau for Compute Providers and also in embedded IoT devices etc.

What’s next

  • Clean up mess left over from getting the stablediffusion endpoint finished in a rush

  • Get bprotocol deployed again!

  • Simplify deploying private Bacalhau clusters

  • Project Frog 🐸

  • Production ready Airflow DAG integration

  • OpenLineage integration

Questions/comments? Let us know!

  • Our Website

  • Our Google Group

  • Our Slack

  • Our Repo

  • Our Docs

  • Our Build Instructions

  • Our Place To Complain about Missing Features/File an Issue (and in Our Slack)

Thanks for reading!

Your Humble Bacalhau Team

2
Share this post

Bacalhau Project Report - Jan 25, 2023

blog.bacalhau.org
Share
Comments
Top
New
Community

No posts

Ready for more?

© 2023 Expanso
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing