Use AWS S3 for data input and output 🪣🤩

We’ve just landed support for using data in Amazon AWS S3 buckets as input to and output from Bacalhau jobs! All the data that you store in S3 can now be accessed easily from Bacalhau, with an appropriately configured compute node. And once your job is complete, your results can also be written back to S3. Wow!

What’s more, our support is compatible with S3-like providers like Minio – so if you want to run S3-like data stores locally, you can now make this data available to Bacalhau jobs too!

Amplify persistence, web UI, image-resize & video-resize 🌠🔁

Our data automation platform Amplify has received some major upgrades! Amplify data is now persisted so that the current state of workflows survives restarts, making Amplify more able to handle interruptions. We now also have a web UI allowing operators to inspect Amplify jobs, demoed above by the legendary Dr Phil.

We’ve also been working hard on new pipelines for Amplify and are proud to release two new pipelines for rescaling images and videos. Now, whenever Amplify detects new image or video data, it will automatically rescale the media into a variety of resolutions and formats – making it easier to work with that data next time. Awesome!

Compute over private data with insulated jobs 💿🔐

We released our first insulated job features this week! This allows owners of private data to moderate access to their data and only allow certain jobs to run against it. This unlocks running Bacalhau networks between organisations, in federated environments and even as an easy way to access huge datasets made available as a public good.

Moderators can use a simple dashboard to see what jobs are wanting to make use of their data, and approve or reject them appropriately. Above, I demo the act of submitting a job against a moderated cluster and moderating the result. Check it out!

⭐️ Star us on Github

Log streaming for WASM jobs 🪵👩‍💻

In our previous project update we announced that you can now stream logs from Docker jobs whilst the job is still running – a great way to see how your long-running job is progressing.

We’ve expanded this support to now include WebAssembly jobs too! So no matter what kind of job you’re running, you can use the bacalhau logs command to see the current output from your job.

Waterlily deployed to mainnet (with automatic artist training pipeline) 🪷🏭⚡️

Our ethical AI art generator Waterlily.ai now is deployed to FVM mainnet, and with it comes a way for artists to submit their artwork to be a part of the Waterlily program.

Artists can now upload a portfolio of their existing work and Waterlily will automatically train an image generator model using their art (using Bacalhau of course!). Once the generator is trained, it is available to all Waterlily users and the artist can start earning revenue from any image generated from their model! It’s a super simple system that is bringing some much needed control back to the generative art space 😤

⭐️ Star us on Github

Questions/comments? Let us know!

• Our Website

• Our Google Group

• Our Slack

• Our Repo

• Our Docs

• Our Build Instructions

• Our Place To Complain about Missing Features/File an Issue (and in Our Slack)

Thanks for reading!

Your Humble Bacalhau Team

Log streaming for Docker 📝

As you can see in this very nice demo, you can now stream live logs from a Bacalhau job! For example, you can run:

$ bacalhau docker run -f myimage
# or 
$ bacalhau logs <jobid>
$ bacalhau logs -f <jobid>

So, you can stream logs directly while running `bacalhau docker run`, you can fetch logs so far from a job, or you can connect to the live log stream of a “detached” job.

This is cool because it means that you can run long-running jobs on Bacalhau, for example training an AI model, and see the progress that is streamed from that job while it’s running.

It works by extending the bprotocol, which is our custom libp2p protocol that runs between requester nodes and compute nodes, to support giving the requester node a custom multiaddress to connect to in order to fetch a log stream. The requester then muxes between libp2p and the clients’ websocket connections in order to get the log streams all the way back to the CLI. Nice!

Amplify deployed 🔊

The Amplify API is now deployed at http://amplify.bacalhau.org/api/v0!

As a reminder, Amplify is a service to augment data in IPFS and Filecoin CIDs (content addressible IDs) with automatic analysis of the data, and based on the content type (image, video, audio, tabular data, text, etc) will run conditional follow-on jobs to automatically transcode and interpret that data in interesting ways. This will be a value-add service for anyone using the IPFS and Filecoin networks - the idea is that any new data uploaded will automatically be analysed (subject to being able to scale this efficiently!) It will also be a useful service for folks to run in on-prem Bacalhau clusters, to automatically categorize and augment data flowing through private instances of the system.

The way Amplify is implemented is as a lightweight DAG system with a queue, where each step in the DAG is a Bacalhau job, and data flows between jobs automatically, with the ability to track lineage of all the jobs automatically, as shown in the first section of this video demo:

Retries 🔁

We now retry failed executions on other nodes in the network, including failures to execute the job, verification failures, failure to publish the results, and having the ask for bid rejected by the initially selected nodes.

This is a huge improvement for network performance from users’ perspectives. Previously if a job couldn’t be scheduled, it would fail, but now the requester node will try harder to get a job to execute successfully.

The retry strategy is pluggable, and multiple implementations can be chained. Currently we only have a single strategy that always retries as long as there are available nodes in the network to retry on. More strategies can be implemented in the future, specially after we have better breakdown if execution errors, and be able to differentiate between bad jobs and retriable node failures.

Private cluster improvements 🤫

We’ve made a few more improvements to make it super easy to set up a Bacalhau cluster inside your own network. We’ve simplified the setup for a private cluster from:

bacalhau serve --private-internal-ipfs --node-type compute,requester --peer none

To, wait for it:

bacalhau serve

We’ve also made sure that the “private internal IPFS” mode, which makes Bacalhau run an IPFS daemon in-process, without you needing to run one separately, now persists its identity and state, so that when you restart bacalhau or reboot your servers, you don’t lose results CIDs.

What’s more, bacalhau serve also helpfully prints out exactly the command you need to run on other nodes in your network to have them use the first node as a “meeting point” to bootstrap both the bprotocol and IPFS mesh networks over libp2p. Good UX FTW! 🚀

Planning is good and useful 🤔

We’ve done a ton of planning and getting organized on GitHub. Just look at our Starmaps! As we continue to develop in the open, you can see all the items we have planned in advance of the May 9 launch, and our progress towards them here:

We are also doing a similar planning exercise for an exciting new Web3-focused stream of work, watch this space!

Dashboard 📈

We now have a 🔥 new dashboard showing live stats on how many jobs are run on the Bacalhau network and also a breakdown of what kind of jobs they are.

What’s next ⏩

• Log streaming for WASM jobs as well

• Insulated Jobs demo

• Real jobs in Amplify

• Progress on Octostore

Questions/comments? Let us know!

• Our Website

• Our Google Group

• Our Slack

• Our Repo

• Our Docs

• Our Build Instructions

• Our Place To Complain about Missing Features/File an Issue (and in Our Slack)

Thanks for reading!

Your Humble Bacalhau Team