Bacalhau Project Report - Feb 3, 2023
Shipped bprotocol, networking, updated API, Python SDK docs, private IPFS networks, generic endpoint, dockerized.
We finally got our new release stable and deployed to production this week, and with it brought to life lots of new goodness!
Shipped bprotocol 🌟🛸🔌
We have stabilized the new bprotocol point-to-point scheduling protocol that runs over libp2p, meaning that we can now scale to gazillions of nodes without runaway O(N^2) GossipSub traffic!
Networking now live! 🌍🌐🚀
You can now build bacalhau in bacalhau yourself, because Networking is now deployed to the public network! Try it with the following one-liner:
bacalhau docker run --network=http --domain proxy.golang.org --domain sum.golang.org --domain index.golang.org --domain storage.googleapis.com -e GOBIN=/outputs golang:1.19 go install github.com/filecoin-project/bacalhau@v0.3.19
If you want to see how it works, a cheeky:
bacalhau docker run --network http --domain proxy.golang.org busybox env
Will allow you to see the `http_proxy` and `https_proxy` variables set in the environment:
These environment variables are a standard way to have application code use an HTTP(S) proxy, and we set up our own proxy as a Docker sidecar, and your workload gets to access the network via that proxy. This allows us to restrict network traffic to allow-listed domains (view or open a PR against our allowlist here if you want to add to it!)
Python SDK automagic keypair 🐍📖❤️
Our Python SDK now has docs! And it will automatically create a Bacalhau keypair if you don’t have one yet on your machine!
You can install it with:
pip install bacalhau-sdk
And then run a nice clean example:
from bacalhau_sdk.api import submit
from bacalhau_sdk.config import get_client_id
data = dict(
apiversion='V1beta1',
clientid=get_client_id(),
spec=dict(
engine="Docker",
verifier="Noop",
publisher="Estuary",
docker=dict(
image="ubuntu",
entrypoint=["sleep", "15"],
),
deal=dict(concurrency=3, confidence=0, min_bids=0),
inputs=[
dict(
storagesource="ipfs",
cid="QmWG3ZCXTbdMUh6GWq2Pb1n7MMNxPQFa9NMswdZXuVKFUX",
path="/datasets",
)
],
),
)
submit(data)
And voila, the Python SDK will go ahead and submit this job to the Bacalhau network and wait for the result 🎉
Dockerized Bacalhau Server 🐳
Based on user demand, you can now run a Bacalhau server in Docker, instead of running it directly on your node!
This doc also shows you how to run a private Bacalhau network in Docker, running separate IPFS nodes, also in Docker.
Easy local private networks 🚫🔐
We’ve also made it insanely simple to bootstrap a local private network including in-process IPFS. First, start a requester node:
bacalhau serve --private-internal-ipfs --peer none --node-type requester
This command will helpfully output the command you need to run on your compute nodes, and to connect your client! For example (this will change):
To connect another node to this private one, run the following command in your shell:
bacalhau serve --private-internal-ipfs --peer /ip4/212.82.90.222/tcp/1235/p2p/QmSyJ8VUd4YSPwZFJSJsHmmmmg7sd4BAc2yHY73nisJo86 --ipfs-swarm-addr /ip4/212.82.90.222/tcp/37469/p2p/QmXBtaTU7zjy4YvTPcRorDUdodtJcHxqDLhtteS7t2Po9k
To use this requester node from the client, run the following commands in your shell:
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/212.82.90.222/tcp/37469/p2p/QmXBtaTU7zjy4YvTPcRorDUdodtJcHxqDLhtteS7t2Po9k
export BACALHAU_API_HOST=0.0.0.0
export BACALHAU_API_PORT=1234
You literally couldn’t ask for easier than this!
A generic endpoint! 🔌⚡️🚀
We showed off stable diffusion last week, but want to run any job in Bacalhau from anywhere on the internet, including from a web frontend (with CORS enabled)?
The generic endpoint is born 🌟
You can now call http://dashboard.bacalhau.org:1000/api/v1/run with the POST body as a JSON serialized v1beta1 Spec as defined here.
For example:
curl -XPOST -d '{"Engine": "Docker", "Docker": {"Image": "ubuntu", "Entrypoint": ["echo", "hello"]}, "Deal": {"Concurrency": 1}, "Verifier": "Noop", "Publisher": "IPFS"}' 'http://dashboard.bacalhau.org:1000/api/v1/run'; echo
Will return:
{"cid": "QmeNhAA97qtdGHQtd1Qvgk13C6GHkn6aTCT8ih53JLN7vL"}
And that’s it. Couldn’t be simpler!
Questions/comments? Let us know!
Thanks for reading!
Your Humble Bacalhau Team