It was an indeed thrilling experience to participate in ETH Global Paris hackathon, where we hacked on our project for two continuous days. We also got to meet with representatives of great projects, including Protocol Labs.

Our team consisted of two members :

• Dhruv Malik (working as a developer and web3 researcher at Extra labs)

• Charlie Durand (founder of Extra labs, expert in tokenomics, and product manager).

With the aim of building a simple developer tool, we wanted to allow anyone to generate maps from our algorithms by providing inputs such as map coordinates. All the while, the entire pipeline would be running on the Bacalhau framework.

At this point, we had been working together for a few months, building the foundations at Extra Labs, a company that focuses on building developer tools and frameworks to host large-scale, near-real-time 3D maps in the world using decentralized physical infrastructure network (DePIN) infrastructure.

Problem statement

Charlie has been working on apps that rely on maps for many years and has encountered challenges when trying to deploy 3D maps. She observed that existing solutions for 3D maps were lacking in several dimensions, namely level of detail, recency, and accessibility. In terms of maps, the ecosystem suffers from a steep monopoly on 2D maps held by Google Maps. Their massive investments since the early 2000s has brought a great deal of products to life, but this has also led to huge price increases. In 2018 alone, the price for map-related Google APIs increased by an average of 1,400%.

As 3D maps become the next frontier for solving pressing challenges, such as sustainable town planning, protecting heritage monuments, and mapping during natural disasters, we need to build a platform as a public good that helps various stakeholders make sense of various data sources (such as satellite images and sensors like LiDAR or natural photos) and apply data transformations to make high-quality maps available to all at a reasonable cost.

To accomplish this, we require an efficient, scalable, and inexpensive compute framework. Prior to this hackathon, we built a pipeline that connects different open-source algorithms for 3D surface reconstruction (which can be found here) and benchmarked some decentralized computing services. We discovered Compute-over-Data and Bacalhau, which met our needs very well. During ETHGlobal Paris, we explored a demonstration project that would enable us to deploy our pipeline on Bacalhau.

Project Gadus: CLI for running geospatial compute jobs

We named our project Gadus (after the Latin word for fish, similar to Bacalhau). The inspiration of our project was PLEX, developed by LabDAO for running BioML tools on private hosted instances of Bacalhau.

Similarly our CLI tools enable users to orchestrate 3D surface reconstruction operations on Bacalhau and to store them on decentralized storage, simplifying the various intermediate steps in the process: definition of the job specs, querying the requestor node to run the given geospatial transformation container algorithm, then reliably tracking the status, and finally storing the result on local instances/IPFS with a verifier.

We choose Lilypad as the framework on top of Bacalhau, to create a marketplace of the compute resources provider, which hosts the compute jobs of the clients. The resource providers are then compensated on-chain.

Along with that we also tried developing:

• The wallet client to allow users to build a data wallet (using DFNS)

• Role based access for clients (based on their on-chain authentication using Worldcoin).

• A way to define compute jobs and manage payments via an on-chain contract adapter.

Here is the workflow that we have envisioned (first release to be released in coming weeks) using Gadus:

1. User creates their wallet, which will create an external account for the user and drops some lil-ETH token (which is the payment token for lilypad testnet deployment).

2. For the data engineering part, we store the input data model of point cloud images together with the pipeline template information on the storage provider.

3. Next, the user must define the corresponding Docker image in which the job will be executed. To prevent sybil and cyber attacks, access to the compute infrastructure will be restricted to specific compute options.

4. These parameters are passed via CLI or hosted serverless application, which then relays them to the requester node. The user can then approve the result, which will be stored in their personal space on the decentralized storage system, using web3.storage.

We were able to demonstrate the minimal workflow showing the capabilities of the tool. Many thanks to the Bacalhau and Protocol Lab team members that were present during the hackathon and helped us troubleshoot issues. Various detailed examples written in the documentation were also significantly helpful for new entrants to start building end-to-end compute pipelines of various categories of data.

Challenges and the way forward

We appreciate the significant progress that Bacalhau has made since its inception, such as vertical scaling with support for various compute runtimes and inputs from different storage platforms. After the proposed v1 launch of Gadus, there will be progress in supporting real-time processing. However, Bacalhau has yet to address the significant potential for horizontal scaling, such as reducing latency, increasing bandwidth by running compute jobs in parallel across various compute nodes, and scaling resources based on the needs of client jobs.

In the end, we believe Bacalhau and the Compute-over-Data ecosystem represents a promising layer to support large-scale, near-real time 3D maps. We’ll be happy to keep working with the people involved to solve the existing challenges and benefit the DePIN ecosystem.

In coming weeks, we will launch the MVP of Gadus, along with technical blog series on system design for Compute-over-Data platforms, so follow our Medium, Linkedin and GitHub. We also just published a long article detailing the potential benefits of using a Compute-over-Data framework for geospatial data analysis.

Finally, I would like to express my appreciation for the help provided by the dev-rep team of Bacalhau/Lilypad (Ally and Luke), as well as the members of the Bacalhau Slack community. Kudos to this active community that assists users in overcoming production challenges related to Bacalhau.

If you have any questions or feedback, we’d love to hear about it, you can reach out on our discord here.

Cheers and keep BUILDL!

The winning projects at both events effectively leveraged the power of decentralization to build socially beneficial applications that solve real-world problems. Their creative use cases highlight the expanding capabilities of the Bacalhau ecosystem and decentralized technologies as a whole.

Augment Hackathon Winners

At the Augment Hackathon, CarpAI implemented an AI module for conversational chatbots on Bacalhau. Their project allows creating decentralized jobs to run AI inferencing on the Bacalhau network. Users can upload a JSON template prompt to IPFS, then call a dockerized python script through Bacalhau and Lilypad to run on a remote node. The results are uploaded to IPFS, closing the loop for a conversational agent. CarpAI’s work expanded access to decentralized AI through Bacalhau and Lilypad.

Brian provides a transaction builder that allows users to perform transactions via natural language prompts. Brian leverages AI to understand commands and generate the required contract calls. At the Augment Hackathon, the Brian team worked to replace their original closed-source AI model with an open source alternative based on HuggingFace. They used Bacalhau's decentralized compute for efficient training. Brian demonstrates how Bacalhau expands access to decentralized finance by simplifying complex processes through AI.

Avasoul.ai built an innovative platform for generating private AI models. They created an educational Shakespeare model and used Bacalhau to generate engaging video lessons. Their project allows users to create, customize, monetize, and trade specialized private models. Their educational use case provides a glimpse into Bacalhau's potential for decentralized knowledge sharing.

EasyCraft.ai developed an supply chain management system. They outlined a thoughtful whitepaper on the future of human-AI collaboration in manufacturing. It describes a transition from humans operating robots to more symbiotic relationships as AI capabilities advance. EasyCraft.ai envisions a global human-AI manufacturing network interlinked by smart contracts and marketplaces. While recognizing potential risks, they propose mitigation strategies involving monitoring, governance, and proof of humanity. Their innovative application demonstrates Bacalhau's utility in transforming supply chain transparency and efficiency.

ETH Global Paris Winners

At ETH Global Paris, GadusCLI created a convenient CLI tool for 3D surface reconstruction powered by Bacalhau. Their package makes decentralized compute easily accessible for processing geospatial data. It includes an integrated pipeline from raw lidar data to processed 3D visualizations. GadusCLI is working to build open 3D maps as a public good, avoiding centralized control. Their project highlights Bacalhau's scalability for data-intensive geospatial applications.

Deehr Market built an electronic health records management protocol giving patients ownership of their records while enabling secure data sharing for research. It uses Bacalhau for machine learning, allowing researchers to train models without accessing raw data. By compensating patients for data use, Deehr Market facilitates medical advances while preserving privacy. Their novel application demonstrates Bacalhau's utility for privacy-preserving data analysis.

The diverse projects from both Paris hackathons highlight Bacalhau's expanding utility across many domains. From AI assistants to supply chain tracking, medical research to credit scoring, Bacalhau provided the decentralized compute power to make these innovations possible. The hackathon creators showed remarkable creativity in building socially beneficial applications that solve real-world problems. Their pioneering work pushes the boundaries of what is possible with decentralization. Bacalhau is proud to support this new generation of builders. As the ecosystem continues maturing, we look forward to many more groundbreaking applications of decentralized technologies. The future looks bright as more purpose-driven projects leverage Bacalhau to achieve positive global impact.

Companies that generate large data volumes spread over a distributed architecture are realising that centralizing their data doesn’t work – it’s expensive, slow and introduces major orchestration challenges.

“At scale” also means scales of distance and time. Edge-deployed devices may have long network latencies back to a central store (scale of distance). Mars, for example, is three light-minutes away from Earth. Edge devices may also be out of contact regularly or mostly have access to only a self-contained local network shard (scale of time). Traditional centralized architectures break down at these scales.

Instead, Bacalhau is built on the understanding that distributed problems need a distributed solution. The next generation of distributed compute needs to run wherever the data is. That’s why we’re building Bacalhau to run on datacentre servers with quad-GPUs, on consumer laptops, on edge Raspberry Pis in remote locations, on satellites, and maybe one day on Mars.

Supporting such a breadth of devices used to be challening. Thankfully, using WebAssembly (WASM) answers this challenge – it is simple enough to be implemented with little overhead across a wide variety of CPUs whilst maintaining security and job isolation. Today, if you can compile your job to WebAssembly, you can run it on any node on the Bacalhau network.

⭐️ Star us on Github

Self-contained WASM is convenient but inefficent

Most language toolchains today will compile a program into a single large WASM blob containing all the code necessary to run the job. This is convenient – a user can submit such a blob to the Bacalhau network and any node can run it successfully.

The downside is that such self-contained blobs are big and inefficient. Most jobs use shared components. When self-contained blobs are transferred, much of what is sent is common code, especially when similar workloads are submitted regularly. If a job is being run on many hundreds of nodes simultaneously, the load to handle transferring the job to all of them can be overwhelming.

Sending the same code again and again adds unnecessary latency and bandwidth usage. In constrainted environments like a Raspberry Pi, a large blob might occupy most of the available RAM or disk space, meaning at most one job can be held and run at once. Making best use of bandwidth and memory is important in any setting but in constrainted environments it is crucial.

Modules make WASM more efficient

In higher-level languages, this is a problem solved by specifying a program’s code along with its common dependencies. The program can be run immediately if the user has all the dependencies or the user can download them if they are missing.

WebAssembly has a similar concept called “modules”. Each WASM program is a module and every module specifies which functions it provides and which it requires. Module imports and exports in WebAssembly are actually more advanced than most languages as memory blocks and data tables can be imported or exported as well as just functions.

Using modules, the user can specify a much smaller WASM blob which lists the required functions. The remote Bacalhau nodes can link in the required WASM modules if they have them or download them if they are missing. Much less data is transferred and multiple small jobs can be held in memory at once.

However, there is a missing peice of the puzzle: where do nodes download their modules from? Unlike in other languages where there are de facto centralised registries, there is no such thing (yet) for WASM modules.

We share WASM modules using distributed tech

So, is the Bacalhau team sponsoring the formation of a central WASM module repository? In short, no. Distributed problems need distributed solutions.

Central repositories are very convenient, but they don’t take advantage of data locality and so introduce the same bandwidth issues discussed earlier. They also place reliability of module delivery outside of the user’s control and provide mechanisms for modules to that are relied on to be yanked.

Instead, Bacalhau nodes use distributed data technology IPFS to retrieve WASM modules using a peer-to-peer protocol. This allows modules to be shared from the nearest nodes, whether they exist in the same data center or a few hops away on a mesh network. As this is independent of any central registry, it also works well for private deployments and ensures that commonly-used modules can be retained for as long as needed.

⭐️ Star us on Github

All the information needed to identify and retrieve modules can be embeded in the module itself. If a CID or URL is given as the import namespace, Bacalhau will automatically retrieve the module and resolve any new dependencies recursively. For the user, efficient modules are as easy to use as just submitting a well-formatted WASM blob.

(module  
  (import 
    "QmSyVCMvGauE35qfZVfuHEmDtbYvSom2v5r5c85SdF1LoB"
    "do_stuff"
    (func $do_stuff (type 3))) 
...)

You can generate imports like this by using language-specific flags. In Rust, using the wasm_import_module parameter generates the above code:

#[link(wasm_import_module="QmSyVCMvGauE35qfZVfuHEmDtbYvSom2v5r5c85SdF1LoBd")]
extern "C" {
    pub fn do_stuff() -> i32;
}

Intelligent scheduling of WASM jobs on Bacalhau

Understanding what WASM modules a job requires also allows Bacalhau to use data locality features to intelligently orchestrate the job. For example, Bacalhau can send WASM jobs to nodes that already have required modules, avoiding the need to download dependencies and speeding up job execution.

This also allows nodes to make use of proprietary modules. Service providers who make functionality available via a WASM module can advertise this module from their node, and Bacalhau will only schedule jobs that need the module to these provider-owned nodes.

Providers could, for example, make a machine-learning model available via WebAssembly and allow users to perform arbitrary interaction with the model in a way that is richer and quicker than a web API. Their model is still secure and private as the user’s execution is contained with the WebAssembly runtime. What’s more, load balancing and fault tolerance is provided automatically by the Bacalhau network.

A bright future for WebAssembly on Bacalhau

WebAssembly’s ability to be both secure and portable has already unlocked a lot of new compute environments for Bacalhau. We think peer-to-peer module provision and intelligent scheduling are two powerful features that make running WebAssmebly even simpler, quicker and easier than ever before. We’ve very excited to see what our partners are able to achieve with WebAssembly!

⭐️ Star us on Github

We are thrilled to announce the upcoming Compute over Data Summit (CoD Summit^3) on 9-10 May 2023 at The Westin Copley Place in Boston, Massachusetts. This two-day conference, organized by the Compute over Data working group, is set to be an insightful experience for both in-person and online attendees. With approximately 1000 participants expected, the CoD Summit^3 aims to foster conversations about increasing adoption, CoD directions, and collaboration by bringing together brilliant minds in the field. The summit aims to cover distributed datasets, edge centers, WASM technologies, reproducible builds, and consensus.

Exploring the Spectrum of Compute Over Data

Compute Over Data (CoD) is an innovative approach emphasizing decentralized compute, AI, and data processing. The CoD Summit^3 is dedicated to exploring the possibilities, challenges, and advancements across the spectrum of Compute Over Data, covering distributed datasets, edge, data centers, and technologies such as WASM, reproducible builds, and consensus.

Who's Behind the CoD Summit^3?

The CoD Summit^3 is curated by the Compute over Data working group. This open, community-driven organization aims to create a space for collaboration between teams building decentralized Compute over Data platforms. This group comprises professionals, researchers, and enthusiasts who share a passion for developing and promoting the field of Compute over Data.

What will the event entail?

The CoD Summit^3 will feature talks from field experts, including:

• David Aronchick, Head of Compute over Data at Protocol Labs

• Peter Wang, CEO at Anaconda

• Brooklyn Zelenka, Co-Founder and CTO at Fission

• Bernhard Borges from Fluence Labs

Notable attendees include:

• Juan Bennet, Founder, and CEO at Protocol Labs

• Molly Mackinlay, Engineering and Research Lead at Protocol Labs

• David Dias, Creator of libp2p and IPLD

These experts will delve into various themes and topics revolving around Compute over Data, such as platforms, data sources, supporting systems, and communities needed to make Compute over Data a reality. Their talks will be recorded and posted on the Bacalhau YouTube channel for online access.

Additionally, the summit will feature unconference tracks and a hackathon to build on the ideas generated during the event. Sample topics include:

• Pain points with compute & data processing today

• Key user experiences to address

• New technologies on the horizon (e.g., WASM, reproducible builds, zkSNARKS, etc.)

• Improving reliability, portability, and cost-effectiveness

• Integrating content addressing with data processing for reproducibility

• Pitfalls to avoid in Compute over Data

The CoD Summit^3 aims to make 2023 a breakthrough year for numerous products in the distributed computing space. By attending, you'll have the chance to engage in fascinating discussions, expand your knowledge, and contribute to the field's growth. Don't miss this opportunity to learn from and connect with some of the brightest minds in Compute over Data.

Join Us

If you want to deliver a talk at the CoD Summit^3, please submit your Call for Papers form by 19 March 2023.

Compute Over Encrypted Data

Parth Shukla from 180 Protocol took us through multiple designs for running decentralized compute over encrypted data in IPFS and Filecoin. This led to multiple discussions regarding encryption over data via Estuary, replication to Filecoin, and future tooling for Service Providers. Parth explained there are performance implications to encrypted compute, but based on their recent research with the Filecoin Foundation, benchmarking results show a relatively small performance penalty.

Decentralized Compute at the Edge with Blocz

Lee Norvall’s talk on Decentralized Compute at the Edge covered their ongoing work to help the health and safety of the City of Las Vegas. We learned how significant the cost savings can be for state and local governments when they generate petabytes of data at the edge.

What happens when you drive a truck into an edge compute location? Decentralization! Having multiple regional backups ensures the customer is not dependent on the availability of a single centralized location.

Aleph.IM and Using IPFS to Decentralize Backends

Hugo and Olivier took us through their designs to offer long-term guarantees for backend applications, similar to the durability of smart contracts.

The initial design is to support Lambda functions and micro-VMs invoked on demand. Their architecture had a dual network design: one for publishing messages and the other for compute execution.

For privacy concerns, there were discussions around when to trust the node operator vs when to trust the VM secret store for server-side VM secure computation (e.g. AMD Trusted Execution Environments).

SaaS Service Models via Smart Contracts on FVM

Taosheng from Ken Labs discussed how SaaS-style services can be built with Smart Contracts on FVM. Many projects are thinking through how they can find the right economic incentives for their communities.

Other highlights from the day include:

• Decentralized Reputation Scoring with Taosheng from Ken Labs

• Encrypted Over Open Compute and Storage Systems with Parth Shukla from 180 Protocol

• Reinventing Research Publishing via Open Data Formats with Christopher Hill from DeSci Labs

• City of Las Vegas IoT with Lee Norvall and Chris Ward Jones from Blocz.cloud

• Building a Decentralized Video Streaming Service with Yondon Fu from LivePeer 

• IPVM: Decentralized Functions as a Service with the team from Aleph.im

• History of Decentralization with Al Morris from Koii Networks

Remembering the world with content-addressable jobs

Radosław Tereszczuk of Golem presented his ideas about making job executions in CoD environments content-addressable. In this scheme, every job has an identifier generated by hashing all code and input data that are required to run the job. Two jobs doing the same work over the same data have the same hash, even if they are running on different machines that have never communicated.

David Aronchick of Protocol Labs also presented his paper on this topic which refers to a job and all of its inputs as a “curried function” – i.e. a function that can be invoked with no further inputs.

If the execution of jobs is also deterministic, then each job hash also uniquely identifies a set of outputs as well because running the same code over the same data deterministically should result in the same output. This means that it’s possible to see where data output by one job is used as input to another, and how all of the jobs link together in a call tree.

Having hashable jobs is a precursor to three valuable qualities:

1. The job can be stored as an artifact that is entirely reproducible: if someone wants to run the same job later to reproduce the result, they can be confident that the result they get will be the same as the first time it was run. David’s paper shows this to be valuable for decentralized science where verifying results in scientific papers is often challenging.

2. Outputs from previous executions of the same job can be reused: if there’s a mechanism for previous job outputs to be discovered, those previous results can be used instead of running the same work again. Applied at the global scale, this technique is effectively a "global cache” for all computation!

3. Economic incentivization of useful data: using the call tree of jobs and their outputs, it would be possible to see how many jobs ultimately refer to a piece of data (either directly or via the output of another job). This opens the door to directly rewarding the creation and storage of provably useful data.

Hashable jobs are also easy to version. Matthew Blumberg from CharityEngine mentioned that having versioned jobs was another useful quality of any execution system.

It’s clear from the discussion that linked jobs are something that lots of CoD providers are thinking about and starting to implement. Having a common realization of this across CoD platforms is in everyone’s best interest. The more execution that is hashable and discoverable, the more opportunity there is for reusing past outputs and incentivizing useful data. If everyone structures and executes jobs in the same way, they can benefit from work happening on other platforms, for free.

To realize the benefits, the CoD community needs two things:

1. Standardization on a job spec (or “invocation spec”)

2. A discovery mechanism that can relate invocation specs to cached outputs

A standard invocation spec from the IPVM project

Brooklyn Zelenka of Fission presented a talk on a new project to realize exactly this vision, called the InterPlanetary Virtual Machine (IPVM). The goal is to put deterministic, verifiable execution of WebAssembly into every IPFS node, to support automatic codecs and general data transformation straight from IPFS.

Included as part of the project is an invocation spec modeled as an InterPlanetary Linked Data (IPLD) document, which allows different content produced by different parties to be linked together. One document models the job code and its inputs and another model the outputs, linking back to the inputs and any side effects that were generated.

IPVM also is aiming at an "impure effect" concept that externalizes any non-deterministic operation as an effect that is executed outside of the execution environment, also referred to by Radoslaw as “services”. Effects don't have to be declared upfront; they can be decided while a job is running. This means that jobs can take different actions based on their input data.

Effects are signed by the node to prove that they were executed and can also rely on proactive capabilities given by the user. This system can also be used for handling privileged operations as part of a bigger job (e.g. only certain nodes may be able to send an email on behalf of a user so that the operation is modeled as an effect and only certain nodes can run it).

IPVM will also support payments via "state channels" that allow payment to be distributed dynamically amongst all the players that take part in executing a job.

A key question is how to make such a standardization effort successful and balance the forces between the collective agreement and individual innovation. Brooklyn explained that past experience with UCAN standards has shown that identifying the minimum needed for a useful standard and leaving holes and extension points is a valuable approach. She invited any parties interested in IPVM to sign up for their regular community calls.

Make past work discoverable with Octostore

Delegates also discussed the mechanism by which past job results could be archived and retrieved. Although the content produced by a job should be the same for every run, it’s not possible to guess the content hash of the resulting outputs without actually running the job, meaning we can’t just use IPFS for this. Instead, there needs to be a separate mapping from job spec inputs to outputs.

Luke Marsden of Bacalhau suggested that before we try and develop decentralized ways to do this, we should first prove the concept with a simple approach. He suggested building a centralized service that gives out API keys to interested parties, initially the members of the Compute over Data Working Group. David proposed “Octostore” as the name of this service.

The CoD Working Group community will be taking this work forward over the next few months – come and join us on Slack if you'd like to be involved!

Trusting results through reputation and incentives

One key assumption for reusing past results (or in fact, any results at all) is that they can be trusted. In a decentralized system without centralized enforcement of proper behaviour, users and job executors can lie with impunity. If a job executor can find a way to get paid without actually doing any work then the network will be flooded with bad executors and user trust in the network will disappear.

Almost every system of this type includes some idea of “auditing” – carrying out explicit checks that nodes in the network are behaving properly. In some cases, this is done by the user (manually), and in others, it’s built into the network.

Audits are strictly an overhead on top of useful work because they normally involve computing something that the user doesn’t really need, just to prove correctness. Successful networks maximize useful work (e.g. computing something for the first time) and minimize audits (e.g. computing something again to check it).

Two of the key levers in reducing the amount of auditing required are:

1. Reputation: having a network react to how a user or executor has behaved in the past. For example, an executor who has repeatedly fabricated results might no longer be selected for jobs.

2. Incentives: designing the economics of participating in the network to ensure that good behaviour is financially favourable and bad behaviour is not. For example, ideally, the amount of effort required to repeatedly lie would be more than just behaving properly.

There was a lot of discussion about how to design network protocols with these techniques and the different trade-offs between them.

A starting point for most reputation systems is to favour users with consistent past results so that a user with a better history is more commonly selected for jobs hence good behaviour is valuable in the long term. However, we were reminded that we want any reputation system to be resistant to recentralisation – so that big established players are not dominant and small players are able to enter the market and take part even with no reputation.

“Staking” was brought up as an alternative to reputation. In a staking system, people wanting to take part in the network (either as job submitters or as runners) have to put down a deposit that can be confiscated if they are found to be operating badly. This bypasses the need for reputation entirely as bad users have a direct financial implication for their behaviour. There was a suggestion by Levi Rybalov that staking is faster to bootstrap than running many audits and should improve the efficiency of a network.

Reputation and identity

Most reputation systems rely on having some persistent identity associated with executors, clients, and the users who run them. Otherwise, a user with a poor reputation can simply shuffle off their old identity and establish a new one with a clean slate. Establishing identities in a decentralized setting is a hard open problem being actively worked upon.

If identities can be established, then there are lots of techniques that can immediately help. David Aronchick highlighted that if we can establish people are unquestionably operating independently, then the number of positive results you need goes way down, as the chance they are colluding on forging results is minimal.

I pointed out that many real-world domains already have well-established reputation systems. For example, a professor is recognized as a trusted expert in their field, and if they publish a result a lot of weight is duly given to their opinion. If they fabricate results (or make a mistake), their reputation is at stake.

CoD platforms could look to simply piggyback on top of real identities as a (non-scalable) bootstrap mechanism. If a job executor is run by a recognized identity such as a university, their work can attract fewer audits because they have more to lose by acting improperly and the chance they are doing so is much lower.

The more decentralized version of reputation when everyone is anonymous instead relies on volume. The common analogy is with online reviews where thousands of reviews are generally a good signal over a few. These systems are still vulnerable to flooding attacks where bad users flood the system with false positives that seem independent (just like online reviews), and so these would need to be protected against.

Building configurable reputation and verification with Koii

Al Morris of Koii delivered a talk on how his team is solving these problems.

Al reminded us that different people have different risk profiles. In a Koii task, it is up to the task author to decide how much risk they are willing to take, only invite nodes to participate in the task that they trust, and craft verification that mitigates that risk. This externalizes the problem of trust and verification but introduces a new problem of how to aggregate node reputation across different types of tasks.

The other benefit of this approach is that each type of job requires a different type of auditing. For example, scraping web data might use a probabilistic analysis whereas computing something over a large data set in a deterministic way can be audited by comparing outputs.

Al also introduced the Compute, Attention, and Reputation Protocol (CARP) which Koii is using to incentivize correct behaviour in their network. CARP is a mechanism of stakes and reputation pools that can provide observable auditing for Compute over Data systems.

The way that CARP handles badly-behaved users is to employ “reputation slashing”, aggressively eroding the reputation when users are seen doing something bad.

Al also said that being able to turn the network into a self-defensive structure that will take down aggressive nodes (e.g. via denial-of-service attacks) would be valuable – essentially having the rest of the network act as an immune system against bad actors.

Reputation requires shared results and identities

There was general agreement that sharing results and identities were a primitive piece of infrastructure needed to build a wide array of different reputation systems. The CoD Working Group agreed to proceed with the Octostore proposal – to build a service to make compute results discoverable across networks. With that dataset in place, networks can develop reputation systems on top.

A number of different groups wanted to collaborate on this, so the CoD Working Group community will be taking this work forward over the next few months – come and join us on Slack if you'd like to be involved!

The COD working group was the force behind the summit — the group aims to create opportunities for collaboration for the members who are building decentralized compute over data.

The Bacalhau project took center stage with six presentations across the summit. Bacalhau is a decentralized public computation network that takes a job and moves it near where the data is stored, including across a decentralized server network that stores data and runs jobs inside it. Bacalhau runs the job near where data lives and eliminates data management for the user.

Let’s take a look at the presentations from day 1

David Aronchick, Head of Compute over Data at Protocol Labs, opened the summit with the talk titled “Revolutionizing the Big Data Age With Compute over Data.” He talked about how fast humanity produces a substantial amount of data today, and as it stands, it will reach 175 ZB by 2025!! (1 ZB=¹⁰⁹ TB, that’s A LOT OF data). Then David demonstrated how easy it is to produce 100TB of data in 100 days! Just look at these numbers.

Another astonishing fact is that it takes one day to move 100TB of data using 10Gbps connection! Next, David shared his list of companies that specialize in compute over data. Still, they target a different situation where a user has a centralized, very efficient processing cluster. Users continuously move their data into this processing cluster for computation, with very high costs and high latency. But decentralized data requires different solutions that are central to this summit’s discussions.

The YouTube link to David’s talk is here.

Brooklyn Zelenka, co-founder & CTO of Fission, gave a second talk on “Decentralized Authentication” — her work on UCAN, a User-Controlled Authorisation Network. Brooklyn and her team came up with solutions with only two handshakes (WoW!) compared with web2 OAuth, which has 12 handshakes! She talked about the specifics of her protocol and its capabilities in the browser, which, thanks to WASM and offline compute, is permissionless. Highly recommend watching Brooklyn talk on YouTube!

The third presenter was Dmitry Kurinskiy from Fluence Labs with his talk on “CoD Networks Discovery and Composability.”

Dmitry’s presentation lives here.

An overview of the new Protocol Labs project, “EVM-Compatible Filecoin Virtual Machine” (FVM), was given by Zak Ayesh from the FVM project.

Watch Zak’s presentation here.

David Aronchick announced the Bacalhau Beta release while presenting “Bacalhau — A Platform to Bring Compute to Data”. Bacalhau is a Distributed Computation system for decentralized compute workloads. The Bacalhau team made terrific progress and delivered the beta project in less than nine months! At the end of his presentation, David ran five live demos, with a cherry on top being a live demo on stable diffusion!

The video link is here.

Juan Benet, founder, and CEO of Protocol Labs gave a talk on “Impact of Compute over Data,” where he revealed the Filecoin Master plan. Firstly, to build the world’s largest decentralized storage network, and second, to onboard Humanity’s data. The last point is to bring compute to the data, datasets stored on Filecoin need computation.

Watch Juan’s talk here:)

Presentation on “Structured Data in Web3 — Rethinking Databases” by Sergii Mikhtoniuk, co-founder of Kamu.

The YouTube link is here!

Moving along, Lessons Learned from Incentive Models — Jon Starr, Levi Rybalov from Gridcoin.

The Youtube link is here for you!

“Computing in Large Data Spaces “by Matt Blumberg from Charity Engine is about the Crowdsourced cloud service where a percentage of their profit goes to charities. On top of that, they donate some of the computation resources to academic and medical research.

Here is a video link for this very impressive work.

“Building on Bacalhau for better Web3 data” by — Phillip LeBlanc, Co-founder and CTO of SpiceAI.

Watch his YouTube video.

“WeatherXM: Computing on Weather Data” — Stratos Theodorou, co-founder of WeatherXM.

Watch his presentation here.

“WASM: Our Portal to Portable Computing” — Peter Wang, CEO, and co-founder of Anaconda — highly recommend watching it here!

I hope you enjoyed many great talks on the CoD Summit day 1!

Stay tuned for more content from CoD Summit and CoD Working Group!

“It’s almost impossible to visit Lisbon and not be confronted with Bacalhau.”

The CoD Summit² was held recently in Lisbon, and we in the Bacalhau team had the privilege of attending.

Upon entering the restaurant, the server said, "Ahhhh -  you must be with the fish people," and I found myself in an unintended encounter with Bacalhau.

I was confused, so I searched around for clues as to who and where these fish people were, until I finally saw that everyone on the team was wearing these t-shirts:

After exchanging a knowing glance with the waiter - I found my table and had a great evening with my fellow Bacalhau fish people team members.

The deep dive

While in Lisbon, we announced a few new features that this blog post will attempt to dissect in some detail:

• WASM

• Airflow Support

• FIL+

WASM

You can now run WASM workloads inside Bacalhau! This means that you can now run workloads written in languages such as Rust, C, C++, Go, etc. inside Bacalhau without needing Docker.

WASM is designed as a compilation target for various languages. It can run workloads outside the context of a browser, and most importantly for Bacalhau, it can be made to guarantee determinism.

Bacalhau uses the wazero library to run WASM code in its dedicated WASM executor. It can run any program that is written with the WASI interface in mind.

You can watch the  video of the Bacalhau WASM executor in action here:

WASM: Part 1 - the workload

To run a WASM job in Bacalhau, just as with Docker, we will need some kind of workload - i.e., a job to actually run.

With Docker, this would be an “image” and a “command” to run in that image. For example, we could say, “please run “ffmpeg” with the following parameters to the resize command”

In that case, our image is “ffmpeg” and our command is the resize command.

With a WASM workload, we don't point at a Docker image; rather, we point at some pre-compiled WASM bytecode. We get this bytecode by compiling a Rust program to WASM.  For example, here is a short program in Rust that will print out the contents of a file:

First of all, we would compile the Rust code above and turn it into WASM bytecode, at each point we have a few options to run it:

• run Bacalhau directly against the local “cat.wasm” file we just compiled

• upload the “cat.wasm” file to IPFS, and then run Bacalhau against the IPFS CID

If we choose the first option, Bacalhau will automatically upload the cat.wasm file to IPFS and then run the job against that IPFS CID.

We can use the “bacalhau wasm run” command to do either of these:

bacalhau wasm run \
    {cid-of-wasm | <local.wasm>} \
    [--entry-point <string>] \
    [wasm-args ...] [flags]

The “--entry-point” flag is the function's name to call in the WASM workload, and the additional arguments are passed to the WASM workload as arguments.

Because you can compile code written in Rust, C, C++, Go, etc. to WASM, it opens up some interesting use cases.

For example, the following screenshot shows a WASM workload that is running a Rust program that does Seam Carving on an image:

WASM: part 2 - the data

When we run our WASM workload in Bacalhau, we can also pass in additional CLI flags to the WASM program. For example, if we wanted to run the “cat.wasm” program above against a file called “foo.txt”, we would do the following:

bacalhau wasm run cat.wasm /data/foot.txt

Our program would pick up “/data/foo.txt” as a CLI argument and then attempt to print out the contents of that file.

The problem is we have not yet told Bacalhau where to find the contents of the “foo.txt” file. We can do this by using the “-v” flag combined with an IPFS CID:

# first get a CID of some content
echo "hello world" > /tmp/myfile.txt
cid=$(ipfs add -q /tmp/myfile.txt)
# now run our bacalhau job
bacalhau wasm run -v $cid:/data/foo.txt cat.wasm /data/foo.txt

Now that we can mount data from IPFS inside a WASM job on Bacalhau, we can start to develop some useful WASM workloads and reuse them across different data CIDs.

WASM: part 3 - determinism

We have a verification strategy that depends on the deterministic results of a Bacalhau workload. This means that for the same invocation (i.e., the same code with the same data) - the result should be deterministic across multiple runs.

This is very hard to do with generic Docker workloads. There are various places in the system where entropy creeps in:

• Wall clocks – time is never in sync in distributed systems (and jobs won’t be guaranteed to run at identical times either!)

• Random numbers

• Remote network (and network failures)

• Multi-threading

• Values in garbage memory

• Thermal noise from the CPU

• Cosmic rays

Running WASM inside Bacalhau makes it possible to define how to handle these various sources of entropy. For example, we can configure the system clock to move forward one "tick" every time a command is executed, meaning the same time would be reported each time.

Bacalhau does not allow network access from inside a job, so we don't need to worry about that. Finally, we can configure WASM to use a deterministic random number generator, meaning that the same sequence of random numbers will be generated each time.

Getting our WASM executor to guarantee deterministic behavior unlocks the verification strategy for Bacalhau, and that means we have taken a huge leap toward a functioning decentralized compute network.

Airflow Support

Bacalhau now supports Airflow! This is exciting because it means that if you currently work with Airflow DAGs - you can now run any of your Airflow stages on Bacalhau, consume data on IPFS, and automatically publish the results of each stage to IPFS and Filecoin!

We have written an Airflow Provider that allows an Airflow DAG stage to call out to the Bacalhau network when it runs. Because Bacalhau supports Docker, it means that as long as your Airflow stage can be Dockerized, you can run it on Bacalhau.

You can watch a video of the Bacalhau Airflow support here:

Airflow Support: part 1 - the provider

Our Airflow provider is called “bacalhau-provider” and it lives here.

Using the provider is very simple, you import the following operators:

• BacalhauDockerRunJobOperator - used to run a Docker job

• BacalhauWasmRunJobOperator - used to run a WASM job

• BacalhauGetOperator - used to get the results of a job

And then chain them into a DAG as you would with any other Airflow operator.

We then use the xcoms system in Airflow to pass results from one stage to another.

The following example shows this in action:

Notice how the red arrow highlights where we are using xcoms to pass the results of one stage to another.

Airflow Support: Part 2 - the workload

When a single stage is being run by Airflow, it results in a Bacalhau job being invoked on the network. Because we have both the BacalhauDockerRunJobOperator and the BacalhauWasmRunJobOperator, you can choose if your stage is better suited to be run as a Docker job or a WASM job.

When a single Bacalhau job stage is reached, it will convert the Python spec into a JSON payload that will be submitted to the Bacalhau network in exchange for a JOB id. The operator will then block the pipeline on that Bacalhau job completion before moving on to the next stage.

Each stage can consume data from IPFS  as well as the data from the previous stage. We can also combine the WASM and Docker operators and make a single DAG that uses both types of workload:

The following image shows the output of each stage of our pipeline:

Airflow orchestrates Bacalhau to run both WASM and Docker workloads as part of a DAG running on a decentralized compute network!

FIL+

Bacalhau is integrating with evergreen, so we can start remunerating compute providers with FIL+ in exchange for running jobs.

The concept is simple, if a Bacalhau job operates on existing FIL+ data, chances are the output data that the job produces will also be useful to humanity and, therefore eligible for FIL+.

Because it's the compute providers that run the job that will also be custodians of the output data, it makes sense for them to be the ones to receive FIL+. This has a useful side effect of incentivizing them to run jobs in the first place. Consider this as a bridge towards an incentivized network before we officially release a token for incentivization.

The FIL+ integration with Bacalhau will use human moderators to ultimately decide if a job deserves FIL+ and, if so, how much FIL+ should be awarded to the compute provider that ran the job (and so has the storage deal for the results). The moderators will use a centralized dashboard that constantly scans the Bacalhau network for jobs that are eligible for FIL+.

You can watch our talk all about our FIL+ integration here:

FIL+: part 1 - existing FIL+ data

To help our moderation team, we need to quickly know if a CID used in a Bacalhau job has already been awarded DataCap. To do this, we will constantly scan the public FileCoin network and cache any CIDs we notice have had the “verified” bit set in their deal.

FIL+: part 2 - selection of jobs

When a user submits a new job to the network, compute providers will ask the following question: "is it worth it for me to run this job?"

To help answer that question, the compute provider will reach out to our centralized dashboard and ask if the input CIDs for the job are already blessed with FIL+. The compute node can still choose to run the job anyway, it just means it's a lot more likely to be awarded DataCap if the job operates on existing FIL+ data.

FIL+: part 3 - recording results CIDs

Once the job has been completed, our centralized dashboard will be notified of the results CIDs. This means that a moderator can log in, check the job's nature and look at the results the job produced.

FIL+: part 4 - moderation

Once the moderator has used the dashboard to decide whether the job is useful, they can choose to "verify" the job. This decision is recorded in our centralized dashboards database.

FIL+: part 5 - storage deals

Once the job has been moderated, the compute provider currently holding these results can initiate a storage deal with evergreen.

Evergreen is a notary and has the power to award DataCap to storage deals as it sees fit. In this case, when the compute provider negotiates a storage deal with evergreen, evergreen will reach out to our centralized dashboard and ask if the results are part of a "verified" job. If so, evergreen will award DataCap to the storage deal.

Conclusion

We had a great time in Lisbon at the CoD Summit², we ate a lot of Bacalhau (the fish), bemused at least one waiter, and had a lot of fun talking to people about Bacalhau (the decentralized compute network).

If you like the sound of what we are up to - be sure to check out our website and have a play with Bacalhau today!