Bacalhau

Bacalhau v1.8.0 - Day 4: Seamless Result Storage with Managed Publishers with Expanso Cloud

Federico Trotta — Thu, 26 Jun 2025 15:30:59 GMT

This is part of the 5-days of Bacalhau 1.8 series! Make sure to go back to the start to catch all of them!

Day 1: Announcing Bacalhau v1.8.0: Intelligent Edge Computing Meets Enterprise Integration
Day 2: Rerun, Update, and Version Your Bacalhau Jobs
Day 3: How Bacalhau Boosts Daemon Job Reliability
Day 4: Seamless Result Storage with Managed Publishers with Expanso Cloud (this post)

Let’s be honest. When you’re focused on a complex data problem, the last thing you want to think about is infrastructure. Yet, for the longest time, a simple question has added friction to almost every distributed computing job: "Where do the results go?"

This question immediately triggers a cascade of others:

“How do I get credentials onto the compute nodes securely?”
“Am I accidentally exposing secrets?”
“How do I even find the output once the job is done?”

This dance of configuring storage, managing credentials, and tracking outputs is a tedious, error-prone distraction from the work that actually matters.

At Expanso, we believe you should focus on your code, not your cloud storage configuration. That’s why we’re thrilled to introduce a fundamental improvement to the Expanso Cloud experience: default managed publishers.

The Old Way: A Trail of Configuration and Credentials

Until now, when running jobs on Expanso Cloud, you could face common dilemmas involving:

Manual setup: Explicitly configuring S3 buckets, or any other storage solutions for every job or pipeline you are using.
Credential juggling: The risky situation of distributing AWS keys or other storage credentials to a fleet of compute nodes and clients.
Operational overhead: Managing the lifecycle, cost, and access policies of yet another piece of infrastructure.
Result scavenger hunts: A lack of a standardized approach to storage often left users digging through buckets to find their outputs.

This meant that even simple jobs required additional configuration and infrastructure planning, creating friction for users who just wanted to run their workloads and get results.

The New Way: Expanso Cloud Managed Publishers

With our latest update, Expanso Cloud now provides managed publishers by default. When you submit a job without explicitly specifying a publisher, Expanso Cloud handles the result storage automatically. No publisher configuration needed. It just works.

For example, consider the following simple data processing job: Notice what’s missing?

# Your job specification - no publisher configuration needed!
Name: data-processing
Type: batch
Tasks:
  - Name: data-processing
    Engine:
      Type: docker
      Params:
        Image: python:3.9
        Parameters:
          - python
          - -c
          - |
            import pandas as pd
            # Your data processing logic
            results = pd.DataFrame({'output': [1, 2, 3, 4, 5]})
            results.to_csv('/outputs/results.csv', index=False)
    ResultPaths:
      - Name: outputs
        Path: /outputs
# No Publisher section needed - Expanso Cloud handles it automatically!

When you submit this job to Expanso Cloud, here's what happens behind the scenes:

Automatic publisher assignment: Expanso Cloud automatically assigns its managed publisher to your job.
Secure upload: Compute nodes receive secure, time-limited upload URLs from the orchestrator, eliminating credential management concerns.
Organized storage: Results are automatically organized and stored in Expanso Cloud's managed storage infrastructure.
Seamless retrieval: You can access results through the Expanso Cloud UI or via the standard bacalhau job get command.

This architecture means our developers have eliminated the most common security pitfall: distributing long-term credentials to compute nodes. It’s security by design, not as an afterthought:

Zero credential distribution: Compute nodes never receive long-term storage credentials.
Time-limited access: Upload URLs have configurable expiration times.
Least privilege: Each compute node only receives upload permissions for its specific job results.
Audit trails: All storage operations are logged and traceable.

Getting Your Results: UI or CLI, Your Choice

Accessing your outputs is just as frictionless.

Access your job results directly through the Expanso Cloud dashboard:

Navigate to your completed job
View and download complete result archives
Share results with team members using secure, time-limited links

Results are retained for 30 days, giving you ample time to download and process your outputs.

You can also use the bacalhau job get command:

# Submit your job
bacalhau job run my-job.yaml

# Retrieve results when job completes
bacalhau job get my-job

The results are downloaded to your local machine just like any other Bacalhau job. However, you can now benefit from Expanso Cloud's managed infrastructure, handling all the storage complexity.

Don't Worry, Your Custom Setups Still Work

If you have an existing workflow with your own custom storage, nothing changes. The managed publisher only activates when no explicit publisher is specified, ensuring 100% backward compatibility. You have the freedom to choose, but the default is now effortless:

# Custom storage publisher still works
Tasks:
  - Name: main
    Publisher:
      Type: s3
      Params:
        Bucket: my-custom-bucket
        Key: my-results
    # ... rest of job spec

The managed publisher only activates when no explicit publisher is specified, ensuring complete backward compatibility with existing workflows.

A Better Workflow for Everyone

Why did we do all this? Well, the reality is that this workflow improves everyone’s jobs:

For data scientists: Focus on your analysis. Run your Python scripts, R analyses, or Jupyter notebooks and get results back automatically.
For dev teams: Prototype and test new pipelines without the friction of provisioning and securing storage for every experiment.
For production workloads: Leverage enterprise-grade, secure, and durable storage without having to manage it yourself.

Conclusion

Expanso Cloud's managed publishers eliminate the complexity of result storage while maintaining the security and reliability your production workloads demand. Whether you're running simple data processing jobs or complex ML pipelines, your results are now just a click away.

Ready to experience frictionless distributed computing? Sign up for Expanso Cloud and see how managed publishers can simplify your workflow today.

What's Next?

Ready to experience the enhanced daemon job capabilities? Upgrade to Bacalhau 1.8 and see the difference in your distributed workloads

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. Read more about the difference between open-source Bacalhau and commercially supported Bacalhau in the FAQ. If you want to use the pre-built binaries and receive commercial support, contact us or get your license on Expanso Cloud!

Bacalhau v1.8.0 - Day 3: How Bacalhau Boosts Daemon Job Reliability

Federico Trotta — Wed, 25 Jun 2025 15:31:11 GMT

This is part of the 5-days of Bacalhau 1.8 series! Make sure to go back to the start to catch all of them!

Day 1: Announcing Bacalhau v1.8.0: Intelligent Edge Computing Meets Enterprise Integration
Day 2: Rerun, Update, and Version Your Bacalhau Jobs
Day 3: How Bacalhau Boosts Daemon Job Reliability (this post)

In distributed computing, change is the only constant. Compute nodes join, they leave, they scale up and down. For long-running services, this dynamism can be a source of constant anxiety. Are your monitoring agents running everywhere they should be? Is your log processor catching every new service that spins up?

Daemon jobs in Bacalhau were built for this reality. They are designed to run continuously across all qualifying nodes in your cluster. Think of them as your most dependable employees, tirelessly working in the background. With the latest Bacalhau release, we’ve given these workhorses a significant upgrade, strengthening their ability to adapt to your ever-evolving infrastructure.

So, let's dive into what makes daemon jobs so critical and how we've made them even more resilient.

Why Daemon Jobs are Your Cluster's Backbone

Unlike a typical batch job that runs once and then exits, a daemon job is a persistent process. It’s designed to maintain a constant presence, automatically deploying to new nodes as they come online—thus, adapting to infrastructure changes. This makes them perfect for foundational tasks that need to be everywhere at once.

Here are the scenarios where they excel:

Edge data processing: Imagine continuously processing video feeds from hundreds of cameras or sensor data from a fleet of IoT devices. Daemon jobs ensure your processing logic is always running right where the data is generated.
Real-time log aggregation: As new microservices are deployed, daemon jobs automatically place a log forwarder alongside them, ensuring no message is lost.
Streaming data pipelines: They can maintain persistent connections to stream services Kafka or Kinesis, ensuring your data pipeline is never broken.
Cluster-wide monitoring: They are ideal for running health checks, collecting metrics, and triggering alerts across your entire fleet of machines.

Smarter, Faster, Stronger: The New Daemon Jobs Orchestration

So, what did we change? We rebuilt the core daemon job orchestration logic to be more intelligent and responsive. The Bacalhau orchestrator now has a much deeper awareness of the cluster's state. When a new compute node joins, the system doesn't just see a new machine. It instantly recognizes an opportunity to be deployed to any node that matches the job's constraints.

This enhanced awareness translates into concrete improvements:

Faster node detection: Our developers have drastically reduced the latency between a node joining your cluster and a daemon job being scheduled on it.”
Rock-solid constraint matching: We've hardened the process of verifying a node's capabilities against a job's requirements, eliminating deployment mismatches.
Confirmed deployments: The system now has better end-to-end tracking to confirm that a job has been successfully deployed, adding another layer of reliability.

Essentially, at Expanso we’ve tightened the feedback loop between your infrastructure and the job scheduler. The result? Your daemon jobs achieve their intended cluster-wide coverage with much stronger guarantees.

Let's see what this looks like. Imagine you want to run a log processor on every node that hosts a web service. Your job might look like this:

# A daemon job that reliably deploys to new web service nodes
Name: Enhanced-Log-Processor
Type: daemon
Constraints:
  - Key: service
    Operator: ==
    Values:
      - WebService

Tasks:
  - Name: LogProcessor
    Engine:
      Type: docker
      Params:
        Image: your-org/log-processor:latest
        Parameters:
          - --continuous-mode
          - --output-format=json

Previously, deploying this job to a new node that just came online worked, but now it's faster and more robust. When a new node with the service=WebService label joins the cluster, Bacalhau ensures your Enhanced-Log-Processor is deployed there almost immediately. No manual steps, no anxious waiting.

What This Means for Your Real-World Workloads

These aren't just abstract improvements; they have a direct impact on production environments:

Dynamic scaling, simplified: As you add more compute nodes to handle a traffic spike, your daemon jobs seamlessly expand with you. No need to manually deploy or configure anything again.
Resilient infrastructure: During cluster maintenance or upgrades, the system works harder to maintain full coverage for your daemon workloads. This ensures service continuity.
Reliable edge computing: For edge networks where nodes can be fleeting—frequently joining and leaving—this improved reliability is a game-changer that grants consistent service delivery.

Upgrade with Confidence

The best part? These enhancements are fully backward compatible. You don't need to change a single line in your existing job specifications. Just upgrade to the latest version of Bacalhau, and your daemon jobs will automatically benefit from the more resilient orchestration.

Conclusion

This improvement represents part of our ongoing commitment to making Bacalhau the most reliable platform for distributed computing workloads. The enhanced daemon job orchestration lays the groundwork for future features like more sophisticated deployment strategies and advanced cluster management capabilities.

Whether you're processing data at the edge, aggregating logs across a distributed system, or maintaining persistent services across your compute fleet, Bacalhau 1.8's enhanced daemon job capabilities provide the reliability and responsiveness your production workloads demand.

What's Next?

Ready to experience the enhanced daemon job capabilities? Upgrade to Bacalhau 1.8 and see the difference in your distributed workloads

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Bacalhau v1.8.0 - Day 2: Rerun, Update, and Version Your Bacalhau Jobs

Federico Trotta — Tue, 24 Jun 2025 15:31:50 GMT

This is part of the 5-days of Bacalhau 1.8 series! Make sure to go back to the start to catch all of them!

Day 1: Announcing Bacalhau v1.8.0: Intelligent Edge Computing Meets Enterprise Integration
Day 2: Rerun, Update, and Version Your Bacalhau Jobs (this post)

Remember j-f47ac10b-58cc-4372-a567-0e02b2c3d479? Probably not. And why should you? Until now, managing jobs has meant juggling cryptic, auto-generated IDs. You run a job, copy the ID, paste it to check the status, paste it again to get the logs, and hope you don't mix it up with the other dozen UUIDs in your terminal history.

We knew there had to be a better way.

That's why we're thrilled to introduce a fundamental overhaul to job management in Bacalhau. We're moving beyond throwaway IDs to a durable, name-based system that lets you rerun, update, and version your jobs with intuitive commands. This isn't just a quality-of-life update: it's a new paradigm for managing the entire lifecycle of your computational work.

Let’s see what’s new in Bacalhau 1.8.0. about that!

A Fundamental Shift: From Cryptic IDs to Meaningful Names

Let's be honest: the old way of working with jobs was functional, but could be frustrating. Every bacalhau job submit created a completely new job with a new ID. This led to a trail of disconnected job runs that were hard to track and even harder to manage.

The pain of UUIDs: Workflows required constant copying and pasting of non-human-readable IDs. It was tedious and error-prone.
No inherent history: If you ran the same analysis five times, you would get five unrelated jobs. Tracking the evolution of your work was a manual, out-of-band process.
A disconnected experience: Describing, logging, and fetching results all felt like one-off operations on ephemeral objects, rather than interactions with a persistent, evolving task.

With this update, we're introducing a more intuitive, name-based approach. You now define a name for your job directly in the job spec file. Bacalhau uses this name to anchor the job's identity, treating subsequent runs as new versions of the same job. This simple change transforms the entire workflow.

Here is how it was until now:

# Before: A trail of generated IDs
bacalhau job submit my-analysis.yaml  # --> j-f47ac10b...
bacalhau job describe j-f47ac10b...    # Must use the generated ID

And this is how we transformed it:


# After: Use your job's name everywhere
bacalhau job submit my-analysis.yaml
bacalhau job describe monthly-sales-report-generator  # So much better!

Best of all? This is fully backward compatible!. All your old scripts and workflows using job IDs will continue to work exactly as they did before. You can adopt the new name-based system at your own pace.

What's New: Your Job Management Superpowers

This update is built on three powerful new capabilities:

Job rerunning: A dedicated rerun command to restart completed or failed jobs, creating a new version while preserving history.
Enhanced job versioning: Automatic version tracking is now built in. All core commands (describe, logs, get) are now version-aware.
Advanced dry-run with diff preview: See exactly what will change before you update a job, like a git diff for your compute.

The `rerun` Command: Stop Re-Submitting, Start Rerunning

Ever had a job fail due to a transient network blip? Or needed to reprocess a dataset after a minor code tweak? Previously, you'd have to go back and submit the job all over again, creating a new, disconnected job ID.

The new rerun command makes this a first-class operation:

# Rerun the latest version of a job by name
bacalhau job rerun monthly-sales-report-generator

# Need to rerun an older, specific version? No problem.
bacalhau job rerun monthly-sales-report-generator --version 2

When you rerun a job, Bacalhau intelligently creates a new version under the same name. This preserves the original job's history and artifacts while letting you iterate. It also includes safety checks to prevent you from rerunning jobs that are already pending or in-flight.

Enhanced Job Versioning: Manage Your Job's Evolution

Your workflows aren't static. You tweak parameters, update Docker images, and refine your analysis over time. Bacalhau now embraces this evolution with built-in versioning.

Every time you submit a job with an existing name, a new version is created. You can then interact with any version of the job using a simple --version flag.

# View details of a specific job version
bacalhau job describe my-ml-training --version 3

# Get execution details from the very first run
bacalhau job executions my-data-pipeline --version 1

# View logs from an earlier version to debug a change
bacalhau job logs my-analysis --version 2

If you don't specify a version, Bacalhau defaults to the latest one. This makes your day-to-day work seamless while giving you the power to dive into the history whenever you need to.

Advanced Dry-Run: The “`git diff`" For Your Compute Jobs

Updating a critical production job can be nerve-wracking. Did you format the YAML correctly? Are the container arguments right?

Our enhanced --dry-run functionality gives you confidence by showing you a server-side "diff" of your proposed changes.

# Preview changes against the latest version before applying them
bacalhau job run --dry-run updated-job.yaml

Because the job name is in your YAML file, Bacalhau finds the latest existing version and generates a precise, color-coded diff showing you exactly what will be added, removed, or changed. This isn't a simple text comparison; it uses the same validation logic as a real job submission, so you can be sure that what you see is what you'll get. No more "pray and run."

Your New Workflow, Transformed

This update fundamentally changes the job management lifecycle from a series of disconnected, one-shot commands into a fluid, continuous workflow.

Before: Basic, Disconnected Operations

# Basic job operations
bacalhau job describe 
bacalhau job run --dry-run job.yaml  # Basic client-side validation only

After: Rich, Version-Aware Lifecycle Management

# Use names OR IDs, and interact with any version
bacalhau job describe my-job --version 2
bacalhau job rerun my-job --version 1
bacalhau job run --dry-run job.yaml    # Server-side diff preview!
bacalhau job executions my-job --version 3

Conclusion

These new capabilities are available now. Just update to the latest Bacalhau client to get started. We believe this new, name-based approach will make managing your computational workflows more intuitive, powerful, and reliable.

What's Next?

Ready to experience the enhanced daemon job capabilities? Upgrade to Bacalhau 1.8 and see the difference in your distributed workloads

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Announcing Bacalhau v1.8.0: Intelligent Edge Computing Meets Enterprise Integration

Federico Trotta — Mon, 23 Jun 2025 15:31:03 GMT

We're excited to announce Bacalhau v1.8.0: a groundbreaking release that transforms how you can approach distributed computing at the edge. This new release introduces:

Intelligent cost optimization through an advanced Splunk integration.
Enhanced daemon job orchestration for dynamic infrastructure.
Enterprise-grade improvements—docs, security, and more.
A completely reimagined job lifecycle management system.

In a nutshell, Bacalhau v1.8.0 delivers the integrations and capabilities that you need to reduce costs, improve operational efficiency, and unlock the full potential of your distributed infrastructure.

Let’s dive into it to learn more!

Slash Your Splunk Bill: Intelligent Logging at the Edge

Let's talk about the elephant in the room: logging costs. For too long, you've faced a terrible choice: ship all your raw data to Splunk and watch your budget evaporate, or sample your data and risk missing the one critical event you need.

So, what if there could be an alternative solution? Bacalhau’s new Splunk integration offers a third, much smarter path. Instead of moving mountains of data, Bacalhau lets you ship the computation to the data source. This fundamentally changes the cost equation.

With v1.8.0, Bacalhau's compute power can be integrated directly into Splunk so that you can:

Process data at the source: Deploy smart logging agents that filter, aggregate, and analyze data on the edge, sending only high-signal results to Splunk.
Fork your data stream: Send critical, processed alerts to Splunk for real-time monitoring while simultaneously archiving the full, compressed raw logs to cheap object storage.
Query live data without ingestion: Run queries directly on remote nodes to investigate issues in real-time, without moving the data or paying ingestion fees.
Replay history on demand: Need to analyze old logs? No problem. Use Bacalhau's compute power to pull data from your archive, process it, and analyze it in Splunk.

This isn't a minor tweak. It's a paradigm shift that can cut logging costs by 60-80% while giving you more analytical power. The best part? It all feels native. You use the Splunk dashboards and search queries you already know, while Bacalhau’s distributed compute engine works transparently in the background.

Enhanced Daemon Jobs

Bacalhau v1.8.0 strengthens daemon job reliability with enhanced orchestration. The goal is to ensure comprehensive coverage across dynamic infrastructure.

Our developers have beefed up the entire system for the reliability that production environments demand:

Node discovery is faster.
Constraint checking is more robust.
Deployment tracking gives you a clear view of your job's footprint. When a new node spins up in your dynamic infrastructure, you can be confident your daemon job will deploy to it automatically and reliably.

These enhancements are particularly valuable for the Splunk integration, where logging agents must reliably deploy to new infrastructure automatically.

Enterprise-Ready by Default

Rounding out the release are features designed to make Bacalhau seamless in an enterprise environment:

Managed result storage: With Expanso Cloud, you no longer need to configure your own publisher or manage storage credentials. Job results are automatically and securely stored in our managed infrastructure. Just run your job and fetch your results.
Documentation overhaul: We’ve rebuilt our docs from the ground up with task-based guides, architectural deep dives, and real-world examples for things like distributed log processing and edge ML.
Hardened foundation: We've continued to strengthen security, authentication (with SSO), and reliability to ensure Bacalhau is ready for your most demanding production workloads.

Transformative Job Lifecycle Management

Bacalhau v1.8.0 introduces a fundamental transformation in how you manage computational workflows. We've moved beyond cryptic, auto-generated job IDs to a powerful name-based system that supports versioning, updates, and intelligent rerunning.

Goodbye j-f47ac10b-58cc-4372-a567-0e02b2c3d479

If you've ever felt the pain of managing workflows using nothing but UUIDs, this update is for you. We've completely reimagined job lifecycle management to be intuitive, version-controlled, and built for humans.

Jobs are now identified by meaningful names you define in your spec:

# Before: Cryptic UUID management
bacalhau job describe j-f47ac10b-58cc-4372-a567-0e02b2c3d479

# After: Intuitive name-based operations
bacalhau job describe monthly-sales-report-generator

Advanced Versioning and Dry-Run

Every job submission with an existing name creates a new version. The enhanced --dry-run functionality provides server-side diff previews, showing exactly what will change before applying updates:

# Preview changes with detailed diff
bacalhau job run --dry-run updated-pipeline.yaml

# Work with specific versions
bacalhau job logs data-analysis --version 2
bacalhau job describe ml-model-training --version 1

Intelligent Job Rerunning

The new rerun command eliminates the need to resubmit jobs. This avoids you from creating new versions of existing jobs while preserving history:

# Rerun the latest version
bacalhau job rerun data-processing-pipeline

# Rerun a specific version
bacalhau job rerun ml-training-job --version 3

Also, this versioning system is fully backward compatible: existing workflows using job IDs continue to work while new capabilities are available for adoption at your own pace.

What’s next?

Bacalhau v1.8.0 is available now with full backward compatibility. Existing workflows continue to operate unchanged while new capabilities are available for immediate adoption:

For Splunk Users: The Expanso Splunk application is available through Splunkbase and can be deployed in minutes to start realizing immediate cost savings.
For Existing Users: Update to v1.8.0 to access enhanced daemon job orchestration, managed result storage, and the new job lifecycle management capabilities.
For New Users: Our completely rebuilt documentation provides clear pathways for getting started with distributed computing, whether for cost optimization, edge processing, or advanced analytics.

Subscribe now

Join Us on the Journey: 5 Days of Bacalhau

Stay tuned for our "5 Days of Bacalhau" series, in which we'll go deeper into these exciting new features:

Day 1: Announcing Bacalhau 1.8.0—Intelligent Edge Computing Meets Enterprise Integration (this post)
Day 2: Rerun, Update, and Version Your Bacalhau Jobs
Day 3: How Bacalhau Boosts Daemon Job Reliability
Day 4: Seamless Result Storage with Managed Publishers
Day 5: Distributed Logs Management with Bacalhau and Splunk

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

The Elephant in the Room: Why Is Your Data Bill So High?

Federico Trotta — Tue, 10 Jun 2025 16:34:24 GMT

Technology has been on a relentless march toward "more." More data, more powerful AI, more features, more connectivity. You attend conferences, read the industry reports, and the promise is always the same: a bigger, more feature-rich future. But this constant pursuit of "more" comes with a hidden, and rapidly growing, cost.

For most enterprises, data infrastructure has likely become a complex, bloated, and expensive beast. Your ingestion pipelines may be duplicative, and your analytics workloads over-provisioned. The sheer noise in the system drives costs sky-high, leaving you with little budget for actual innovation. It's the elephant in the room: as your capabilities grow, so does the waste.

A Different Message: You Should Be Paying Less

It's time to shift the conversation from "more" to "smarter." At Expanso, we believe it's time to stop accepting data waste as the cost of doing business.

Expanso's platform was born from a simple, radical idea: you should be able to slash your data infrastructure costs by 40–80% without ripping out your existing tools and without compromising on performance or security. The platform achieves this by fundamentally simplifying your ingestion pipelines, making intelligent use of your compute resources, and working with the data storage solutions you already have. You get the impact of immediate savings without the pain of replatforming or complex data migrations.

As our CEO, David Aronchick, often says, “No one wants to compromise on security or visibility. But everyone wants to stop paying for data waste. We built Expanso to give enterprises instant cost control—so they can reinvest in what matters.”

The future shouldn't be about paying more; it should be about doing more with what you have.

Read more in our press release on BusinessWire.

How About Something Practical?

Do you want to learn more about how Expanso, through the Bacalhau project, can help you save money, while providing computational power to your data? Read the following articles from our blog and website:

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

If you don’t have a node network available and would still like to try Bacalhau, you can use Expanso Cloud. You can also set up a cluster on your own (with setup guides for AWS, GCP, Azure, and more 🙂).

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Why Cloud-Centric Architectures Are Breaking Under Data Scale

Federico Trotta — Fri, 30 May 2025 15:30:28 GMT

If high costs, slow data, and compliance headaches sound like your problems, your current cloud setup might be the cause. So, let's explore a more efficient way to manage your data.

The cloud changed how IT works, offering great ways to grow and be flexible. But for those who are dealing with huge amounts of distributed data, the cloud isn't always the best answer. What often happens? Costs go up, data moves slowly, and following the compliance rules becomes a big problem.

This post explains why the old way of using the cloud for data isn't working well anymore. We'll also show you a newer, better solution: Bacalhau and a computing paradigm called Compute-Over-Data.

The Cloud Isn't Always Cheap: Costs Go Up, Benefits Go Down

When you first look at it, the cloud seems cheaper. You pay for what you use instead of buying your own hardware. But when you have data in many different places, that changes. Moving all your data to the cloud and working on it there can get very expensive. Here’s why cloud costs can get too high:

Fees for moving data: You pay to move data into the cloud because you need its powerful computers. If your data is in many places, these fees add up quickly.
Costly data storage: Cloud companies have different ways to store data. If you need to get to your data fast for quick checks, you often have to use expensive "hot" storage.
High costs for central computers: Working on huge amounts of data in one main place needs a lot of computer power. Companies often buy more than they need just to be safe, which costs more. You can try to fix this by paying only for what you use, but this is more like a quick fix than a real solution.

These high costs make companies ask if sending all their data to one place is still a good idea.

Slow Data: Why Your Information Can't Keep Up

Slow data is a problem for many companies. When you send all your data to a centralized cloud service, it takes time for information to go from where it starts, to the cloud, and back. This delay can make information old and useless by the time it’s ready.

As you get more and more data, this problem gets bigger. Trying to push tons of data through pipes to one main spot causes traffic jams. This can take days for very large amounts of data. This is a big problem for:

Edge computing: This is when you need to make decisions right away, where the data is generated. Sending data far away to a cloud and waiting for instructions is too slow for things that need to happen fast.
Quick data checks: Slow data makes it hard to get fast answers. People who study data often need information quickly to make good choices. If they wait too long for data, they might miss chances to act on time.
Real-time warnings: Delays can make you miss big problems in some cases. For example, systems that find fraud must spot bad actions right away. Hospitals need to warn staff immediately about big changes in a patient's health.

When data has to travel far because it's spread out, it slows down. Adding more computers or faster internet to cloud services doesn't fix the problem.

Rule Troubles: Following Data Laws in a Central Cloud

Following data rules is a big and tricky job for businesses. Laws like Europe's GDPR, California's CCPA, and healthcare's HIPAA are very strict. If you don't follow them, you can get big fines. Using a centralized cloud can make it harder to follow these rules because:

Data location: Many laws say that companies must keep people's data in their own country. Using a central cloud could mean you store data in the wrong country. This can break laws and lead to big problems and fines.
Moving data across borders: Sending data to computers in other countries is not simple. It needs careful legal and tech planning because the country it goes to must protect data just as well. This means more paperwork, costs, and risks, especially with private data like medical records.
Showing you follow rules: It's hard to prove you're following the rules when all your data is in one big place. Showing officials exactly where certain data is, who has used it, and that everything you do follows the rules can be a very big job.
Dealing with many rules: Companies that work worldwide have to follow international, national, and local data rules. Trying to use all these different, and sometimes clashing, rules with data in one main place can be hard. Because of this, companies might make things too strict and costly, or they might miss some rules to try and follow others.
Information leaks: Even if your main data is safe, systems in one place often gather logs, passwords, and details about computer jobs. If this extra information (called metadata) leaks, it can show private details about your data, what you're computing, or your computer systems. This can give attackers useful information for possible attacks.

Why Your Current Infrastructure Tools Struggle with Spread-Out Data

Even common infrastructure tools have trouble with today's spread-out data. As we talked about in our "Kubernetes vs Nomad vs Bacalhau" article, these tools are made to manage apps that are close to their data. Usually, this means in a single data center or cloud area with a good, fast network.

Also, how they are built still expects data to be easy to get to and brought to the apps. So, trying to make these tools work well with data that's very spread out often means using complicated fixes. This can include:

Setting up complex data pipelines to move parts of the data.
Trying to link groups of computers (clusters) across far distances.
Building your own special ways to get data ready and keep it in sync.

This way of doing things adds more complications, makes more work to keep things running, and can reduce the real benefits these tools are supposed to give.

The "Send Everything" Habit Costs a Lot

A big part of the problem is a common IT habit: “We send everything." Many systems, data setups, and processes are made to collect all data from everywhere and send it all to one main place–like a data lake or warehouse–before anyone even looks at it or figures out if it's actually useful.

This habit means you pay a lot to move, store, and work on "data noise." This "data noise" uses up good resources and causes many costs:

Costs to move data out of where it resides.
Costs for the main system to take in data.
Storage costs.
Costs to search and study the data.

This leads to high bills for internet use, storage space, and computer time. All for a lot of data you don't really need.

You keep doing this inefficient thing because most tools you have are built for it. You put everything in one place because you need the computer power to study the data.

Luckily, now we have a good, modern answer to this problem!

A Smarter Way: Compute-Over-Data with Bacalhau

The problems with working on data in one central place show that you need a new way. Instead of fighting against where your data is, you should work with it. This means:

Stop moving lots of data to computers.
Start using a Compute-Over-Data (CoD) plan.

The New Idea: Send Computer Power to Your Data, Not Data to Your Computers

Compute-Over-Data changes how you work with data. You don't send large, raw amounts of data to one place just because you don’t have enough hardware resources. Instead, Compute-Over-Data moves computer tasks to where your data already is.

Your data might be on edge devices (like sensors), local company computers, or spread across many data centers. The CoD way of computing moves much less data and solves the main problems with central models, like:

Cost.
Speed.
Rule-following.

When you work on data near where it starts, you can filter, combine, and change it locally. Then, you only send to the cloud the data that's useful for other tasks, and only if you need to!

Subscribe now

What is Bacalhau? Your Tool for a World of Spread-Out Data

Bacalhau is an open-source tool for computing that's spread out. It's built for the Compute-Over-Data idea. Here are some of its main features:

Simple to use: Bacalhau works as a client (user tool), an orchestrator (manager), and a compute node (worker computer) all in one small package. This makes it easy to set up and use in many different places. It also lets you do fast tests on your own computer that act like a whole cluster deployment, but without all the work of managing infrastructure.
Flexible design: It supports different ways to run code, like Docker containers and WebAssembly (WASM). It works with different storage systems (like S3, IPFS, and gets data from websites).
Handles different job types: It supports different kinds of jobs: "batch" jobs for tasks you do once, "ops" jobs for specific tasks on certain computers, "daemon" jobs for tasks that keep running in the background, and "service" jobs for programs that run for a long time.
Works reliably: Bacalhau is made for systems that are spread out. It can handle times when network connections are not steady. This is important for edge computing and data that is far apart.

How Bacalhau Solves Data Problems

By using Compute-Over-Data, Bacalhau fixes the headaches of working with data in one central place:

Cost savings: Working on data locally with Bacalhau cuts fees for moving data. By preparing and filtering data where it starts, you can stop paying to move and store "data noise." This saves a lot on network, storage, and central computer costs.
Faster data: Running jobs close to where data is made means less network travel and faster results. You get answers quicker, can make decisions in real time, and have faster apps.
Better data security and following rules: Bacalhau lets you work on private data within its safe and legal limits. This means less risk of data getting exposed when moving it. It also makes it easier to follow data rules like GDPR and data location laws.

Real Examples: Where Compute-Over-Data With Bacalhau Makes a Difference

Bacalhau's Compute-Over-Data way helps in many areas, such as:

Working with large amounts of logs: Study, filter, and combine logs on the computers or edge devices where they are made. This can cut the amount of data sent to central logging systems by over 90%, saving on network, intake, and storage costs.
Spread-out data warehouses: Run search queries directly on data stored in different local databases or cloud storage. This speeds up queries and helps follow data location rules.
Machine learning at the edge (edge ML): Train ML models or run them directly on edge devices. This allows for quick predictions, uses less internet by working on raw sensor data locally, and improves privacy by keeping private data at the edge.
Managing spread-out devices: Safely run commands, update software, and collect information from many spread-out devices or computers without needing direct access to each one or constant network connections.

Conclusion

Using one central cloud for everything causes problems when you have large amounts of data distributed over different locations. These problems include higher costs, slow data movement, and complex rule-following.

There's a different way, and it's called Compute-Over-Data (CoD). Bacalhau is a system designed for this CoD idea that sends computer tasks to where the data is, instead of moving large amounts of data.

Using Bacalhau reduces data movement, which lowers transfer and storage costs. It also makes it easier to follow data rules by letting you work on data within certain areas or security limits.

For companies working with distributed data, Bacalhau gives a way to fix the limits of old central ways of working with data.

Subscribe now

What's Next?

To start using Bacalhau, install Bacalhau and give it a try.

If you don’t have a group of computers (a network) ready and would still like to try Bacalhau, we suggest using Expanso Cloud. You can also set up your own group of computers (we have setup guides for AWS, GCP, Azure, and more 🙂).

Get Involved!

We'd like you to be part of Bacalhau. There are many ways to help, and we’d love to hear from you. You can find us here:

Commercial Support

Bacalhau is open-source software, but the official Bacalhau program files are made by Expanso with a careful security, checking, and signing process. You can read more about the difference between open-source Bacalhau and Bacalhau with commercial support in our FAQ. If you want to use our pre-built program files and get commercial support, please contact us or get your license on Expanso Cloud!

Cross-Border Data Processing With Privacy Compliance Through Expanso

Chris Chinchilla — Thu, 22 May 2025 15:46:19 GMT

Many organizations work with clients and infrastructure around the world and face significant challenges ensuring they follow privacy regulations as their application data flows across borders.

Data privacy regulations such as the General Data Protection Regulation (GDPR) in Europe and the California Consumer Privacy Act (CCPA) in California impose requirements on how applications can store, process, and transfer personal and sensitive information across borders.

The core challenge lies in maintaining data sovereignty and conforming to these rules while enabling cross-border analytics. For instance, when collecting personal data in the European Union, regulations often require that this data remain within EU borders. However, organizations still need to perform analytics on this data in other regions, creating a complex compliance challenge that requires careful architectural considerations.

Organizations often have to process large volumes of data efficiently. They need a distributed processing approach that maintains data locality and sovereignty while providing reliable job orchestration and monitoring that can scale in parallel based on demand.

This post looks at how you can use Bacalhau to handle distributed cross-border processing and anonymize data with Microsoft Presidio to help meet some of these requirements.

Bacalhau is an open-source distributed platform that enables you to run compute jobs where data is generated and stored.

A practical guide

This tutorial migrates data from the EU to the USA by creating a synthetic dataset, and anonymizing it before migration with Microsoft Presidio to analyze, extract, and anonymize sensitive data.

Subscribe now

Prerequisites

Before starting, ensure your system meets the following minimum requirements:

20 GB of free disk space
4 CPU cores
Docker and Docker Compose are installed

Set up multi-region deployment environment

The Docker Compose file below sets up a multi-region Bacalhau deployment with an orchestrator node that receives and schedules jobs, three compute nodes that execute the jobs, and one MinIO storage node per region, which in this case are the US and EU, using labels and constraints.

The storage nodes use MinIO, an S3-compatible object storage server, to store the data.

git clone https://github.com/bacalhau-project/bacalhau-network-setups
cd bacalhau-network-setups/docker-compose/multi-region
docker compose up -d

The Bacalhau solution implements a multi-regional data processing architecture that strictly adheres to data sovereignty requirements while enabling efficient cross-border analytics. The architecture consists of three main components:

Regional compute resources in each geographical region.
Distributed storage system spread across each region that can be queried individually or as a whole
An orchestration layer that coordinates jobs and requests across the system.

Install the Bacalhau CLI

To interact with the newly created Bacalhau deployment, install the Bacalhau CLI:

curl -sL 'https://get.bacalhau.org/install.sh' | bash

To verify that you are targeting the right Bacalhau deployment, run the command below.

bacalhau node list

You should see a list of 7 nodes: 1 orchestrator and 6 compute nodes. Something like the below:

 ID            TYPE       APPROVAL  STATUS     LABELS                                                              CPU   MEMORY   DISK     GPU

 compute-eu-1  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=eu              8.0   6.6 GB   861 GB   0

 compute-eu-2  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=eu              8.0   6.6 GB   861 GB   0

 compute-eu-3  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=eu              8.0   6.6 GB   861 GB   0

 compute-us-1  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=us              8.0   6.6 GB   861 GB   0

 compute-us-2  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=us              8.0   6.6 GB   861 GB   0

 compute-us-3  Compute    APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=us              8.0   6.6 GB   861 GB   0

 orchestrator  Requester  APPROVED  CONNECTED  Architecture=arm64, Operating-System=linux, region=global, type=orchestrator

Clone example job repository

In a separate directory, clone the examples repository and navigate to the data anonymization example folder.

git clone https://github.com/bacalhau-project/examples.git
cd examples/data-engineering/data-anonymization-with-microsoft-presidio/

You can find the job specifications used for the rest of this post in the jobs folder, and more details on the possible specification options in the Bacalhau documentation.

Generate fake sensitive data

The data-generator.yaml job consists of a bash script that runs in a Docker container and generates 30 files. These files simulate a memo full of personal data such as names, phone numbers, and addresses. The job generates the data in the node’s location in EU regions and then pushes those files to a MinIO bucket.

Submit the job to the compute nodes in Bacalhau cluster labeled with eu using the command below:

bacalhau job run -V Region=eu jobs/data-generator.yaml

Anonymize the data

The anonymize-job.yaml job runs a Python script on a Docker image that uses Microsoft Presidio to analyze, extract, and anonymize sensitive data in the EU-based MinIO bucket files.

Presidio is an open-source toolkit that uses NLP models to identify and anonymize sensitive information in structured and unstructured data formats. It can process different content types, including:

Unstructured text documents and communications.
Emails and business correspondence.
Internal memos and reports.
Images containing sensitive information.
Business documents and forms.

Presidio’s strength lies in recognizing multiple types of Personally Identifiable Information (PII). It can identify and sanitize sensitive elements such as names, addresses, identification numbers, and other personal information while maintaining the document’s structure and meaning. This capability is ideal for preparing data for cross-border transfers while maintaining compliance with data protection regulations.

The job outputs the anonymized files to a US-based MinIO bucket.

Submit the job to the compute nodes in Bacalhau cluster labeled with eu using the command below:

bacalhau job run -V Region=eu jobs/anonymize-job.yaml

This job takes a while to process, and you can check the job status using the job executions command.

Presidio anonymizes the data by replacing sensitive information with generic placeholders. For example, it replaces names with , IBAN codes with , and dates with .

Bacalhau input and output configuration

The InputSources and ResultPaths sections of the anonymize-job.yaml specification are the key components that enable the cross-border anonymous data processing.

InputSources:
  - Target: /inputs
    Source:
      Type: s3
      Params:
        Bucket: my-bucket
        Key: "confidential-memos/"
        Endpoint: "http://storage-local:9000"
        Region: "eu-central-1"
        Filter: ".*txt$"

This job uses MinIO and simulates an S3 bucket in the EU region.

This connects to the Publisher section of the data-generator.yaml job that defines where it writes the output of the job. In this case, a MinIO S3-compatible bucket in an EU region.

Publisher:
  Type: s3
  Params:
    Bucket: my-bucket
    Key: "confidential-memos/{nodeID}/"
    Endpoint: "http://storage-local:9000"
    Region: "eu-central-1"
    Encoding: plain

After anonymizing the data, the job writes the output to a different MinIO bucket in a US region, using a different MinIO endpoint.

ResultPaths:
      - Name: anonymized-memos
        Path: /anonymized-output
    Publisher:
      Type: "s3"
      Params:
        Bucket: "my-bucket"
        Key: "anonymized-memos/{date}/{time}/memos-{executionID}"
        Endpoint: "http://storage-us:9000"
        Region: "us-east-1"

The configuration uses a dynamic Key structure using {date}, {time}, and {executionID} to create a well-organized storage hierarchy that makes it easier to track different processing runs.

Cleanup

When you’ve finished with the example, you can clean up the environment with the following commands:

# Stop the stack
docker compose down -v

# Clean up volumes
docker volume prune

Summary

This post showed how you can use Bacalhau to maintain clear boundaries between sensitive and anonymized data by taking the following steps:

Accessing input data only from a source in an EU region
Processing using Presidio within the same region as the sensitive data
Only publishing anonymized results to a US region
US-based compute nodes can then perform analytics on the sanitized data

This process and setup ensure compliance with data sovereignty requirements while enabling efficient cross-region data processing and analytics.

Your Fast Track to Bacalhau: Local Development via Docker-in-Docker

Federico Trotta — Thu, 15 May 2025 14:47:29 GMT

Bacalhau is an open-source platform designed for “Compute-Over-Data”, allowing you to run jobs where your data resides. This avoids the costly process of moving massive amounts of data to other centralized locations. While easy to use, getting started and experimenting with a full Bacalhau deployment can involve:

Setting up cloud resources.
Configuring networks.
Managing multiple components.

This process may feel complicated if you just want to try new things on the fly.

This article explains how this process works and how, at Expanso, engineers have worked to make it more straightforward, accommodating it for situations where speed is key.

Let’s dive into it!

The Bacalhau Basics

Bacalhau is a framework that manages distributed computing networks that can operate across diverse infrastructures. It prioritizes keeping data in place, bringing computational power to data. It does so with the following nodes:

Orchestrator nodes: An orchestrator node is a component responsible for coordinating the execution of jobs across the network. Its primary role is to handle job scheduling, deciding which compute nodes are best suited to run a particular task, based on specified constraints.
Compute nodes: A compute node is a worker machine that executes the actual computational tasks assigned by an orchestrator. These nodes run the jobs, process the data, and generate results. Like orchestrator nodes, compute nodes are also instances of the single Bacalhau binary, but configured to operate in compute mode.
Hybrid Nodes: These nodes serve both roles at once. They are often used for local developments or small setups.

So, Bacalhau consists of an orchestrator that schedules jobs and multiple compute nodes that execute those jobs. These jobs run inside containers—via Docker or WASM—on the compute nodes. Setting this up traditionally can involve:

Provisioning Virtual Machines (VMs) or instances for the orchestrator and compute nodes.
Setting up S3-compatible storage for data input/output.
Installing Docker on all compute nodes.
Configuring networking and credentials for all components to communicate.

Let’s be honest: these are barriers you want to eliminate in the case of rapid experimentation. At Expanso, we know it, and our engineers got you covered!

The Local Solution: Docker Compose To The Rescue

The solution engineers wanted to solve responds to a question: ” How do we eliminate these barriers?”

Bacalhau works on containerized applications, but a container allows you to work only at a node level. However, engineers wanted to mimic the whole traditional setup so that it could be replicated on a single machine without:

Provisioning VMs.
Getting credentials and setting up S3 storage.
Installing Docker on multiple compute nodes.

The solution is straightforward: they used Docker Compose, a tool for defining and running multi-container applications. So, engineers at Expanso developed a Docker Compose image that defines all the services and their connections. This way, with docker-compose up, you bring the entire Bacalhau cluster to life on your machine in a few seconds.

The Inception Moment: Docker-in-Docker (DinD) Comes Into The Game

Here's where it gets interesting. In the Docker Compose setup, the compute nodes are themselves already running as Docker containers. So, here’s the challenge: how does a Docker container (the compute node) launch another Docker container (the actual job)?

This is where Docker-in-Docker (DinD) comes in. Docker provides special docker:dind images. When a container is run using a DinD image, it runs its own independent Docker daemon inside that container.

By basing your Bacalhau compute node containers on a DinD image, you allow them to pull other Docker images and run containers within their own isolated environment.

The workflow looks like this:

You run docker-compose up.
Docker Compose starts the orchestrator, MinIO—an open-source, S3-compatible object storage server—, and compute node containers (which are based on DinD images).
You submit a job to the local Bacalhau orchestrator.
The orchestrator assigns the job to one of the compute node containers.
The compute node container pulls the required Docker image for the job and runs the job inside a new container, nested within itself.

That’s it!

This mimics how a Bacalhau distributed deployment operates, but it is contained within your local Docker environment on your machine without:

Cloud accounts.
Infrastructure management.
Overheads.
Fear of messing things up. You can destroy your container and recreate another one.

Why This Matters: The Benefits

This DinD approach offers you advantages like:

Low entry barrier: No need for cloud accounts or complex infrastructure setup. Just Docker Desktop (or Docker Engine) and the compose YAML file.
Realistic simulation: Faithfully replicates the multi-component architecture and the container-based job execution of a real Bacalhau deployment.
Safe sandbox: You can experiment, break things, test configurations. If something goes wrong, you can simply docker-compose down and docker-compose up to start a fresh instance.
Rapid iteration: You can quickly test changes to Bacalhau configurations, job specifications, or even Bacalhau itself, if you're contributing to the project.
Offline capability: You can develop and test without an internet connection (once the initial Docker images are pulled).

Typical situations where the DinD approach with Bacalhau comes in handy are when:

You need to create a fast Proof Of Concept (POC) for a customer.
You have to try or add a new feature on the fly.
You just wanted to give Bacalhau a fast try.

How To Implement It

After discussing the theory, let’s see how to get started with DinD and Bacalhau.

Prerequisites

To replicate the following steps, your system must match the following prerequisites:

Docker engine: You need to have Docker installed and running on your system.
Docker Compose: You need Docker Compose itself:
- Docker Desktop: If you are using Docker Desktop, Docker Compose is typically included as part of the installation. To verify you have it installed, type docker compose version.
- Server/Manual install: If you installed Docker Engine manually, you might need to install Docker Compose separately.

Step 1: Clone The Repository

Clone the repository:

git clone

Note that the documentation for using it is in the docker-compose/ folder.

This is the structure of the cloned repository:

bacalhau-network-setups/
   ├──docker-compose/
   │	├──expanso-cloud/
   │	├──multi-region/
   │	├──single-region/
   │	
   └──README.MD

Step 2: Choose Your Setup

You can choose from different setups:

Single region.
Multi region
Expanso Cloud

Suppose you want a single region. From the main folder, you have to move to the single-region/ folder:

cd docker-compose/single-region

Very well. You are now ready to launch Docker Compose!

Step 3: Launch Docker Compose

From the single-region/ folder, launch Docker Compose:

docker compose up -d

After a few seconds, the process will be completed:

Good. You now have a working single-regional Bacalhau instance.

Step 4: Run Your Jobs

Connect to the client container:

docker compose exec client sh

Create a job list:

bacalhau job list

The expected result is:

CREATED  ID  JOB  TYPE  STATE

Now you can run your jobs

bacalhau job run ...

Good! You have launched your jobs with a Bacalhau instance all on your machine.

Step 5: Conclude And Cleanup

When all the jobs are done, after closing the instance, you can clean everything up with:

docker compose down -v --remove-orphans

This will:

Stop all containers.
Remove all containers.
Remove all volumes.
Remove any orphaned containers.
Remove all networks.

Here is the expected result:

Your instance is now clean, and you can start new jobs!

Conclusion

By leveraging the Docker-in-Docker technique, you can create self-contained Bacalhau environments locally. This approach:

Lowers the barrier to entry, providing an efficient environment for you to learn, develop, and test Bacalhau.
Allows you to test new features on the fly, without worrying about infrastructure management. This saves time and overhead.

If you want to know more about the architecture and the setup, read this README file.

Subscribe now

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Getting Started with Machine Learning on Bacalhau

Chris Chinchilla — Thu, 08 May 2025 15:39:31 GMT

Machine Learning requires vast amounts of resources, and distributing these resources across multiple devices and regions helps with cost, speed, and data sovereignty. Bacalhau is an open-source distributed orchestration framework designed to bring compute resources to the data where and when you want, drastically reducing latency and resource overhead.

Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data’s location, reducing latency and resource overhead.

How does Bacalhau work?

Bacalhau is a single self-contained binary that you can run on bare metal, in containers, or as WebAssembly. Bacalhau can function as a client, orchestrator, and compute node or all at once. Bacalhau integrates with S3, the local file systems, and other sources via HTTP endpoints, letting you pull data from various sources.

You can install Bacalhau on any UNIX-like operating system with one a one-line command:

curl -sL https://get.bacalhau.org/install.sh | bash

Or, for more flexibility, you can also install with Docker, depending on whether you want to run jobs in containers or not.

To start an orchestrator node that schedules and manages jobs, run:

bacalhau serve --orchestrator

To start a compute node that executes workloads, run:

bacalhau serve --compute

A node can have both types if you specify both flags.

Bacalhau’s architecture enables you to create compute networks that bridge traditional infrastructure boundaries. When you submit a job, Bacalhau determines which compute nodes are best positioned to process the data based on locality, availability, and defined constraints, without requiring manual data movement or constant connectivity.

Distributed machine learning with Bacalhau

This design allows for simple, flexible, and extensible execution, which is well-suited to distributing machine learning workloads.

For example, say you have a product recommendation model to train, and for regional and regulatory reasons, you want to train versions of it in the USA, Europe, and China.

To do this, use labels, which are key-value pairs that describe a node’s characteristics, capabilities, and properties. You can define these labels in a YAML file or as you start a node.

For example, to start a new orchestrator node that runs in the US. First, create a config file:

# config.yaml
labels:
  region: us

Then, pass it to the node:

bacalhau serve --orchestrator --config config.yaml

Or to pass the config as you start the node:

bacalhau serve --orchestrator -c Labels="region=us"

Bacalhau nodes run jobs either in Docker containers or as WASM payloads. The rest of this post uses Docker.

To submit a job to a node that matches that label, use the --constraints argument:

bacalhau docker run --constraints "region=us" data-processor

Or, more conveniently, you can declare jobs in a job definition file ml-job-us.yaml:

Type: batch
Count: 1
Constraints:
  - Key: region
    Operator: =
    Values:
    - us
Tasks:
  - Name: "data-processor"
    // rest of job definition

You can use Bacalhau to submit jobs to multiple machines in each region and distribute them amongst multiple servers based on the anticipated load for each region. This could be around Singles Day in China, Christmas in Europe, or Black Friday in the USA.

Retrieving processed data

Despite this global and regional distribution, you can aggregate it and use federated learning or analysis across subsets of the data. For example, to get an impression of trends at regional or global levels, but opting to run the learning in a specific region or not.

For basic job retrieval based on a region, you first find the details of jobs based on constraints. For example:

bacalhau job list --labels "region=us"

Then, download the results of a particular job:

bacalhau job get  --output-dir /destination/path

However, doing this manually for every job isn’t particularly productive, so instead, you could expand on the job definition file mentioned earlier to define what to do with the results upon completion.

Type: batch
Count: 1
Constraints:
  - Key: region
    Operator: =
    Values:
    - us
Tasks:
  - Name: "data-processor"
    // rest of job definition
			…
    Publisher:
      Type: s3
      Params:
        Bucket: us-results
        Key: us- results-folder
    ResultPaths:
      - Name: us-results
        Path: /outputs

This YAML configuration introduced a couple of other possibilities for job results. Instead of downloading the job results to a user’s computer, it saves them to an outputs directory where the job is running and then publishes those results to an S3 bucket. You can also fetch the results manually from the outputs directory with the job get command.

Supplying data to process

Machine learning needs data to process, which, again, you often want to keep separated for practical or regulatory reasons.

With Bacalhau, you can mount input data from the local file system, an S3 bucket, IPFS, or an HTTP endpoint.

Either as an --input command line argument:

bacalhau docker run --input : data-processor

Which consists of the URL to the storage location and where to mount it in the destination container.

Or, add the input details to the job definition file:

Type: batch
Count: 1
Constraints:
  - Key: region
    Operator: =
    Values:
    - us
Tasks:
  - Name: "data-processor"
    // rest of job definition
			…
			InputSources:
      - Alias: input
        Target: outputs
        Source:
          Type: /us-sales-data
          Params:
            key: value
    Publisher:
      Type: s3
      Params:
        Bucket: us-results
        Key: us-results-folder
    ResultPaths:
      - Name: us-results
        Path: /outputs

Summary

This post covered getting started with machine learning using Bacalhau, including some of the basic concepts for processing data in and out of Bacalhau securely and privately. To find out more, we recommend the more detailed installation guide, onboarding nodes to your network, and using jobs.

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Stop Paying for Data Noise: Optimize Your Pipeline with Compute-Over-Data

Federico Trotta — Thu, 01 May 2025 15:30:44 GMT

Are your cloud bills making you wince? Are you drowning in terabytes of logs? You're likely caught in a common, yet costly, data pipeline trap!

Many organizations default to a "ship everything" model. They collect data everywhere–servers, devices, applications–and funnel it all to a central location before analyzing it.

This seems logical, but it hides massive inefficiencies. Let’s discuss them.

The Cascade of Costs

This traditional approach triggers a cascade of expenses for every byte generated, especially the noisy, low-value ones:

Egress costs: Paying to move data out of its source environment.
Ingestion costs: Fees charged by your central platform just to receive the data.
Storage costs: Ongoing charges, often in expensive "hot" tiers for data that might never be accessed again.
Query costs: Even analyzing the data costs compute time and resources.

Industry analyses suggest that a staggering amount of this centrally stored data–potentially around 20%–is never used after storage. Yet, organizations pay the full price for this journey.

Beyond cash, this model adds operational drag:

High transfer times.
Engineering complexity.
Managing pipelines.
Increased security risks from moving sensitive data unnecessarily.

Why Does Raw Data Overwhelm Your Systems?

Sure, hot storage in the cloud offers fast access, but it's pricey. But even cheaper object storage isn't efficient if the data sitting there is redundant or useless without preprocessing.

The catch? You often need significant compute power to preprocess or analyze data before deciding if it's worth keeping or shipping–compute power that might not be available where the data originates.

The Real Reason We Ship Everything

So why stick with this inefficient pattern? Primarily because most existing tools are built for it. Log shippers, message queues, observability platforms, and data warehouses assume data must be centralized first. Coupled with the lack of easy-to-deploy compute at the edge, teams feel forced to ship everything, accepting the cost and complexity.

A Smarter Way: Compute-Over-Data

There's a better approach: flip the model and bring the compute to the data. Instead of shipping raw, noisy data first, process it at the source.

Imagine running logic directly where data is born:

Filter verbose logs down to just critical errors before they leave the server.
Aggregate raw metrics on an IoT gateway before sending summaries.
Enrich events with local context instantly.
Compress data intelligently based on its type.
Decide what data is valuable enough to move before incurring network and storage costs.

The Compute-Over-Data approach leads to tangible benefits:

Massive cost reduction: Stop paying egress, ingestion, and storage fees for useless data.
Improved signal visibility: Filter out the noise early to spot important events faster.
Enhanced security & compliance: Keep sensitive data local; only move aggregated or necessary subsets.

Subscribe now

How To Do So? Meet Bacalhau!

This isn't just theory. Bacalhau is an open-source framework designed specifically for Compute-Over-Data. It acts as an orchestration layer, letting you run compute jobs (packaged as Docker containers or WASM modules) directly where your data resides–be it data center servers, edge devices, or even workstations with GPUs.

Instead of pulling data to compute, Bacalhau sends compute to data. It helps you:

Process data at the edge: Execute filtering, aggregation, or analysis before data hits expensive network hops or ingestion endpoints.
Slash data volumes: Send only the valuable results, drastically cutting costs for downstream systems.
Handle diverse workloads: Supports various job types (batch, long-running services, ops, daemon jobs).
Operate reliably: Designed for distributed environments, handling intermittent connectivity gracefully (crucial for edge).

The Bottom Line

The "ship everything" era is proving unsustainable. Compute-Over-Data, powered by tools like Bacalhau, offers a more intelligent, secure, and cost-effective future for handling distributed data. Stop paying for noise and start optimizing your pipelines for value.

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

High-Scale Data Processing: Over Thousands of Devices With Azure Cosmos DB and Expanso

Federico Trotta — Tue, 29 Apr 2025 16:00:51 GMT

If you're dealing with data generated across thousands of devices or locations, you know the pain. Pipelining every raw byte back to a central data center for processing is slow, expensive, and fraught with regulatory hurdles.

Powerful databases exist to store this distributed data, but how do you process it before it gets there?

At Expanso, we recently explored this critical challenge in an Azure Cosmos DB TV episode featuring Mark Brown from Microsoft and our CEO, David Aronchick. The episode dives deep into a modern approach: Compute Over Data. Instead of moving mountains of raw data, why not process it right where it's created?

Let’s break down the main points!

The Distributed Data Dilemma

The world is generating data at an exponential rate, much of it unstructured and originating outside traditional data centers. Centralizing everything runs into several roadblocks:

Network bottlenecks: WANs aren't keeping up, and the speed of light imposes hard latency limits. Moving gigabytes or terabytes takes time and costs money.
Data quality and context loss: Raw data from edge devices is often poorly structured. Moving it immediately means losing context like location, local timestamps, device specifics, and more.
Regulatory compliance: GDPR, CCPA, and industry-specific rules restrict moving raw data, especially PII or sensitive operational details, across borders.
Delayed insights: Waiting for data to traverse networks and complex central ETL pipelines before analysis means delays in taking action, sometimes measured in minutes, hours, or even weeks.

Bacalhau: Bringing Compute to the Data

This is the gap Expanso, powered by the open-source Bacalhau project, is built to fill. Instead of moving data to compute, Bacalhau runs your processing jobs directly where the data resides.

This allows you to perform pre-processing steps locally like:

Schematization: Transform raw data streams into well-structured formats suitable for use in Cosmos DB.
Enrichment: Add metadata right at the source, preserving context.
Sanitization: Filter or modify sensitive information before it leaves the local environment, aiding compliance.
Aggregation: Reduce data volume by calculating summaries over time windows locally, sending only the essential information. This cuts network traffic and central processing costs.

The Synergy: Smarter Processing, Global Storage

The combination is powerful. Bacalhau acts as the edge/distributed processing layer, preparing and refining data. Optimized data, then, flows into the nearest Cosmos DB regional replica.

Key benefits of this approach include:

Cost savings: Reduced data transfer, lower central compute/storage needs.
Faster insights: Analyze data quicker by processing it closer to the source and landing analysis-ready data in Cosmos DB.
Enhanced security and compliance: Minimize raw data movement and sanitize sensitive information locally.
Increased resilience: Better handling of intermittent network connectivity at the edge.
Simplified operations: Declaratively manage distributed jobs without building complex custom orchestration.

The Takeaway

If you're building applications that span multiple locations, deal with IoT/edge devices, or face data gravity challenges, the traditional "move-then-process" model is holding you back. The "Compute Over Data" approach, enabled by Bacalhau working in concert with globally distributed databases like Azure Cosmos DB, offers a more efficient, cost-effective, and compliant path forward.

Want the full story and see the live demo? Watch the complete Azure Cosmos DB TV episode now! 👇

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

Cloud orchestration cost optimization

Chris Chinchilla — Fri, 25 Apr 2025 15:30:31 GMT

The move to the cloud promised to save users money and give them insights into their usage and costs.

However, the opposite happened. A 2025 report from AAG stated that around 82% of respondents found cloud spending challenging. A cloudzero report from 2024 states that more than 20% of respondents had no clear idea of their cloud costs, with reports for large users sometimes consisting of thousands of rows of hard-to-read usage data.

This is compounded by many companies and teams using more than one cloud provider for a hybrid cloud strategy to provide redundancy in case of outages or issues. Instead of taking advantage of the flexibility of cloud-native computing, many engineers still build as if they are using fixed on-premise servers, over-provisioning instances far beyond the capacity they need. All these factors mean that users overspend on idle and duplicated services and incur ingress and egress costs between providers.

Financial costs aside, complex cloud orchestration also increases energy impact, latency, and speed issues as multiple services pass bits and bytes back and forth across continents.

Most crucially, cloud applications write and access vast quantities of data, which are rarely stored in the same place where they are accessed and processed, causing yet more latency and cost.

It’s hard to get accurate cost comparisons between cloud providers, but a rough comparison is the following for on-demand rates:

AWS t4g.xlarge, $0.1344 per hour
Azure B4ms, $0.1660 per hour
Google Cloud Platform e2-standard-4, $0.1509 per hour

For spot instances, using comparable services on Azure and GCP:

AWS t4g.xlarge, $0.044 per hour. A 67% saving
Azure, A4 v2, $0.0348 per hour. An 85% saving
Google Cloud Platform, e2-standard-4, $0.0602 per hour. A 60% saving

Common Cloud Cost Optimization Techniques

To combat rising expenses, organizations try different common techniques, including:

Rightsizing resources, continuously monitors resource utilization and performance metrics to ensure you use appropriately sized instances and services. Organizations can eliminate the waste associated with over-provisioning by matching infrastructure to workload demands and ensuring they pay only for the capacity they need.
Leveraging pricing models and purchase options. Cloud providers offer pricing models beyond standard on-demand rates. For example, AWS offers Reserved Instances (RIs) and Savings Plans that give discounts in exchange for committing to a certain usage level. Spot Instances offer access to spare cloud capacity at reduced rates for fault-tolerant workloads that can handle interruptions.
Implementing auto-scaling to automatically adjust the number of compute resources allocated to an application based on demand. Automating shutdown schedules for non-production environments during off-hours also prevents paying for unutilized resources.

Limitations of Traditional Optimization

While conventional optimization techniques are valuable, they often have limitations, especially in data-intensive environments, such as:

Increased operational complexity. Managing rightsizing, reservations, and spot instances is complex and time-consuming in multi-cloud or hybrid cloud setups.
Challenges with dynamic workloads. Accurately forecasting usage for Reserved Instances or Savings Plans is difficult for applications with variable or unpredictable demand patterns. This can lead to over-committing and not solving the cost issue.
Inability to address data transfer costs. Most methods focus on optimizing compute and storage resource costs but fail to tackle the fundamental issue driving significant expense - the cost and performance impact of moving large volumes of data between storage locations and compute services, often across different regions or cloud providers.

The Solution: Bringing distributed compute and data together

Cloud cost optimization is challenging for businesses seeking to leverage cloud benefits without breaking the budget. However, the right solution to cloud cost optimization may be shifting your thinking about computing.

Bacalhau is an open-source solution that enables users to run compute and processing jobs where they generate and store data. Instead of running computations in one location that request data from another, process it, and send it back to another, with Bacalhau, you can run the whole process in one place.

With WASM and Docker support, you can run jobs with different programming languages. Bacalhau has GPU and edge device support, meaning you can still use the same cloud services you already use for compute or storage.

This brings the crucial flexibility you need, such as location, security, and device support, without the expense.

It reduces compute processes sitting idle waiting for something to do, as it’s the infrastructure you already use.

One-line install with many possibilities

Bacalhau is a one-line install that is configurable with a YAML file to determine what to run, where, and how. For smaller jobs, you can run everything on one Bacalhau instance. For more complex jobs, you can create a distributed network of orchestrator and compute nodes for distributing or processing job submissions. This is also useful if larger data sets are sharded across different locations. Each Bacalhau compute node can run over the data it can access and coordinate with the orchestrator to give the overall status and results.

This approach is also useful for global datasets that require processing differently for regulatory or security reasons. Bacalhau can process data in the same place as the data, returning aggregated and anonymized results and reducing concerns about security and privacy.

Reduce costs, not flexibility

The cloud promised reduced costs and complexity, but as countless reports and solutions for managing spiraling costs and complexity show, this hasn’t been successful.

For data processing, Bacalhau offers a simple and flexible solution to reduce cloud orchestration costs. Try the open-source version today, and if you want to know more about the hosted version, speak to the team to find out more.

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

If you don’t have a node network available and would still like to try Bacalhau, you can use Expanso Cloud. You can also set up your own cluster (with setup guides for AWS, GCP, Azure, and more 🙂).

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Reach out at any of the following locations:

Commercial Support

K8s vs Nomad vs Bacalhau: Choosing Your Compute Orchestrator Wisely

Federico Trotta — Fri, 18 Apr 2025 07:33:05 GMT

Are you wrestling with massive datasets scattered everywhere? Well, then you know that moving petabytes of data to a central cluster is slow, costly, and risky. While Kubernetes and Nomad are powerful orchestrators, they assume you'll bring data to the compute.

But what if that's your bottleneck? Bacalhau offers a different path: bring compute to the data.

Let’s see the differences!

Comparing Kubernetes, Nomad, and Bacalhau: Key Differences

Let's break down the key differences between these three.

What They Run

Kubernetes: Is the king of containers (Docker and more) with a huge ecosystem around them. WASM support is growing, but containers are its core strength.
Nomad: More flexible than k8s. Natively handles containerized and non-containerized apps. Great for mixed legacy and modern workloads.
Bacalhau: Natively runs Docker & WASM. Its pluggable architecture allows custom binaries too. Built to run compute where data lives, including leveraging specific hardware like GPUs.

Architecture & Footprint

Kubernetes: Powerful but complex control plane. Best suited for relatively homogenous clusters, usually in one data center/cloud.
Nomad: Lightweight (single binary). Simpler setup than K8s, good for managing diverse architectures within a cluster.
Bacalhau: Designed for distributed environments. Lightweight agent runs anywhere, focusing on executing jobs across varied nodes, not managing the infrastructure itself.

How They Handle Jobs

Kubernetes: Manages long-running services and batch jobs (Jobs/CronJobs). However, often needs to rely on frameworks like Spark or Flink for complex distributed data processing.
Nomad: Supports service, batch, and system jobs. Handles various task types efficiently.
Bacalhau: Optimized for "embarrassingly parallel" tasks common in data processing. Offers batch, service, ops, and daemon job types, giving flexibility for different distributed compute needs.

Dealing With Queues & Disconnection:

Kubernetes: Supports scheduling and queuing. Not natively designed for environments with frequent disconnections (like the edge).
Nomad: Handles job queues well. Clients can tolerate temporary network disconnects.
Bacalhau: Built for disconnected or intermittently connected environments (like the edge). Features explicit job queuing and can wait for nodes to reconnect.

Security Approach

Kubernetes & Nomad: Robust security features (RBAC, ACLs). However, the centralized model (moving data or needing broad access) increases risks like metadata leakage.
Bacalhau: Security advantage via "Compute Over Data". Minimizing data movement inherently reduces exposure. Less centralized metadata lowers breach impact.

When to Choose Which

So, how do you choose one or another for your application? Let’s break down the big ideas:

Kubernetes: Best for complex, containerized microservices needing rich networking and auto-scaling, primarily in a central location.
Nomad: Ideal for simpler orchestration of mixed workloads in one place, or when K8s feels too heavy.
Bacalhau: Your go-to if:
- Processing large, distributed datasets is the main goal.
- Data gravity (cost/time of moving data) is high.
- Data sovereignty/privacy rules prevent moving data.
- You need robust edge computing capabilities.

The Bottom Line

No solution suits all needs. You need to choose the orchestrator that fits your particular case.

K8s and Nomad excel at managing applications where compute and data are centralized. Bacalhau, on the other hand, shines when your data is spread out, tackling the challenges of data gravity and distribution head-on by bringing the compute to where your data already lives.

Want to know better how to make the right choice when searching for the right orchestrator? Read the complete article on our website!

What's Next?

To start using Bacalhau, install Bacalhau and give it a shot.

However, if you don’t have a network and you would still like to try it out, we recommend using Expanso Cloud. Also, if you would like to set up a cluster on your own, you can do that too (we have setup guides for AWS, GCP, Azure, and many more 🙂).

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations:

Commercial Support

While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. You can read more about the difference between open-source Bacalhau and commercially supported Bacalhau in our FAQ. If you would like to use our pre-built binaries and receive commercial support, please contact us or get your license on Expanso Cloud!

Expanso Wins 2025 Data Breakthrough Award

Laura Hohmann — Thu, 03 Apr 2025 16:45:47 GMT

Bacalhau Named Top Open Source Data Platform for the Second Year in a Row

For the second consecutive year, Expanso has been recognized in the Data Breakthrough Awards, this time winning “Open Source Data Platform of the Year” for Bacalhau. Last year, Bacalhau took home the award for “Data Processing Solution of the Year”—and now, its role in redefining how enterprises and developers orchestrate workloads at scale has earned it another top spot.

The Data Breakthrough Awards saw over 3,000 nominations globally, with winners including industry leaders like Cloudera, Alteryx, Fivetran, Red Hat, and Mercedes-Benz. This recognition underscores Bacalhau’s growing role in modern distributed computing.

"Expanso’s Bacalhau is a true breakthrough in the world of open-source data platforms, redefining how organizations process and manage data at scale,” said Steve Johansson, Managing Director, Data Breakthrough. “By bringing computation directly to the data—whether in the cloud, at the edge, or on-premises—Bacalhau is driving unprecedented efficiency, security, and real-time insights. We are thrilled to recognize Expanso with our ‘Open Source Data Platform of the Year’ award in the 2025 Data Breakthrough Awards, honoring their innovation and leadership in the evolving data landscape."

Why It Matters

Data-intensive organizations face a fundamental challenge: running workloads across clouds, regions, on-prem environments, and remote edge locations—without introducing unnecessary complexity or infrastructure sprawl. Bacalhau solves this.

With Bacalhau, enterprises can submit thousands of jobs, and the platform ensures they’re executed efficiently, wherever needed. It handles workload scheduling across dynamic environments, resolves failures and consensus when networks go down, and seamlessly integrates into existing data architectures.

This award is a testament to the developers, contributors, and customers who have helped shape Bacalhau into a powerful enterprise solution while remaining true to its open-source roots.

Looking Ahead

This is just the beginning. Bacalhau is evolving with new capabilities that further simplify workload orchestration, increase efficiency, and integrate deeper into enterprise ecosystems. Stay tuned—there’s more to come.

Read the official Data Breakthrough announcement here →

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Bacalhau v1.7.0 - Day 5: Distributed Data Warehouse with Bacalhau and DuckDB

Chris Chinchilla — Fri, 28 Mar 2025 19:29:06 GMT

This is part of the 5-days of Bacalhau 1.7 series! Make sure to go back to the start to catch all of them!

Distributed Data Warehouse with Bacalhau and DuckDB

With many applications that rely on data warehouses, you need to keep data sources in different locations. This could be due to privacy or regulatory reasons or because you want to keep processing close to the source. However, there are still times when you want to perform analysis on and across these data sources from one location but not move the data.

This post uses Bacalhau to orchestrate the distributed processing and DuckDB to provide the SQL storage and querying capacity for some mock sales data based in the EU and the US.

Prerequisites

To reproduce this tutorial, you need the following prerequisites:

Bacalhau CLI
Docker and Docker Compose
The example multi-region setup
- git clone https://github.com/bacalhau-project/examples.git

Architecture

The example Docker Compose file and Bacalhau job definitions in the example repository setup simulate the following architecture:

Bacalhau Orchestrator: Central control plane for job distribution
Compute Nodes: Distributed across regions, running close to data
Regional Storage: Regional data stores, using MinIO in this setup
DuckDB: SQL query engine running on each compute node. Bacalhau has a custom image that adds several user-defined functions to help with partitioning large data sets across nodes based on the following methods:
- partition_by_hash: Even distribution of files across partitions
- partition_by_regex: Pattern-based partitioning
- partition_by_date: Time-based partitioning

You can find more details on how these functions work in the custom image documentation.

You can see each component’s setup in the Docker Compose file. Create the architecture by running the following command:

docker compose up -d

The Docker Compose file uses several Bacalhau configuration files, which you can see in the configuration folder that label the compute nodes as US and EU nodes, respectively.

They also configure the orchestrator nodes to write data to the regional MinIO buckets.

Generate sample data

With the simulated infrastructure in place, you can now create sample data using the data generator job to write 3000 records to each region in JSON format to the relevant MinIO bucket.

# Move to the jobs directory cd ../jobs

Generate the US data:

bacalhau job run -V Region=us -V Events=3000 \ -V StartDate=2024-01-01 -V EndDate=2024-12-31 \ -V RotateInterval=month data-generator.yaml

Generate the EU data:

bacalhau job run -V Region=eu -V Events=3000 \ -V StartDate=2024-01-01 -V EndDate=2024-12-31 \ -V RotateInterval=month data-generator.yaml

Accessing data for analysis

Bacalhau supports two ways to access the regional data:

Bacalhau input sources

InputSources: - Type: s3 Source: Bucket: local-bucket Key: "data/*"

This method provides more control and preprocessing options and supports other source types in addition to S3.

Direct DuckDB access

SET VARIABLE files = ( SELECT LIST(file) FROM partition_by_hash('s3://local-bucket/**/*.jsonl') ); SELECT * FROM read_json_auto(getvariable('files'));

This method is a simpler and more familiar option for SQL-only jobs. The job definitions also use SQL queries to process data from an input source.

Run analysis

With data in place, you can send analysis tasks as Bacalhau jobs. In each case, after running the job, use bacalhau job describe to see the job results, passing the job ID from the output of the bacalhau job run command. All the examples show using US data. You can also change Region to eu to see the results from the EU region.

Monthly trend analysis

bacalhau job run -V Region=us monthly-trends.yaml

Job definition.

Sample output:

month | total_txns | revenue | unique_customers | avg_txn_value ------------|------------|----------|------------------|--------------- 2024-03-01 | 3,421 | 178,932 | 1,245 | 52.30 2024-02-01 | 3,156 | 165,789 | 1,189 | 52.53 2024-01-01 | 2,987 | 152,456 | 1,023 | 51.04

Operational monitoring

Hourly Operations

Tracks operational health metrics
Monitors transaction success rates
Shows hourly patterns

bacalhau job run -V Region=us hourly-operations.yaml

Anomaly Detection

Identifies unusual patterns
Uses statistical analysis
Flags significant deviations

bacalhau job run -V Region=us anomaly-detection.yaml

Business analytics

Product Performance

Analyzes category performance
Tracks market share
Shows sales patterns

bacalhau job run -V Region=us product-performance.yaml

Monthly Trends

Long-term trend analysis
Monthly aggregations
Key business metrics

bacalhau job run -V Region=us monthly-trends.yaml

Customer analysis

Customer Segmentation (Two-Phase)

Phase 1: Compute local metrics
Phase 2: Combine and segment

# Run Phase 1 bacalhau job run -V Region=us customer-segments-phase1.yaml # Note the job ID, then run Phase 2 bacalhau job run -V Region=us -V JobID= customer-segments-phase2.yaml

Summary

This post combined the distributed computing power of Bacalhau with the flexible SQL capabilities of DuckDB to create a distributed data warehouse spread across regions. The example Bacalhau jobs provide a range of analysis tasks, from operational monitoring to customer segmentation, all while keeping the data in its original location and using SQL to query data stored in S3-compatible buckets.

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Bacalhau v1.7.0 - Day 4: Using AWS S3 Partitioning With Bacalhau

Federico Trotta — Thu, 27 Mar 2025 16:03:02 GMT

This is part of the 5-days of Bacalhau 1.7 series! Make sure to go back to the start to catch all of them!

Processing large datasets from S3 can be a challenge, particularly when the size of the data exceeds certain values. The good news is that we made it a lot easier with Bacalhau!

While we introduced generic partitioning in Bacalhau v1.7, our new S3 partitioning feature handles data distribution automatically across multiple executions. It’s complete with failure handling, and independent retry of failed partitions, specifically on S3.

Let's dive into how this changes the game for distributed data processing.

Why Partition at All?

Before this feature, processing large S3 datasets was challenging. You had to create multiple jobs or write custom code to split the work. Without this effort, your machine would crash or slow down due to limited computing and disk throughput.

To solve this challenge, you had to figure out which part of your job was running on which machine. Then, you had to tell each part its position in the overall task. This was hard to manage. For example, if slice #3 of 8 failed, how would you know? Or how would you know which data should handle the #7 slice? More generally: how would you see the big picture of the entire job?

The Power of Automated Partitioning

Bacalhau 1.7.1 orchestrates everything for you. You just need to choose your partitioning strategy, and each task automatically gets its assigned subset of S3 objects. Your code stays clean and focused on its main job. If a partition fails, Bacalhau automatically retries only that and keeps the results from the successful ones.

For example, suppose you run a job and obtain a result as follows:

Job with 5 partitions: Partition 0: ✓ Completed Partition 1: ✓ Completed Partition 2: ✓ Completed Partition 3: ✗ Failed -> Scheduled for retry Partition 4: ✓ Completed

This means that Partition 3 has failed. However, Bacalhau will automatically retry the failed partition while preserving the results of the other four successfully completed jobs.

Partitioning Strategies for Every Need

Let’s now go through an overview of different partitioning strategies in different scenarios with data retrieved from S3.

No Partitioning: When Sharing Is Good

There are cases where every execution needs access to all the data and partitioning is not needed. Typical scenarios are:

Loading shared reference data
Processing configuration files
Running analysis that needs the complete dataset

In these cases, you can process the whole dataset you have like so with Bacalhau:

name: shared-reference-data

count: 3

...

tasks:

- inputSources:

- target: /data

source:

type: s3

params:

bucket: config-bucket

key: reference-data/

# No partition config - all executions see all files

In this case, the partition block is not present in the code. So your dataset remains as is.

Also, type: s3 under the source field specifies the type of data source used for this task. So the input data is coming from an S3-compatible storage system.

Object-Based Distribution: When Balance Matters

If you need to process many files without any specific grouping, object partitioning provides an even distribution of the load.

This solution is ideal for:

Processing large volumes of user uploads
Handling randomly named files
Large-scale data transformation tasks

Here is how Bacalhau handles this for you:

name: process-uploads

count: 5

...

tasks:

- inputSources:

- target: /uploads

source:

type: s3

params:

bucket: data-bucket

key: user-uploads/

partition:

type: object

In this case, the count: 5 processes data for 5 units, thanks to the partition block.

Processing by Date: Time-Series Analysis

Time-series analysis is the cross and delight of every data professional—well, sometimes more a cross than a delight!

With Bacalhau, you can use partitioning to process each day's data in parallel. This is the perfect case for:

Daily analytics processing
Log aggregation and analysis
Time-series computations

Here is how you can do so:

name: daily-log-analysis

count: 7 # Process a week's worth of logs in parallel

...

tasks:

- inputSources:

- target: /logs

source:

type: s3

params:

bucket: app-logs

key: "logs/*"

partition:

type: date

dateFormat: "2006-01-02"

In this case, the count:7 processes data for 7 units, representing the week’s data.

Processing by Region: Geographic Analysis

Analyzing geographical data is another scenario that may come with a lot of data. A solution is to distribute processing by region with partitioning. This enables scenarios like:

Regional sales analysis
Geographic data processing
Territory-specific reporting

Here is how you can manage this in Bacalhau:

name: regional-analysis

count: 3 # One execution per region

...

tasks:

- inputSources:

- target: /sales

source:

type: s3

params:

bucket: global-sales

key: "regions/*"

partition:

type: regex

pattern: "([^/]+)/.*"

For example, if you have data in regions/NA/, regions/EU/, regions/APAC/, etc., each execution will process one region's worth of data. The pattern: "([^/]+)/.*" is a standard Regex that does the following:

([^/]+): This part matches and captures one or more characters that are not a forward slash (/). This is the first capturing group.
/.*: This matches a forward slash (/) followed by zero or more characters (.*).

As a result, if the S3 key is regions/europe/sales.csv, the regex will capture europe.

Processing by Customer Segment

Another typical example of the usage of partitioning is customer segmentation. In this case, common analysis scenarios are:

Customer cohort analysis
Segment-specific processing
Category-based computations

You can handle your analysis with Bacalhau partitioning as follows:

name: segment-analytics

count: 4

...

tasks:

- inputSources:

- target: /segments

source:

type: s3

params:

bucket: customer-data

key: segments/*

partition:

type: substring

startIndex: 0

endIndex: 3

Combining Partitioned and Shared Inputs

In certain cases, you may need Bacalhau’s jobs to process partitioned data while sharing reference data that all executions need to access. Common scenarios are:

Processing daily logs with shared lookup tables
Analyzing data using common reference files
Running calculations that need both partitioned data and shared configuration

As an example, consider this job that combines static reference data with daily logs partitioned by date:

name: daily-analysis

count: 7# Process a week of data

...

tasks:

- inputSources:

- target: /config

source:

type: s3

params:

bucket: config-bucket

key: reference/*

# No partitioning - all executions see all reference data

- target: /daily-logs

source:

type: s3

params:

bucket: app-logs

key: logs/*

partition:

type: date

dateFormat: "2006-01-02"

This code partitions only the /daily-logs, subdividing them into weekly data with count:7.

The reference data (/config), instead, is not partitioned as it is not under the partition block.

Why This Changes Your Large Data Set Processing

As you have seen, this feature is simple yet powerful. You are no longer required to write partition-aware code: just clean, focused processing logic with automatic data assignment. We’ve already tested this on scaling to over 1,000 partitions with no code changes needed and automated load balancing. But you tell us if you’d like us to go even further!

Getting Started With S3 Partitioning

If you’d like to try this example on your own, dive right in! Install Bacalhau and give it a shot.

By the way, if you don’t have a network and you would still like to try it out, we recommend using Expanso Cloud. Also, if you'd like to set up a cluster on your own, you can do that too (we have setup guides for AWS, GCP, Azure, and many more 🙂).

What's Next?

Start processing your S3 data today:

Identify your natural data groupings (dates, regions, categories)
Choose the matching partition strategy
Let Bacalhau handle the distribution

Ready to simplify your distributed data processing? Check out our documentation for more examples and detailed guides.

Join our community to share your data processing stories and learn from others!

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Bacalhau v1.7.0 - Day 3: Streamlining Security: Simplifying Bacalhau's Authentication Model

Walid Baruni — Wed, 26 Mar 2025 16:01:32 GMT

This is part of the 5-days of Bacalhau 1.7 series! Make sure to go back to the start to catch all of them!

In the ever-evolving landscape of distributed computing, robust authentication and authorization mechanisms are essential for maintaining security while enabling seamless collaboration. The recent release of Bacalhau 1.7 introduces a significant overhaul to its authentication and authorization systems, offering more flexibility, improved security, and better integration with enterprise environments. Let's explore these exciting new features and understand how they enhance the Bacalhau ecosystem.

The Authentication Evolution

Prior to version 1.7, Bacalhau relied solely on Open Policy Agent for both authentication and authorization. While powerful, this approach placed the burden of generating complex policies on users and operators, creating a steep learning curve for newcomers. Additionally, the lack of OAuth 2.0 support limited integration possibilities with modern authentication systems, and the process of making authenticated API calls involved cumbersome back-and-forth communication.

With Bacalhau 1.7, these limitations have been addressed through the introduction of three distinct authentication paths, each designed to cater to different use cases and environments.

Authentication Options in Bacalhau 1.7

1. Basic Authentication

The simplest approach leverages the time-tested HTTP Basic Authentication protocol, allowing users to access Bacalhau APIs using traditional username and password credentials. These credentials can be defined in the Node Configuration file, offering two options for password storage:

Plain text passwords for simplicity and ease of setup
bcrypt-hashed passwords for enhanced security

For CLI usage, users simply need to set the environment variables BACALHAU_API_USERNAME and BACALHAU_API_PASSWORD. For direct API calls, the standard Basic Authorization header with base64-encoded credentials can be used.

Below is a sample orchestrator config file that defines 3 users that can authenticate through basic auth.

Orchestrator:

Enabled: true

API:

Port: 1234

Auth:

Users:

# User with plain text password

- Alias: Admin User

Username: admin

Password: secureAdminPassword

Capabilities:

- Actions: ["*"]

# User with limited permissions and plain text password

- Alias: Read Only User

Username: reader

Password: readerPassword

Capabilities:

- Actions: ["read:*"]

# User with bcrypt hashed password

- Alias: Job Manager

Username: jobmanager

# This is a bcrypt password hash for the password "MySecretPassword"

Password: "$2a$10$3ZvxUe5OudgRIQQheomjMO/Ufx1Bb04SH/y0PXnR19oDRXNGps3r2"

Capabilities:

- Actions: ["read:job", "write:job", "read:node"]

In this configuration:

We have three users with different permission levels:
- An admin user with full access to all capabilities
- A read-only user who can view but not modify any resources
- A job manager who can view nodes and has full control over jobs
The first two users have plain text passwords, while the third uses a BCRYPT hashed password for added security.

To help users and operators to generate secure hashed passwords, a convenient CLI command was added that generates a BCRYPT hash of a password of your choosing:

bacalhau auth hash-password

To use this configuration with the Bacalhau CLI, you would set the following environment variables:

# For admin access

export BACALHAU_API_USERNAME=admin

export BACALHAU_API_PASSWORD=secureAdminPassword

# For read-only access

export BACALHAU_API_USERNAME=reader

export BACALHAU_API_PASSWORD=readerPassword

# For job management

export BACALHAU_API_USERNAME=jobmanager

export BACALHAU_API_PASSWORD=MySecretPassword

For direct API calls using curl, you would encode the credentials in base64:

# For admin (base64 of "admin:secureAdminPassword")

curl -X GET -H "Authorization: Basic YWRtaW46c2VjdXJlQWRtaW5QYXNzd29yZA==" "http://orchestrator:1234/api/v1/orchestrator/nodes"

# For reader (base64 of "reader:readerPassword")

curl -X GET -H "Authorization: Basic cmVhZGVyOnJlYWRlclBhc3N3b3Jk" "http://orchestrator:1234/api/v1/orchestrator/nodes"

# For Job Manager (base64 of "jobmanager:MySecretPassword")

curl -X GET -H "Authorization: Basic am9ibWFuYWdlcjpNeVNlY3JldFBhc3N3b3Jk" "http://orchestrator:1234/api/v1/orchestrator/nodes"

2. API Tokens

For applications and scenarios where password-based authentication isn't ideal, Bacalhau 1.7 introduces API token support. Instead of username and password pairs, users can generate and use API keys as bearer tokens in authorization headers.

Configuration is straightforward – API keys are defined in the orchestrator config under user profiles. To use them with the Bacalhau CLI, users set the BACALHAU_API_KEY environment variable. For direct API access, the token is included in the Authorization header using the Bearer scheme.

Please note that API Keys are opaque tokens.

Here's a sample configuration for API tokens in Bacalhau:

Orchestrator:

Enabled: true

API:

Port: 1234

Auth:

Users:

# Administrator API token with full access

- Alias: Admin API Token

APIKey: 8F42A91D7C6E4B3DA5E9F8C12B76D3A4

Capabilities:

- Actions: ["*"]

# Read-only API token

- Alias: Monitoring Token

APIKey: C5D8E3F1A7B94026895C1D4E3F2A0B78

Capabilities:

- Actions: ["read:*"]

# Job management API token

- Alias: CI/CD Pipeline Token

APIKey: 2E8D7F5B3A9C41608D2E6B7F4A5C3D9E

Capabilities:

- Actions: ["read:job", "write:job", "read:node"]

# Agent management API token

- Alias: Agent Management Token

APIKey: 1A3B5C7D9E0F2G4H6I8J0K2L4M6N8P0

Capabilities:

- Actions: ["read:agent", "write:agent"]

In this configuration:

We have four API tokens with different permission levels:
- An administrator token with full access to all capabilities
- A monitoring token with read-only access to all resources
- A CI/CD pipeline token that can view nodes and has full control over jobs
- An agent management token that has full control over agents
Each token has a unique, randomly generated API key. You should generate strong, unique keys for your production environment using a secure random generator. Please note that API keys do not support BCRYPT hashing

To use these API tokens with the Bacalhau CLI, you would set the following environment variable:

export BACALHAU_API_KEY=8F42A91D7C6E4B3DA5E9F8C12B76D3A4

For direct API calls using curl, you would use the Bearer token authentication scheme:

curl -X GET -H "Authorization: Bearer 8F42A91D7C6E4B3DA5E9F8C12B76D3A4" "http://orchestrator:1234/api/v1/orchestrator/nodes"

3. Single Sign-On via OAuth 2.0

Perhaps the most significant addition is the support for OAuth 2.0 using the Device Code Flow. This enables Bacalhau to integrate seamlessly with enterprise identity providers such as Okta, Auth0, Azure Active Directory, and Google SSO.

This approach eliminates the need to define users directly in Bacalhau's configuration, instead delegating user management to the identity provider – a considerable advantage in corporate environments with existing identity infrastructure.

The configuration process involves specifying OAuth 2.0 endpoints, client IDs, and desired scopes. When users need to authenticate, they run bacalhau auth sso login, which presents a device code and URL. After completing authentication through their browser, they receive a JWT token that's automatically used for subsequent API calls (this token exchange will be done seamlessly and the user is not required to perform any extra actions).

Here's a sample configuration for OAuth 2.0 SSO in Bacalhau:

Orchestrator:

Enabled: true

API:

Port: 1234

Auth:

Oauth2:

# Identity provider details

ProviderId: "okta"

ProviderName: "Okta SSO"

# OAuth 2.0 endpoints

DeviceAuthorizationEndpoint: "https://your-domain.okta.com/oauth2/v1/device/authorize"

TokenEndpoint: "https://your-domain.okta.com/oauth2/v1/token"

Issuer: "https://your-domain.okta.com"

JWKSUri: "https://your-domain.okta.com/.well-known/jwks.json"

# Client details

DeviceClientId: "0ab2c3d4e5f6g7h8i9j0"

PollingInterval: 5

# Application settings

Audience: "https://bacalhau.your-company.com/api"

Scopes:

- "openid"

- "profile"

- "email"

For this to work properly, you need to:

Register an OAuth 2.0 application in your identity provider (Okta, Auth0, Azure AD, etc.)
Configure it to support the device code flow. Make sure the provider supports OAuth2 Device code flow.
Set up appropriate roles or groups in your identity provider to map to Bacalhau permissions

The permission mapping would happen in your identity provider. For example, in Okta you might create:

A "Bacalhau Admins" group with permissions: ["*"]
A "Bacalhau Readers" group with permissions: ["read:*"]
A "Bacalhau Job Managers" group with permissions: ["read:job", "write:job", "read:node"]

These permissions should be included in the JWT token under the custom claim permissions when users authenticate.

To authenticate using this setup, users would run:

# Login

bacalhau auth sso login

# Logout

bacalhau auth sso logout

The CLI would display something like:

To login, please:

1. Open this URL in your browser: https://your-domain.okta.com/activate

2. Enter this code: ABCD-EFGH

Or, open this URL in your browser: https://your-domain.okta.com/activate?user_code=ABCD-EFGH

Waiting for authentication with Okta SSO... (press Ctrl+C to cancel)

After completing authentication through their browser, the user would receive a JWT token that's automatically used for subsequent API calls. The token can be inspected with:

# Inspect JWT token obtained when logiing in using SSO

bacalhau auth sso token

Authentication Priority in Bacalhau 1.7

When configuring Bacalhau authentication, it's important to understand the precedence rules that determine which authentication method takes effect.

Environment variables take highest precedence in the authentication hierarchy, overriding any other configured methods. This means that if you have set BACALHAU_API_USERNAME and BACALHAU_API_PASSWORD for Basic Auth, or BACALHAU_API_KEY for API token authentication, these will be used regardless of any SSO tokens that may be stored locally from previous bacalhau auth sso login sessions.

This design provides flexibility for users who need to temporarily switch between different authentication contexts without modifying configuration files

For example, a developer could have an SSO session for regular work but quickly switch to using an API key for testing by simply setting the appropriate environment variable. When the environment variable is unset, Bacalhau will fall back to the next available authentication method, typically returning to the previously established SSO session if available.

A command was added to inspect the current authentication status for a CLI:

# Inspect current authrentication status

bacalhau auth info

Granular Authorization in Bacalhau 1.7

Bacalhau 1.7 introduces a sophisticated authorization system built on a resource and capability model that brings fine-grained access control to the platform. This system divides API actions into specific combinations of resources and capabilities, enabling administrators to implement the principle of least privilege across their Bacalhau deployments.

Resource and Capability Framework

The permission structure is organized around two key dimensions:

Resources: The objects being accessed or modified (Nodes, Jobs, and Agents)
Capabilities: The types of operations being performed (Read and Write)

This creates a permission taxonomy following the pattern of action:resource, where permissions can be assigned individually or using wildcards for broader access grants.

Core Permission Set

The complete set of permissions available in Bacalhau includes:

"*" - The master permission granting full access to all capabilities across all resources
"read:*" - Provides read-only access across all resource types
"write:*" - Grants write access to all resource types
"read:node" - Allows viewing node information
"write:node" - Permits actions on the node
"read:job" - Enables querying job status, details, and logs, etc
"write:job" - Allows submitting, canceling, and managing job execution
"read:agent" - Provides access to agent information via "bacalhau agent" commands
"write:agent" - Any write actions on the agent. Currently no write actions are supported

Creating Role-Based Access Patterns

These permissions can be combined to create practical access patterns for different user roles and service accounts:

Administrator: ["*"] - Full access to all system functions
Read-only Analyst: ["read:*"] - Can view but not modify any resources
Job Manager: ["read:job", "write:job", "read:node"] - Complete control over jobs with visibility into nodes
Monitoring Service: ["read:node", "read:job"] - View-only access for system monitoring
CI/CD Pipeline: ["write:job", "read:job"] - Can submit and monitor jobs but can't access node details

Benefits for Different User Profiles

These authentication enhancements offer distinct advantages for different types of Bacalhau users:

Individual developers benefit from the simplicity of Basic Auth for quick setup and experimentation
DevOps teams can leverage API tokens for automation, CI/CD pipelines, and service-to-service communication
Enterprise environments gain seamless integration with existing identity infrastructure through OAuth 2.0
Security teams appreciate the granular permission model that enforces the principle of least privilege

Practical Implementation

Implementing these new authentication mechanisms is straightforward. For Basic Auth and API Tokens, you simply update your orchestrator configuration to include user definitions with appropriate capabilities.

For OAuth 2.0, you configure the connection to your identity provider and ensure appropriate permissions are assigned to users.

Backward Compatibility with Previous Authentication Methods

Bacalhau 1.7 maintains backward compatibility with the previous authentication mechanism based on Open Policy Agent, ensuring a smooth transition path for existing deployments.

Users can continue to use their established OPA policies without immediate migration to the new authentication paths. However, it's important to note that while backward compatibility is preserved, mixing the old and new authentication methods within the same deployment is not supported.

Organizations must choose either to continue using the Open Policy Agent approach exclusively or to migrate fully to the new authentication system with Basic Auth, API Tokens, or OAuth 2.0.

This clean separation prevents potential security inconsistencies and configuration conflicts that could arise from overlapping authentication mechanisms.

For organizations planning to migrate, the Bacalhau team recommends first setting up the new authentication in a test environment, validating access patterns and permissions, and then performing a complete cutover rather than attempting a gradual or partial migration. This approach ensures security integrity throughout the transition while still providing flexibility in timing the upgrade to the enhanced authentication capabilities.

Looking Forward

Bacalhau Auth 2.0 represents a significant step forward in the platform's security and integration capabilities. By providing multiple authentication paths and a resource-based authorization model, Bacalhau has become more accessible while simultaneously enhancing its security posture.

Looking ahead, the Bacalhau development team is committed to further expanding these security capabilities. A key focus for upcoming releases will be the implementation of even more granular namespace-based authorization and authentication. This enhancement will allow organizations to create logical boundaries within their Bacalhau deployments, enabling multi-tenant environments where different teams or projects can operate independently with their own security contexts and resource limitations.

We understand that changes to authentication can be challenging, so we're here to help with your migration. Join our community discussions for support.

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Bacalhau v1.7.0 - Day 2: Scaling Your Compute Jobs with Bacalhau Partitioned Jobs

David Aronchick — Tue, 25 Mar 2025 17:01:40 GMT

This post is part of the 5-days of Bacalhau 1.7.0 series. Be sure to go back and catch any that you missed!

We read it every day: big data is everywhere and growing constantly. Companies are sitting on goldmines of data–both their own data and data they’ve retrieved from elsewhere. However, the amount of data is often too large for a single instance. In the mid-2010s, the IT community adopted parallel machines to accelerate workloads by increasing throughput. It was able to do this by leveraging innovations such as Docker and Kubernetes, but the approach was typically limited to a single data center and compute-focused. To this day, the community needs more flexibility.

The most pressing challenges include:

How do you split your data so that each node can work efficiently?
How do you tell each node exactly which part of the data it is responsible for?
What happens if one of the computing nodes fails?
What happens when part of your processing fails?
How do you ensure consistency and reliability?

Fear no more! The new Partitioned Jobs feature in Bacalhau v1.7.0 addresses these challenges.

What are Partitioned Jobs?

When processing large datasets, splitting the work across multiple nodes can significantly improve performance and resource utilization. Bacalhau's new Partitioned Jobs feature makes this process straightforward by:

Distributing work across a compute pool
Managing partition assignments and tracking
Handling failures at a partition level
Providing execution context to each partition

Core Features

Let's look at the core features of Partitioned Jobs.

1. Partition Management

Bacalhau v1.7.0 manages the two key aspects of partition management: distribution and independent execution.

Distribution

When you specify multiple partitions, Bacalhau creates N partitions (0 to N-1) and assigns them to available compute nodes that match the data and other specified constraints.

Bacalhau maintains consistent partition assignments throughout the job lifecycle, and it ensures that each partition finishes correctly.

Independent Execution

Each partition runs independently, can be processed on different nodes, and manages its own lifecycle and error handling.

2. Error Handling and Recovery

A key strength of the Partitioned Jobs feature is its approach to error handling because partition-level isolation contains failures in individual partitions. This enables the system to continue processing while the failed partitions recover.

For example, let’s say you have a job subdivided into five partitions. Running a partitioned job in Bacalhau will generate an output like the following:

Job with 5 partitions:

Partition 0: ✓ Completed

Partition 1: ✓ Completed

Partition 2: ✓ Completed

Partition 3: ✗ Failed -> Scheduled for retry

Partition 4: ✓ Completed

In this case, Partition 3 failed. However, the system automatically recovers. In a few minutes, the job status is as follows:

Job with 5 partitions:

Partition 0: ✓ Completed

Partition 1: ✓ Completed

Partition 2: ✓ Completed

Partition 3: ✓ Completed

Partition 4: ✓ Completed

Bacalhau automatically recovers and takes care of rescheduling, even over widely distributed systems. Neat!

3. Execution Context

Each job receives information about its responsibilities in the parallel execution through environment variables. For example, with every job execution, it has access to the following information:

BACALHAU_PARTITION_INDEX # Current partition (0 to N-1) BACALHAU_PARTITION_COUNT # Total number of partitions

# Additional context variables BACALHAU_JOB_ID # Unique job identifier BACALHAU_JOB_TYPE # Job type (Batch/Service) BACALHAU_EXECUTION_ID # Unique execution identifier

Thus, each node can make smart decisions about the data it needs from the pool and any special execution criteria. There is no need to check in with a central catalog or other jobs, resulting in more reliable and faster parallel execution.

Using Partitioning in Your Jobs

To use partitioning, specify the number of partitions when running your job. Here is how you can specify three partitions:

bacalhau docker run --count 3 ubuntu -- sh -c 'echo Partition=$BACALHAU_PARTITION_INDEX'

In the above code, the --count flag specifies the number of partitions. Thus, this is the actual trigger of the partitioning.

Here is how to do the same via YAML:

# partition.yaml Name: Partitioned Job Type: batch Count: 3 Tasks: - Name: main Engine: Type: docker Params: Image: ubuntu Parameters: - sh - -c - echo Partition=$BACALHAU_PARTITION_INDEX

To run the YAML file, execute the following:

bacalhau job run partition.yaml

That’s it! Bacalhau automatically takes care of scheduling and everything else.

Technical Benefits

Bacalhau's partitioning feature offers significant technical advantages for processing large datasets and compute-intensive tasks:

Enhanced performance and scalability: By distributing work across multiple nodes, partitioning enables horizontal scaling and parallel processing. This approach eliminates any constraints related to I/O and memory bandwidth of a single machine, maximizing utilization and reducing processing time.
Increased reliability and resilience: Partitioning provides granular failure recovery by isolating errors in individual partitions. If one partition fails, the system continues processing the other partitions without interruption. Bacalhau's built-in retry mechanism ensures that failed partitions are automatically rescheduled, enhancing the resilience of your jobs. This approach preserves results from completed partitions, preventing unnecessary reprocessing.
No rewriting code: Your data processing jobs do not require a new SDK. If you're already using WASM or containers (or just about any other execution environment), we support it!

Getting Started with Partitioned Jobs

If you’d like to try this example on your own, dive right in! Install Bacalhau and give it a shot.

What's Next?

Start processing your data today:

Identify your natural data groupings (dates, regions, categories)
Choose the matching partition strategy
Let Bacalhau handle the distribution

Ready to simplify your distributed data processing? Check out our documentation for more examples and detailed guides.

Join our community to share your data processing stories and learn from others!

Subscribe now

Get Involved!

We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Announcing Bacalhau 1.7.0: Empowering Enterprises with Enhanced Scalability, Job Management, and Support

Sean M. Tracey — Mon, 24 Mar 2025 17:10:17 GMT

This post is part one of the 5-days of Bacalhau 1.7.0 series. Be sure to go back and catch any that you missed!

We’re thrilled to announce the release of Bacalhau v1.7.0, taking one more step in our continuing mission to bring robust, scalable distributed computing to developers and enterprises alike. We’ve packed this release with an array of features designed to streamline operations and enhance security, as well as introducing comprehensive support to enable teams and organizations to confidently deploy Bacalhau at scale.

With v1.7.0, we’re rolling out the following:

Self-service licensing configuration
Enhanced job execution with partitioned workloads
Streamlined job management and templates in Expanso Cloud
Simplified authentication via username/password, API key, and single sign-on (SSO)

Self-service Licensing Configuration

Bacalhau v1.7.0 introduces enterprise support options, offering flexible licensing models to meet the needs of any organization. Whether you prefer the fully managed orchestration of Expanso Cloud or on-premises support, we've got you covered:

Self-Service Licensing: Easily purchase support licenses through the Expanso Cloud portal, tailored to your node count.
Tiered Support: Choose the support level that matches your organization's needs, ranging from startups to large enterprises.
Simplified Deployment: Download your license from Expanso Cloud and deploy it to your orchestrator—no need for complex node-level licensing.

Enhanced Job Execution

Process terabytes of data efficiently with our new partitioned execution system. Each node automatically understands its specific workload portion, enabling parallel processing that scales linearly with your infrastructure—all without writing complex coordination code:

Deterministic Partitioning: Distribute work intelligently across compute nodes using environment variables that indicate each execution's position.
Efficient Data Processing: Process large datasets with coordinated distribution, ensuring each node understands its specific workload portion.
S3 Native Partitioning: Amazon Web Services recommends native partitioning, and we natively integrate with it, making it much easier to us!

Simplified Setup and Networking

One thing that we really want to do with Bacalhau is remove (as much as possible) the challenges of setting up a Bacalhau network. We have made some big steps towards that here:

Bacalhau Docker-in-Docker: The Bacalhau and Expanso teams now support a containerized version. This makes it particularly easy to deploy with docker tools (including Docker Compose) and Podman!
Smarter Default Networking: One of the biggest challenges we have seen is people setting up and managing their networking. We have chosen some smarter defaults that should make it simpler to use (or lock down) exactly what you need. This includes native WASM networking as well!
Host Environment Variables: While many jobs are treated as ephemeral, we do want to be able to pass through information about the underlying node so that jobs can act smartly. We have added that natively as well!

Streamlined Job Management and Templates in Expanso Cloud

Expanso Cloud now offers enhanced job submission capabilities through a web interface. The enhancements include:

Job Templates: Choose from predefined templates for common use cases like log processing, data analysis with DuckDB, and Apache Iceberg operations.
Custom Workflows: Create and share custom job templates, simplifying complex distributed computing tasks.
Customizable Placeholders: Reuse templates with different values to streamline common tasks.

The enhanced Expanso Cloud platform dramatically reduces the learning curve for distributed computing. Data scientists and developers can now launch complex distributed workflows in minutes instead of days, using pre-configured templates for common use cases.

Simplified Authentication and Single Sign-On

Bacalhau v1.7.0 features a completely redesigned authentication system:

Standardized Approach: Adopts familiar, widely-used authentication standards.
Enterprise Choice: Whether using username/password, API keys, or single sign-on (SSO), Bacalhau supports the way you want to authenticate.
Fine-Grained Permissions: Bacalhau supports an entire array of nouns and verbs that you can use to provide granular authentication.

Our redesigned authentication system delivers enterprise-grade security without the complexity. SSO, API tokens, and fine-grained permissions make it easy to integrate Bacalhau into your existing security infrastructure while maintaining strict access controls.

Join Us on the Journey: 5 Days of Bacalhau

Stay tuned for our "5 Days of Bacalhau" series, in which we'll delve deeper into these exciting new features:

Day 1: Announcing Bacalhau 1.7.0 (this post)
Day 2: Scaling Your Compute Jobs with Bacalhau Partitioned Jobs
Day 3: Streamlining Security: Simplifying Bacalhau's Authentication Model
Day 4: Using AWS S3 Partitioning With Bacalhau
Day 5: Distributed Data Warehouse with Bacalhau and DuckDB

Bacalhau is rooted in community-driven advancements. If you have ideas or features that you’d like to see—get involved! Check out Bacalhau v1.7.0 and dive into our community:

GitHub: Contribute to the project.
Slack: Join our community.
Website: Learn more about Bacalhau.
Follow our social media.

We're excited to learn how you leverage the power of Bacalhau v1.7.0 to transform your distributed computing workflows.

Onwards together!

The Bacalhau Team

Subscribe now

Get Involved!

Have a unique use case? We’d love to hear about it! Share your projects and ideas, and let’s build the future of distributed systems together.

There are many ways to contribute and get in touch, and we’d love to hear from you! Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

The Modern Data Stack: A Scalable Future with Distributed Computing

Mandy Moore — Thu, 20 Mar 2025 17:01:49 GMT

Data is being generated at an incredible, growing volume, with projections estimating 394 zettabytes of global data generation by 2028. Companies across industries - whether in finance, cloud technology, energy, or healthcare - depend on modern data platforms to process, analyze, and extract insights from this ever-growing volume of information. However, as that volume surges, traditional modern data stack tools are increasingly strained, facing challenges in scalability, efficiency, and cost management.

This is where modern solutions like distributed computing and Compute Over Data (CoD) can change the game. By shifting computation closer to the data source, these approaches eliminate inefficiencies inherent in traditional pipelines. Bacalhau exemplifies this shift, providing a powerful tool that brings computation to where the data is—whether on edge devices, in the cloud, or on-premises—optimizing performance, reducing latency, and ensuring cost-efficient data processing.

What is the Modern Data Stack?

The modern data stack refers to the tools and technologies used for collecting, storing, processing, and analyzing data at scale. A robust modern data platform consists of the following layers:

Data Sources – Applications, IoT devices, log generators, and business systems generating data.
ETL/ELT Pipelines – Transformation tools that structure and clean raw data before storage.
Storage Solutions – Cloud data warehouses, data lakes, and databases that store structured and unstructured data.
Transformation & Analytics – Modern data tools for processing data into meaningful insights.
Visualization & Business Intelligence – Business intelligence tools that turn data into actionable insights.

The Shortcomings of the Traditional Data Stack

Despite its success, the traditional data stack presents major challenges:

Fragmentation – Stitching together multiple modern tools increases complexity, often requiring businesses to invest heavily in system integration.
High Costs – Cloud storage and query processing costs escalate as data volumes grow.
Latency & Bottlenecks – Real-time analytics and parallel processing are hindered by centralized architectures.
Siloed Data – Legacy stacks prevent seamless data access across departments.
Security Concerns – Centralized cloud-based tools create security vulnerabilities.

How Bacalhau Reinvents the Modern Data Stack

While many popular solutions rely on moving data through costly and time-consuming pipelines, Bacalhau solves these challenges by enabling Compute Over Data processing workloads directly where the data is located.

Key Benefits of Bacalhau

1. Faster, More Efficient Data Processing

Bacalhau eliminates latency by executing workloads at the data source, enabling near real-time insights.

2. Reduced Infrastructure and Cloud Costs

Bacalhau leverages existing compute resources instead of requiring expensive centralized cloud services, drastically reducing operational costs.

3. Enhanced Security & Compliance

By processing data in place, Bacalhau minimizes exposure to cyber threats and ensures compliance with regulations like GDPR and HIPAA.

4. Seamless Integration Across Environments

Bacalhau works effortlessly across cloud data warehouses, on-premises solutions, and edge computing environments.

Transforming Business Intelligence with Compute Over Data

Business intelligence tools traditionally rely on pre-aggregated datasets and scheduled reports. With Bacalhau, organizations can unlock new capabilities by running compute directly where data resides, enabling:

The generation of real-time insights by supporting real-time analytics.
More informed decision-making through efficient processing for predictive analytics.
AI-powered insights by accelerating access to distributed data.
Improved data visualization by facilitating seamless integration with leading visualization tools.

Edge Computing: Bringing Processing Closer to the Data

Traditional cloud-based data warehouses process data centrally, leading to considerable time delays and increased costs. Edge computing with Bacalhau addresses these issues by processing data at the source.

Key Advantages of Edge Computing with Bacalhau

Minimized latency – Instant insights by reducing data transfer.
Lower bandwidth costs – Process only relevant data before sending it to the cloud.
Enhanced security – Sensitive data stays closer to its origin.
Faster decision-making – Supports AI-driven self-service analytics tools.

Why Bacalhau is the Future of the Modern Data Stack

The modern data stack must evolve beyond cloud-based tools to meet the growing needs of business intelligence, AI, and real-time analytics. Bacalhau offers a scalable, cost-efficient, and flexible alternative to legacy systems.

By integrating Bacalhau, you gain:

A unified platform for managing workloads across cloud, edge, and on-premises environments.
Instant access to real-time insights through efficient data processing.
Lower IT costs with reduced cloud dependency.
Optimized security & compliance for business users in finance, healthcare, and energy industries.

Get Started with Bacalhau Today

Ready to revolutionize your IT infrastructure? Download Bacalhau now or contact our team to explore how we can help optimize your modern data platform.

Subscribe now

Get Involved!

Expanso’s tools and templates make it easy to get started. Check out our public GitHub repository for examples and guides, and join the conversation on social media with #ExpansoInAction.

Have a unique use case? We’d love to hear about it! Share your projects and ideas, and let’s build the future of distributed systems together.

There are many ways to contribute and get in touch, and we’d love to hear from you! Please reach out to us at any of the following locations.

Commercial Support

⭐️ GitHub

Slack

Website

Bacalhau

Bacalhau v1.8.0 - Day 4: Seamless Result Storage with Managed Publishers with Expanso Cloud

The Old Way: A Trail of Configuration and Credentials

The New Way: Expanso Cloud Managed Publishers

Getting Your Results: UI or CLI, Your Choice

Don't Worry, Your Custom Setups Still Work

A Better Workflow for Everyone

Conclusion

What's Next?

Get Involved!

Commercial Support

Bacalhau v1.8.0 - Day 3: How Bacalhau Boosts Daemon Job Reliability

Why Daemon Jobs are Your Cluster's Backbone

Smarter, Faster, Stronger: The New Daemon Jobs Orchestration

What This Means for Your Real-World Workloads

Upgrade with Confidence

Conclusion

What's Next?

Get Involved!

Commercial Support

Bacalhau v1.8.0 - Day 2: Rerun, Update, and Version Your Bacalhau Jobs

A Fundamental Shift: From Cryptic IDs to Meaningful Names

What's New: Your Job Management Superpowers

The rerun Command: Stop Re-Submitting, Start Rerunning

Enhanced Job Versioning: Manage Your Job's Evolution

Advanced Dry-Run: The “git diff" For Your Compute Jobs

Your New Workflow, Transformed

Conclusion

What's Next?

Get Involved!

Commercial Support

Announcing Bacalhau v1.8.0: Intelligent Edge Computing Meets Enterprise Integration

Slash Your Splunk Bill: Intelligent Logging at the Edge

Enhanced Daemon Jobs

Enterprise-Ready by Default

Transformative Job Lifecycle Management

Goodbye j-f47ac10b-58cc-4372-a567-0e02b2c3d479

Advanced Versioning and Dry-Run

Intelligent Job Rerunning

What’s next?

Join Us on the Journey: 5 Days of Bacalhau

Get Involved!

Commercial Support

The Elephant in the Room: Why Is Your Data Bill So High?

A Different Message: You Should Be Paying Less

How About Something Practical?

What's Next?

Get Involved!

Commercial Support

Why Cloud-Centric Architectures Are Breaking Under Data Scale

The Cloud Isn't Always Cheap: Costs Go Up, Benefits Go Down

Slow Data: Why Your Information Can't Keep Up

Rule Troubles: Following Data Laws in a Central Cloud

Why Your Current Infrastructure Tools Struggle with Spread-Out Data

The "Send Everything" Habit Costs a Lot

A Smarter Way: Compute-Over-Data with Bacalhau

The New Idea: Send Computer Power to Your Data, Not Data to Your Computers

What is Bacalhau? Your Tool for a World of Spread-Out Data

How Bacalhau Solves Data Problems

Real Examples: Where Compute-Over-Data With Bacalhau Makes a Difference

Conclusion

What's Next?

Get Involved!

Commercial Support

Cross-Border Data Processing With Privacy Compliance Through Expanso

A practical guide

Prerequisites

Set up multi-region deployment environment

Install the Bacalhau CLI

Clone example job repository

Generate fake sensitive data

Anonymize the data

Bacalhau input and output configuration

Cleanup

Summary

Your Fast Track to Bacalhau: Local Development via Docker-in-Docker

The Bacalhau Basics

The Local Solution: Docker Compose To The Rescue

The Inception Moment: Docker-in-Docker (DinD) Comes Into The Game

Why This Matters: The Benefits

The `rerun` Command: Stop Re-Submitting, Start Rerunning

Advanced Dry-Run: The “`git diff`" For Your Compute Jobs