Achieving Scale and Savings with Bacalhau and AWS Spot Instances
5 Days of Bacalhau (05:46)
Big Data usually means big costs, but it doesn’t have to. With Bacalhau, you can tackle your Big Data challenges while significantly reducing expenses. By leveraging a distributed compute network, Bacalhau lowers bandwidth usage, optimizes compute resources, and eliminates the need for centralized infrastructure.
But, we can still do more to keep our costs low whilst working on the big problems we have today. And we can do that with AWS Spot Instances.
What is a Spot Instance?
A Spot Instance is a type of EC2 instance from a pool of unused EC2 capacity that AWS offers at a substantial discount. The utilization of Spot Instances enables us to spin up a great number of EC2 instance types, but for a fraction of the cost of their on-demand counterparts. This makes them great options for running as Compute nodes as part of a Bacalhau network, but they come with a caveat: A Spot Instance can be reclaimed by AWS at any time.
This presents a double-edged sword. While we can access affordable compute resources for our Bacalhau networks, we can’t guarantee their availability for a fixed duration.
Not a Problem!
Bacalhau excels in environments with unreliable network and node availability. If a job can’t be completed on a given node, Bacalhau will automatically reschedule it elsewhere on the network. So, if a Spot Instance is reclaimed by AWS, Bacalhau will reassign those jobs to other nodes, fully utilizing all available compute resources to ensure your jobs complete successfully.
To follow this content, head to GitHub and clone the examples repo to your local system - or, you can run git clone https://github.com/bacalhau-project/examples in your terminal. This will create an `examples` folder on your system.
Spinning up a Bacalhau Orchestrator
If you already have a Bacalhau Requester Node set up, you can skip this section.
To get our Bacalhau network running, we first need to spin up an Orchestrator or ‘Requester node’ that our Compute nodes can connect to, and that we can interact with to fire off jobs to our network.
In the examples/spot directory, you’ll find scripts for provisioning our Spot Instances. But, the script we're going to use first is the `create_requester.py` script. This script will create an On-Demand t2.medium EC2 instance and configure it to run as a Bacalhau Requester node.
To run the script, execute:
python3 create_requester.pyThis should take a few seconds to run, and will create a Security Group, ingress rules, and the EC2 instance in the us-west-2 region. Once completed, it will output the public IPv4 address of your new Requester Node.
Take note of this IP address, we'll need it shortly.
Provisioning AWS Spot Instances
Once we have a Requester Node spun up, we can start provisioning AWS Spot Instances. cd up one level to the examples/spot directory. Here, we have a main.py file which we can use to specify the number of Spot Instances we want to provision and the IP address of our Requester node. We can also use this script to list the already provisioned Spot Instances and to destroy them once you’re finished.
Installing Dependencies
First, you'll need to install the necessary requirements for the Python script. To ensure our dependencies don't conflict with anything on your system, we'll create a virtual environment.
1. Create the virtual environment:
python3 -m venv VENV2. Activate the virtual environment:
source VENV/bin/activate3. Install the required packages:
pip3 install -r requirements.txtThis will install everything our Python script needs to provision our Spot Instances.
Creating our Spot Instances
Once we have all of our dependencies installed, we're ready to spin up our Spot Instances!
In your CLI, run the following command:
main.py create --orchestrators "nats://<YOUR REQUESTER NODES IP ADDRESS>:4222" --no-of-instances 10
A table will appear showing you the progress of the Python script as it provisions your instances. This will provision 10 Spot Instances of the t2.medium type.
The --orchestrators flag tells the script where the Compute Nodes can find the Requester Nodes once they’ve finished spinning up and Bacalhau is installed and running.
The --no-of-instances flag lets you set how many Spot Instances you want to provision and configure as Bacalhau Compute nodes.
Verifying our Setup
Once all of our instances have been provisioned, we can use the Bacalhau CLI to connect to our requester node and list all of the Compute Nodes that have connected to it and are ready to receive jobs. Run the following command in your terminal:
bacalhau --api-host=<YOUR REQUESTER NODE IP ADDRESS> node list --filter-status="connected"You should see a list of nodes connected to your Requester Node, and they should have an APPROVAL status of APPROVED and a STATUS of CONNECTED.
Running a Job
Now that we have a Bacalhau network all spun up, we can submit a job to the Compute Nodes.
In the examples/spot folder, there's a demo_job.yaml file. It's an Ops Job, so it will run on all Compute Nodes connected in our Bacalhau network, and all it's going to do is echo out a "Hello, World." program for us. Not a complex task, but enough to show that everything is working as expected.
To run the job on all nodes, enter the following command into your CLI:
bacalhau --api-host=<YOUR REQUESTER NODE IP ADDRESS> job run demo_job.yaml --template-vars "greeting=Hello,name=World"This should take a few seconds to complete, and once it's done, you should see something like the following:
To get the results of all of the Jobs that have been run, we can run the following command:
bacalhau --api-host=<YOUR REQUESTER NODE IP ADDRESS> job describe <JOB ID RETURNED BY BACALHAU CLI>
Cleaning up
If you've finished with your Spot Instances, you can delete the instances and associated resources by running:
python3 main.py destroyConclusion
In summary, combining Bacalhau and AWS Spot Instances offers a powerful way to run your distributed computing tasks at scale without incurring huge costs. Bacalhau’s resilience ensures that even if Spot Instances are reclaimed, your jobs will still complete successfully.
How to Get Involved
We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.
Commercial Support
While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. You can read more about the difference between open-source Bacalhau and commercially supported Bacalhau in our FAQ. If you would like to use our pre-built binaries and receive commercial support, please contact us!
Onwards together!
The Bacalhau Team.
