Managing TLS Certificates with Bacalhau
(06:16) Simplify TLS certificate deployment across vast infrastructures with Let’s Encrypt and Bacalhau
It’s hard to believe in today’s world of security best practices, end-to-end encryption and 2FA that little over a decade ago, less than half of the traffic over the web was encrypted.
As more and more of our lives went online in the 2010’s, the realization of the value of that data and the need to protect it came along too. But back in 2014, enabling HTTPS on a website wasn’t a simple proposition. For one, there were a whole host of certificate authorities offering paid-for services, and more. For another, the average developer just wasn’t familiar with the intricacies of setting up HTTPS on their infrastructure.
Enter Let’s Encrypt! a not-for-profit organization with a singular goal: Encrypt all of the traffic on the web.
Hey you! Wanna Encrypt Some Web Traffic?
Let’s Encrypt brought a couple of things to the industry that hadn’t really been around before.
All certificates generated were free
Certificate generation was automated
Certificates would expire after 90 days
Points 1 and 2 stand out for themselves: Free certificates that can be generated programmatically would greatly simplify the adoption of HTTPS on the web, but point 3 is a little odd…
Before Let’s Encrypt, SSL (and later TLS) certificates would typically be valid for 1 or more years, which makes sense. If setting up HTTPS on your platform is a pain in the backside, why would you want to do it more frequently than annually? Because… if someone nefarious gets access to that certificate, they could potentially hoover up your traffic and do very naughty things with it!.
The 90 day renewal policy was designed to achieve 2 goals:
Ensure that certificates have a limited shelf-life if compromised
Encourage developers to automate their certificate renewal
I’m not lazy, I just really enjoy doing nothing…
10 years since Let’s Encrypt came onto the scene something close to 100% of traffic over the web is now secured. With over 500,000,000 domains secured with their certificates, Let’s Encrypt is a modern triumph with few equals.
I personally use Let’s Encrypt certificates on every project I build these days - and I have done since early 2015, but I fear I’m about to lower myself in your estimations, dear reader… In all that time, I’ve never once automated my certificate renewals. Why? Because at no point over the last 10 years has the chore of generating a certificate and SCPing it up to a server been more taxing than setting up my various servers to handle renewal.
But my laziness has come to an end, for after all of this time I now build and maintain systems that are more than just the odd Linux server sitting on a fixed IP for a few years. These days, I build platforms, CDNs, applications and more. And while generating the certificates is still simple, SCPing them to dozens of servers individually is more taxing than just setting up a renewal.
And, as you may have guessed from the title of this blog, Bacalhau is super-handy when it comes to getting TLS certificates to where they need to be. So, let’s do it!
The Setup
For this demo, I have 4 Ubuntu 24.04 servers running across Europe:
Each server has Bacalhau installed.
1 is configured as an Orchestrator Node
3 connect to it as Compute Nodes
Each Compute Node in the network is also running an NGINX server configured to only accept connections over HTTPS
Bacalhau has write access to the directory where NGINX reads its TLS certificates from.
With Bacalhau, we’re going to create a job which wraps up a TLS certificate that we’ll generate with Let’s Encrypt and deploys it to every server in our network.
No more lazy SCPing, just one command to deploy certificates to potentially hundreds (or more!) servers in an instant.
Prerequisites
If you want to follow this through yourself, you’ll need:
To have Let’s Encrypt’s certbot installed (instructions)
The Bacalhau CLI installed on your system (instructions)
A TLD you want to use with certbot
You can find full code and helpful scripts for setting up your servers on GitHub.
Obtaining our Certificate
First things first, let’s get ourselves a TLS certificate to use with our domain. We’re going to be using a wildcard certificate so that any of our servers that are configured to be accessible through our TLD can utilise it.
To generate a certificate with certbot, run the following command in your CLI:
sudo certbot certonly --manual --preferred-challenges dns -d "*.<YOUR>.<DOMAIN>"
Note the ‘*’ character at the start of the space where you’ll enter your domain. This tells Let’s Encrypt that we want to generate a wildcard certificate. If you want to use a naked or specific subdomain, you can use those too - but each of the servers you deploy the certificate to must be configured to use that domain.
When generating the certificate, you’ll see something like the above and you’ll need to create a TXT record with your DNS provider to validate that you own the domain you’re generating the certificate for.
Once you’ve done that, you’ll have certificates stored on your system. Now it’s time to use Bacalhau to get them onto all of our servers simultaneously!
🐟 Riding Bacalhau to Get Our Certs on Our Servers 🐟
In order to get our certificates onto our servers, we’re going to create a Bacalhau Job which hoovers up our certificates along with some code which will write the certificates to where they need to go on our server.
If you read the shell scripts that configured our compute nodes/NGINX servers, you may have noticed a particular line of interest where we run Bacalhau on our system.
--config Compute.AllowListedLocalPaths="/etc/nginx/ssl:rw"
With this config line, we’re telling Bacalhau that any job dispatched to this node can read and write to the SSL directory specifically. This gives us a fixed point to write our new certificates to, without granting access to the wider system.
Submitting our Job
Most Bacalhau Jobs are defined in YAML files which tell Bacalhau what to run, how to run it, and what permissions it has available. In this instance, we’re going to use the following job.yaml file:
Much like our previous demo where we used Bacalhau to deploy static assets to multiple servers simultaneously, we’ll use a Containerless Python image to deploy some Python script which encapsulates our certificates and writes them to the appropriate directories across our infrastructure.
Just like last time, we’re going run our Bacalhau job with the following command:
bacalhau job run job.yaml --template-vars "fulltext=$(sudo ./wrap_certs.sh)"
You’ll notice that we’re running the wrap_certs.sh script with sudo, as we need elevated privileges to read the certificates on our systems, but this does not mean that Bacalhau is running as root, or that the Job will have elevated privileges when executed on each of the compute nodes.
Once you run the command, you should see something like this:
If all of the Jobs have successfully executed on each node, then each server should now have a newly configured certificate enabling connections over HTTPS!
That’s It!
And that’s it! A single Bacalhau command and you can configure and secure a practically unlimited number of webservers in just a few minutes.
The journey from manual certificate management to fully automated deployment across distributed systems is a microcosm of how modern DevOps practices are revolutionizing the way we think about infrastructure.
With Let’s Encrypt, securing web traffic has become a default expectation rather than an aspirational goal. Coupling that with Bacalhau allows us to extend this ethos of simplicity and security into infrastructure automation, turning previously tedious tasks like certificate deployment into a seamless, scalable process.
No more juggling SCP commands or logging into multiple servers to update certificates. With Bacalhau, you’re not just managing your infrastructure—you’re empowering it to securely self-maintain.
At Expanso, we believe that security is the foundation of innovation. With Bacalhau developers can operate at scale without compromising on security. By providing controlled, limited access to critical directories while allowing full infrastructure-wide execution, Bacalhau embodies the principle of least privilege - a core tenet of modern security best practices.
So whether you’re running a handful of servers or managing a global infrastructure, the combination of Let’s Encrypt and Bacalhau ensures your systems stay secure, reliable, and ready to adapt to the challenges of tomorrow
Rejoice! The days of managing certificates over SSH are over!
Get Involved!
We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.
Commercial Support
While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. You can read more about the difference between open-source Bacalhau and commercially supported Bacalhau in our FAQ. If you would like to use our pre-built binaries and receive commercial support, please contact us!
