The Expanso Football - A Distributed Compute Cluster in a Briefcase
The Expanso Football - A Distributed Compute Cluster in a Briefcase
(11:39) We built a compute cluster into a briefcase for TechCrunch Disrupt - and we did it in 36 hours
(This post is long one with lots of pictures, and if you’re reading this in your email - almost certainly truncated by your email client. Get the whole post here.)
We attend a lot of conferences as part of spreading the good word about Bacalhau, and as you may have noticed, conference season is in full swing.
For those of you who’ve never been on the exhibitor side of things at a conference, there are a couple of things that are a given. At a minimum you’ll have a small table to display demos and wares, a big-ish screen to show off your products, and (if the budget stretches) a roll-out banner with your company branding.
Here’s the thing… we all like to think our companies, products and branding are the bee’s knees, but when you’re in a conference hall with thousands of other vendors, it can be hard to stand out no matter how unique or exciting your offering is.
We like a challenge here at Expanso, and we always enjoy the opportunity to show-off Bacalhau in new and exciting ways, so we set ourselves the task of bringing Distributed Computing to life on the floor of TechCrunch Disrupt.
A Brief History of Footballs
After a little back and forth, we decided that we wanted to bring something to the floor that would let people interact with Bacalhau in a practical way. Distributed Computing is a subject with a whole bunch of abstract concepts and connections that people have to keep in their heads. With a little bit of time these concepts become second nature, but on the conference floor, you’ve got something like 30 seconds (max!) to bring people up to speed before you can have a really good conversation.
That’s not a great deal of time at all! And with so much to keep in one’s head we thought it would be helpful to have something physical that people could see and play with. But what form could that take?
Well… I happen to know what the Nuclear Football is. To save you a click, it’s the US President’s mobile command center for reacting to a nuclear strike. It’s terrifying that such a thing exists, but world-ending capabilities aside, it’s a cool, impressive piece of technology. And perfect for taking a whole bunch of compute capacity with you to various conferences and venues.
So, after a Slack message or two, David gave the thumbs up and my task was set: I was going to build the Expanso Football.
The Build
So, I had a little under 3 days, and one of those is going to be spent in the air flying to San Francisco from London - if the Football was going to be ready in time, I had to do some creative shopping. Normally, you’d order all of the parts online, but even with Amazon Prime, it wasn’t possible to get everything on this short a notice.
For this to work, we’d need at a minimum:
A briefcase
A group of small computers
A thin, light, but big monitor
Some kind of portable network router/switch
A whole bunch of cabling
Something to hold it all together!
Friday - To Cambridge!
First I started with the computers. If you’re looking to fit a bunch of computers into something as small as a briefcase, and not need a huge power supply to power the things, you’re probably going to go for a group of Raspberry Pis - and so we did!
But where can you buy a bunch of Pis on short notice? Well, the Raspberry Pi Store in Cambridge, UK, of course! And I live in London which is just a stones throw away, so first thing Friday morning, I hopped on a train. Within a few hours I had a veritable bounty of small, powerful, Linux-powered computers sitting with me on the train back to London.
The next obvious thing to grab was the briefcase - however, the noble briefcase doesn’t quite have the same standing in society that it once did. Whereas getting something functional and inexpensive might once have been as simple as walking down the high street and picking out from a selection at a bag shop, retail isn’t what it used to be. And again, we didn’t have time to wait on Amazon, so this would have to be a problem for the weekend.
Saturday - LHR to SFO
There’s not much you can do in the air when planning to build a compute cluster into a briefcase. For one, there’s not a great deal of space, and for two, I imagine passengers might get a little nervous when one of their fellow travelers has a bunch of electronics and loose wires hanging around. So there wasn’t much else to do but sit back and enjoy the inflight entertainment - for 10 hours…
That’s a huge chunk of time to lose when you’re working to a 48 hour deadline, but there’s nothing you can do about it except hit the ground running when you land. Where most people might disembark from their flight, head to their hotel and kick back for the evening trying to stave off jetlag, I instead set about getting additional materials from anywhere and everywhere I could in the Bay Area.
My main goal for the evening was to find any kind of briefcase that fit the bill for our needs. It needed to be reasonably deep so we could fit everything in, with a lid that could fit in a decent sized screen and handles so that I could still carry the thing.
After a bit of Googling and an Uber trip I ended up at a Staples store just outside of Burlingame, CA. This wasn't my first port of call, but various other hardware stores didn’t quite have what I was looking for, so it was more out of desperation than anything else. After a bit of a wander around the store THERE IT WAS - the perfect case! Big enough to fit everything in, small enough to carry around, and deep enough to fit a screen into it.
With the briefcase secured, I retired for the evening. I didn’t know it yet, but I’d be lying if I said I didn’t suspect that it may be a while before I got a full night’s sleep again.
Sunday - 70 miles of Uber, A trolley of tools, and the Ultimate Rapid Prototype
After a full and deep sleep it was now the day before Techcrunch, which means there were 24 hours to:
Get all of the remaining parts
Get a bunch of tools
Build the the thing
Profit(?)
I had the computers and the box to put them in, but literally nothing else at this point. So began a 4 hour odyssey around the Bay Area in more Ubers than I could ever recall to gather all of the remaining parts.
Here’s looking at you!
First up, the screen. My first thought was to try and get something off the shelf from a Target, pop off the case and then affix it with a plate to the inside of the briefcase lid. That’s a lot of work, and frankly I wasn’t confident that it would even work. Luckily, in my explorations of what was available, Best Buy had a whole range of “Travel Screens”. This is a class of monitor that I wasn’t familiar with, but in short, they’re screens which have the form factor of a laptop screen, but bigger, without the laptop, and powered over USB-C. Perfect for our build!
Acquiring Developer Tools
So at this point, I had everything I needed to build a functional cluster of computers and be able to interact with them. Now came the tricky part: Developer Tools.
And by “Developer Tools” I mean hack saws, glue guns, nuts, bolts, and foamboards that I, as a Developer, would be using to build this thing. This was actually fairly straightforward, I quickly learned that Lowe’s is one of the go-to places to buy a bunch of hardware stuff nearby, and quickly headed down to fill up a trolley of everything I thought I might need.
Getting the equipment was easy enough, but getting it into the hotel definitely earned me some strange looks!
Wires and SD Cards
Once I had all of the tools together, I decided to do a quick audit of everything I had and, to my horror, I realised that I hadn’t purchased SD cards or plugs for the Raspberry Pis! There’s no point having all of this stuff if you can’t switch it on and boot it!
Mercifully, Central Computers in the San Francisco Downtown was open and had pretty much everything you could need for a bunch of Pi. After a quick stop there I was good to get going. As a bonus, it was right next to Moscone Center where TechCrunch Disrupt was being held the next day, so at least I knew where I’d be bringing everything!
Putting it all together
Now I had everything I needed to start assembling the Football. Well… everything except time. At this point there was 18 hours to put everything together and get it “show ready”. It was also a Sunday, so if there was anything missing, there wouldn’t be the opportunity to pick up anything else from the stores.
With that in mind, I decided to lay everything out and make sure it would all fit in the case, as well as start shaping some of the padding that would hold everything together.
Voila! It fits! Not the prettiest thing, but I was on my way.
Monday - (Tech)Crunch Time
Configuring Bacalhau
I’ve learned the hard way in my career that if you’re planning to glue something down - especially a computer - you should check that it works as expected first. So, before I went any further putting together the enclosure, I decided to spend some time configuring the Raspberry Pis and getting Bacalhau configured.
Of the four Raspberry Pi I had to put inside the briefcase, one was going to be the orchestrator node responsible for managing access and distributing jobs, and three were going to be compute nodes where workloads could be executed.
At this point, it had passed midnight on the day of TechCrunch Disrupt kicking off, so there was less than 12 hours before this thing had to be spun up and functional for demos. We’ve worked really hard to make Bacalhau super-simple and easy to configure and that work clearly paid off as it took only 20 minutes to get every Pi working together as a Bacalhau cluster.
Part of what made this so easy was being able to set a known hostname for the orchestrator node using the Raspberry Pi imager. The other part that makes this so easy is how adaptive and reliable NATS is as a communication protocol, and it’s built right into the heart of Bacalhau.
The Assembly
Now that I knew everything was working together, it was time to commit and start cutting and gluing things to other things.
When I first started this project, I thought that cutting through foam to create a mould that could hold everything in place would be one of the simpler aspects of this project, but it. was. hard.
The foam was just a litttttllllllee bit thicker than my craft knife, which meant that each slice would have to be done from both sides, and there was no guarantee that they would be lined up. There was also the bonus issue that the foam I’d picked up at Lowe’s was incredibly brittle, and as I sliced more and more of it away, the probability that the whole thing would snap or crumble grew. Thankfully, that didn’t come to pass.
All in all, cutting everything out and gluing it together took 4 hours(!) and left something of a mess. Don’t worry, I cleaned up the hotel room before I left - I’m not a monster! At 06:00 on the day of the event, most everything was together - all that remained was to permanently fit the screen to the lid of the briefcase.
Everything in the Football is fairly light and well-held in place by foam and glue. The screen, however, has a little bit of weight, and nothing to support it against its continuous struggle with gravity. Fortunately, there’s nothing in the world that enough Gorilla Glue can’t hold in place.
And after a little bit of patience, the whole thing was assembled - and it didn’t look half bad! At this point it was 07:30 in the morning, so there wasn’t any time to install some killer LEDs that I had bought. But, what did have was a fully functional Bacalau-powered Football with 16 ARM cores able to run Bacalhau jobs no matter where it was in the world. It was Go-time!
The Unveiling
Part of the motivation for building the Football was TechCrunch Disrupt were running a “Coolest Booth” competition, and if an open-source compute cluster packed into a briefcase with a hi-resolution screen that can be accessed from anywhere on the planet isn’t cool, then I don’t know what is. The jury is (literally) still out on that, but we have our fingers crossed.
Once the Football was ready, it was time to get it to the event. When closed up, two very important things happened:
Everything stayed in place!
It looked pretty cool, even though you couldn’t see the technology within.
And at TechCrunch it looked even cooler!
Our original goal with building the Football was to stand-out and attract attention in a room filled with thousands of people all looking to get people excited about their technology - and I can say with great pride that we definitely had that effect. Sure, a slick presentation might capture people’s attention for 5 - 10 seconds, but having a big, black box with a whole bunch of computers and a bright screen gleaming out into the hallway captured something else: Curiosity.
Most everyone who passed by our stall made a point of coming up to us and asking “What’s this, then?” to which we would eagerly respond with “It’s a Bacalhau cluster - let me show you what it can do.”
Instead of having an abstract conversation about distributed computing, this or that type of node, and which Cloud platform you could run Bacalhau on, we could show people right before their eyes how everything interlinked and worked together beautifully.
The Expanso Football is one of the best marketing tools I’ve ever had in my career, and again, we built it in 36 hours with a transatlantic flight in the middle!
After TechCrunch, and the Future
Once TechCrunch was finished, it was time to take the Football home. Suffice it to say, despite it being one of the safest hardware projects I’ve ever put together, I didn’t want to risk explaining what this big black box filled with wires and glue was to the TSA, so I mailed it back to the UK.
At the moment, it’s sitting somewhere in Heathrow waiting for the Royal Mail to pick it up and bring it back to me. I‘m very much looking forward to being reunited with it, not least so I can show it off to everyone in London.
As for the future, the Football has been a great conversation starter for us, and we’re gonna build more. We’ve already got a list of features and enhancements that we want to build in when we have more than 36 hours to do it, but of this you can be certain - it’s gonna be a whole lot more purple.
So, until then, we’re gonna keep doing what we do best: Making Bacalhau the best, most versatile piece of software for distributed computing on the planet.
Addendum
I may not have had time to install the LEDs into the Football, but I did get the chance to take this sleep-deprived photo of me wearing them at 05:00 in the morning.
Get Involved!
We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations.
Commercial Support
While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. You can read more about the difference between open-source Bacalhau and commercially supported Bacalhau in our FAQ. If you would like to use our pre-built binaries and receive commercial support, please contact us!
