Part of our disaster mitigation plan has included developing a rapid server deployment process. In addition to saving us a lot of time when launching a new server, rapid deployment ensures that existing servers stay up-to-date with the latest configuration changes and software releases. It also means that when we need to expand our services, populate a new data center or CDN, or recover from a crash, we can be on top of things quickly. Our servers are located in over 40 data centers and cloud providers worldwide and collect over a billion-and-a-half measurements every day. These measurements are crucial to the decision-making power of Openmix.
Deploying on Ubuntu with Puppet
Our deployment plan has several steps. First, we begin with the latest Ubuntu Long Term Support (LTS) release available. Each of our service providers already supports Ubuntu LTS, typically by providing a pre-built virtual machine image. Their configurations can vary from provider to provider, so we have built specialized bootstrap shell scripts that we can use to deploy our own very basic configuration. These scripts are customized by both cloud provider and topographic location of the server. Much of our configuration is based around the hostname of the new server, which the bootstrap script sets based on the location and service provider of the new host.
The bootstrap process also installs and configures Puppet Enterprise from Puppet Labs. This tool provides a configuration management system that allows us to specify the state we want each server to be in. Everything from specific applications to firewall configurations can be defined in Puppet modules, which are gathered into site manifests. The granularity is incredible—even the permissions and ownership of individual file can be specified.
Using Puppet, we are then able to make sure that the new server is working the way we want it to. We know, even before deploying any Cedexis software, that the server is in a known state and running smoothly. We’re also able to keep an eye on hundreds of servers across dozens of data centers at once using Puppet, which makes it an invaluable tool. We love using Puppet, and if you’d like more information about how and why we selected it, you should read our Puppet Labs case study.
Installing the Cedexis Tools
The next step is to use Puppet to add the Cedexis apt repositories to the new server’s configuration. We prepare our software releases as Debian packages, which gives us all of the power of dpkg, the Debian package management system. This allows us to specify dependencies, track software versions, and automatically deploy new packages and revisions. Once the server is aimed at our apt repositories, installing the Cedexis tools is quick and painless.
The new server is just about ready. Because Openmix makes decisions using only very recent measurements, only a small amount of data needs to be transferred to the new server before it can begin making intelligent decisions and joining the rest of our workforce.
We monitor with both Puppet and our own monitoring software. If problems develop, we notice them quickly. From beginning to end, the process of deploying a new server takes about an hour, and the tools we have chosen give us the confidence that each new server is playing nicely with the Cedexis network and doing its job appropriately.