The plan to unclog the build pipeline

Our Journey to Continuous Delivery: Chapter 1 of 4

4 min readFeb 14, 2017

It Begins with a Road.

CI/CD is the road from Development to Production

There are many definitions out there, but let’s keep it simple — Continuous Integration/Continuous Deployment, or CI/CD, is the path to transport a code change from a development environment to a production environment. The engineers within our technology organization drive vehicles on this metaphorical road every day, as such, we want that road to be safe and fast; our business depends on it.

In this series of articles we will cruise through our journey toward a better CI/CD over the years. This includes the challenges we faced, the tools we used, but also the mistakes we made. Along the way we will open source some of our code in the form of working examples on GitHub; referencing them where appropriate. So, let us begin by…

Building the first Road

At first, we had a few projects, mostly small. We built a road from development to production, and our application builds would take that road. This was a small, quiet road with low traffic.

In our case, we already had a data center, our policy was to host our applications on it. We decided to host a Jenkins server.

Dealing with traffic

As an increasing number of projects integrated with production, we realized this simple road wasn’t enough and began to maintain the road and add a center line. The projects became bigger generating more code, more tests, more verification.

As traffic increased, some vehicles had to wait before getting on the road. We began to use traffic lights to control the flow. However, this caused other problems.

Some of our engineers had to wait longer for their builds to complete. At busy hours, for example before lunch or EOD, there was a traffic jam.

Upgrading the road

Our code base grew as time went on; the result was that our projects were bigger and their tests became more complex. For example, some integration tests required external services, such as Database Servers or Configuration Management Servers. These builds are represented by the trailer trucks.

We upgraded our infrastructure with a plan to add another lane. However, by the time the new lane was ready, the traffic was already clogged up.

This became even more of a burden as we embraced a micro-services architecture causing the number of repos and builds to increase dramatically. The road was not designed for such a high volume of traffic, accidents would happen: I/O errors, server crashes, etc. They became very expensive as we would have to send metaphorical emergency vehicles and tow-trucks get things flowing again. This would also impact other drivers and there was a lot of work to setup and maintain. Not to mention, a few terrible accidents where a lot of people were stuck and frustrated. Soon, engineers started to complain that builds were slow, but we had to cope with it as this was the only road we had to get to production; c’est la vie.

Changing Transportation Mode

At this stage, we knew we had to fix this congestion problem, but we didn’t want to just add another lane this time.

That’s when we heard this story about Malcolm P. McLean, a truck driver from North Carolina. He proposed a more efficient way to transport cargo, using containerized cargo. It was the spark we needed. Quickly adopting this principle, we began to redesign our road to production. Instead of having different size and shape vehicles traverse the road, we will use containers. Every driver will bring a container and the container will be loaded on a train to production. Our goal was to have a continuous flow, removing the traffic and associated maintenance.

This is when Ticketfly decided to use Docker for our build system as it provides an additional layer of abstraction and automation.