The plan to unclog the build pipeline

Our Journey to Continuous Delivery: Chapter 1 of 4

It Begins with a Road.

CI/CD is the road from Development to Production

There are many definitions out there, but let’s keep it simple — Continuous Integration/Continuous Deployment, or CI/CD, is the path to transport a code change from a development environment to a production environment. The engineers within our technology organization drive vehicles on this metaphorical road every day, as such, we want that road to be safe and fast; our business depends on it.

In this series of articles we will cruise through our journey toward a better CI/CD over the years. This includes the challenges we faced, the tools we used, but also the mistakes we made. Along the way we will open source some of our code in the form of working examples on GitHub; referencing them where appropriate. So, let us begin by…

Building the first Road

At first, we had a few projects, mostly small. We built a road from development to production, and our application builds would take that road. This was a small, quiet road with low traffic.

In our case, we already had a data center, our policy was to host our applications on it. We decided to host a Jenkins server.

Road to production: first months old

Dealing with traffic

As an increasing number of projects integrated with production, we realized this simple road wasn’t enough and began to maintain the road and add a center line. The projects became bigger generating more code, more tests, more verification.

Road to production: year 1

As traffic increased, some vehicles had to wait before getting on the road. We began to use traffic lights to control the flow. However, this caused other problems.

Some of our engineers had to wait longer for their builds to complete. At busy hours, for example before lunch or EOD, there was a traffic jam.

Road to production: year 2

Upgrading the road

Our code base grew as time went on; the result was that our projects were bigger and their tests became more complex. For example, some integration tests required external services, such as Database Servers or Configuration Management Servers. These builds are represented by the trailer trucks.

We upgraded our infrastructure with a plan to add another lane. However, by the time the new lane was ready, the traffic was already clogged up.

Road to production: year 3

This became even more of a burden as we embraced a micro-services architecture causing the number of repos and builds to increase dramatically. The road was not designed for such a high volume of traffic, accidents would happen: I/O errors, server crashes, etc. They became very expensive as we would have to send metaphorical emergency vehicles and tow-trucks get things flowing again. This would also impact other drivers and there was a lot of work to setup and maintain. Not to mention, a few terrible accidents where a lot of people were stuck and frustrated. Soon, engineers started to complain that builds were slow, but we had to cope with it as this was the only road we had to get to production; c’est la vie.

Road to production: year 5

Changing Transportation Mode

At this stage, we knew we had to fix this congestion problem, but we didn’t want to just add another lane this time.

That’s when we heard this story about Malcolm P. McLean, a truck driver from North Carolina. He proposed a more efficient way to transport cargo, using containerized cargo. It was the spark we needed. Quickly adopting this principle, we began to redesign our road to production. Instead of having different size and shape vehicles traverse the road, we will use containers. Every driver will bring a container and the container will be loaded on a train to production. Our goal was to have a continuous flow, removing the traffic and associated maintenance.

This is when Ticketfly decided to use Docker for our build system as it provides an additional layer of abstraction and automation.

Road to production, new way of transportation

Follow along as we walk through how we implemented the re-envisioned road to production at Ticketfly with the following goals:

  • Automate: zero maintenance, and end to end automation.
  • Reduce Build Time: we have a 10 minutes goal.
  • Growth: scalable architecture, on the cloud.
  • Engineering Satisfaction: self service, easy to use.
Trends observed CI/CD over time. Goals on the right side

Check the continuation of our story in the next chapter, until then go straight to examples on GitHub.




Opta — open source Infra-as-Code

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

HPE strategist Mark Linesch on the surging role of containers in advancing the hybrid IT estate

Matrix Labs Monthly Update (March 2022)

Why we chose Rust over Lua for our project?

Continuous Release Pipeline with Travis-CI

Get the latest file from Azure Data Lake in Databricks

Feature Monkey — Journey so far 🎉

Replay transactions in Remix IDE

OpenStack , Simplified!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Remy DeWolf

Remy DeWolf

Opta — open source Infra-as-Code

More from Medium

Blue-green deployments with Materialize

Using Auto-Scaling and Load Balancing for High Availability

Add IAM Users to AWS-AUTH config to access EKS Cluster

Hadi Live DevOps Case Study