Automate your configuration with Jenkins DSL

Our Journey to Continuous Delivery: Chapter 3 of 4

At Ticketfly, we have approximately 50 active Git repos: backend applications, micro-services, front-end services, plugins. We follow a dev to master Github work-flow. For each project, we have a few active branches, to which Pull Requests are created.

We want our system to build and run tests at each stage. Some projects have been broken down to speed up build time. This means that instead of having one long running job, it would generate four jobs executed in parallel (using multi-job plugin). This means more jobs to generate overall.

For an organization like Ticketfly, it translates into having a few hundred jobs to configure in Jenkins. If we individually manage each job, it would:

• Create a bottleneck for engineers when they need to update their CI.
• Increase the risk of error associated with manual operations.
• Prevent our dev-tool engineers to work on other projects.
• Encourage inconsistency over time and a culture of, “if it ain’t broke, don’t fix it.”

At this scale, manual configuration is not an option.

Thankfully, there is a way to automate job configuration: the job dsl plugin. To illustrate the benefits of the DSL, we will review two examples of real use-cases and provide some insights on what we have done at Ticketfly.

What is Jenkins DSL?

Provides a structure to describe jobs using a Groovy-based language instead of XML. Some of the highlights are:

• Programmatically create jobs based on some custom rules.
• Batch update your managed jobs.
• Facilitate core and plugin upgrades.

Example 1: Introduction to Jenkins DSL

We begin with an example of a job generated with the DSL for a gradle project:

Example of DSL to generate a gradle job

You can run this example locally using the following command comparing the output with is in the video below.

docker run --name jenkins -p 8080:8080 -v /var/jenkins_home ticketfly/jenkins-example-job-dsl

For more info, check the source code for this example

Example 2: Automate all your jobs based on your Github repos

Alright, so we’ve gone through an introductory example, but the question remains: How will it scale?

We could define a DSL script for each job and automate this part, but in our experience maintenance would still be high.

Pro Tip: Because DSL is programmatic, you could use external libraries and define common structure for the jobs to be managed.

Before jumping into creating jobs, let’s ask a few questions about your engineering team:

• Who are the users of the build system?
• What is their need?
• When do they need CI?
• Where is the source code for your applications?
• Why do we need build pipeline for?
• How are the projects built? Can we identify commonalities to group by?

For this example, let us imagine that we will setup a CI pipeline to build open source projects hosted on Github. Lets use the questions above:

• Who — our users are developers contributing to open sources projects.
• What — every commit should be tested on the CI pipeline.
• When — any time they commit a change, they would like to get it tested.
• Where — the code is hosted on Github.
• Why — to attract contributors, we want the projects to be stable and be notified if a commit introduce compile errors or test failures.
• How — we will regroup the projects by the build tool used (make, maven, gradle, npm, grunt, sbt…) For this example, we will pick gradle and proceed with automation for all the projects using it. The common pattern is to check out the source and run the wrapper command: ./gradlew build

At this point, we can write a DSL script to:

1. List the Github repos for an organization: we will use the GitHub API for Java.
2. Select the projects using gradle.
3. Generate the Jenkins jobs.

This implementation has been packaged into a docker image, it can be run locally:

docker run --name jenkins -p 8080:8080 -v /var/jenkins_home ticketfly/jenkins-example-job-dsl-for-github-org

Below is a video where we ran this example for a few Github organizations with a lot of open source projects: Netflix, Google and Facebook. It found approximately 100 projects and configured them on Jenkins.

For more info, check the source code for this example

Customize the CI flow for Ticketfly

When we started this project, we asked the 5W1H.

• Who: the users are Ticketfly engineers.
• What: they need a reliable CI, they also would like to manage their builds without having to go through some gate keepers. At the organization level, we are at a stage where we need to adopt standards on how projects are built and tested.
• When: any time they commit a change, they would like to get it tested.
• Where: the code is hosted privately on Github.
• Why: if it takes too long to get their code merged or if the CI system is unreliable, it will impact their productivity.
• How: we have identified 3 types of projects that we support based on our tech stack: grails, scala/sbt and ember. We have defined standards and conventions for our projects. Some projects had to be slightly changed to follow the expected standards, but the upside is that it enables the dev team to manage their build configuration without going through a gatekeeper.

From the above, we implemented the following workflow:

An engineer managing the jenkins.json file for the Backstage application

Here is an example of jenkins.json file for a scala project:

jenkins.json for a scala SBT project

• Each Github repo has a jenkins.json file to drive the configuration of the Jenkins jobs.
• If a team needs to change their Jenkins config for a given repo, they would push a change to the jenkins.json. It takes 15 min at most, for Jenkins config to be updated, an admin job is scheduled to scan our Github repos.

Note: We could have used the Pipeline plugin, but we wanted to have a very minimal structure and hide common operations such as authentication, common config files, default tools parameters and versions.

Follow along with this video for a step-by-step update to the Jenkins config via the jenkins.json file:

Update Java version for a Scala/SBT project

We have been using Jenkins DSL for over a year, here is the bottom line:

No more Jenkins maintenance as all our configuration is automated. We have removed ourselves as gate keepers by enabling our teams to set up their build configs from their Github repos.

Now that we have automated all the configuration, how can we scale our infrastructure to handle large traffic spikes? Stay tuned, this will be covered in our next chapter.