Continuous Deployment Using Jenkins: A Real-World Pipeline at Scale

This article is based on a case study originally published by Thumbtack Engineering (thumbtack.com), describing their continuous deployment journey starting in late 2015.

Background: Why Thumbtack Needed a Better Deployment Pipeline

When the author joined Thumbtack in late 2015, the engineering team already had a continuous delivery infrastructure in place for monolith builds. As the team grew, a recurring problem emerged: a significant amount of engineering time was being consumed by the act of deploying the latest build. Deploys were getting larger — so-called “train deploys” — and rollbacks were becoming increasingly difficult to execute cleanly.

These symptoms pointed to a clear need: invest in a proper, automated deployment pipeline that reduces manual overhead, limits blast radius, and gives engineers clear control at every stage.

Continuous Delivery: The Foundation

The existing continuous delivery setup at Thumbtack used Gerrit for code review and as the Git repository host. The pipeline worked as follows:

1. An engineer submits a Code Review in Gerrit.
2. Jenkins automatically triggers a build.
3. Source code is fetched from Gerrit’s Git repository.
4. Unit tests are executed. If any fail, the build is aborted and discarded.
5. If unit tests pass, integration tests are triggered. Again, failure discards the build.
6. A build that passes all tests is marked as safe to deploy and becomes available for promotion.

This gate-based approach ensured that only validated builds could proceed, but the deployment step itself was still manual and error-prone.

Scoping the Continuous Deployment System

Before building the continuous deployment layer, the team defined clear requirements. These goals shaped every design decision:

• Engineer control: Engineers must be able to manually deploy a specific build version to canary or production at any time.
• Easy stop and rollback: Halting a deployment in progress and rolling back must be straightforward and fast.
• Automated gradual rollout: Traffic should shift incrementally, not all at once, reducing exposure to defects.
• Prevent commit pile-up: No more than 3–4 commits should be in-flight at a time. Smaller batches make it easier to identify which commit introduced a regression.
• Restricted submission windows: Code submission should be blocked during off-hours (9 PM to 9 AM on weekdays, all weekend, and when the office is closed), with an exception for critical bug fixes.

Gerrit Plugin: Enforcing Repository Locks

Previously, repository locking during restricted periods was entirely manual. Engineers were expected to check a dedicated Slack channel for a “lock” signal before submitting code. Locks were placed manually when entering off-hours or during active rollbacks. Predictably, this was unreliable — engineers occasionally submitted code to a locked repository because they missed the Slack message.

To solve this, the team built a custom Gerrit plugin that enforces repository locking at the tooling level:

• The Gerrit UI displays the current lock status of any repository.
• When a repository is locked, Gerrit rejects submissions from any engineer who is not the lock owner.
• Engineers lock and unlock repositories by running a script that issues a Gerrit SSH command — no manual Slack posts required.

Additionally, a Jenkins job was set up to automatically lock the monolith repository during off-hours, removing the need for any engineer to remember to do it manually.

The Continuous Deployment Jenkins Job

With the locking mechanism in place, the team introduced an automated deployment stage into the build pipeline. The deployment job follows a canary-based rollout strategy:

1. Pick up the latest successful build from the CI pipeline.
2. Set canary traffic to 0% — no user traffic reaches the new build yet.
3. Deploy the build to the canary cluster.
4. Gradually increase traffic to the canary cluster over approximately 10 minutes.
5. Once canary reaches 100% of traffic, the production cluster is updated with the same build.

Throughout this process, engineers whose code is included in the rollout receive Slack notifications in the deploy channel. If error rates spike or an alert fires, a notification is posted to the same channel so the responsible engineer can react immediately.

If an engineer detects a problem — via an automated alert, error logs, or metric dashboards — they can acquire the repository lock, which interrupts the deployment. From there, the engineer decides next steps: typically shifting all traffic off the canary cluster, reverting the faulty commit, and unlocking the repository once a clean build is verified.

Key Design Principles Worth Adopting

The Thumbtack approach illustrates several principles that apply broadly to any team building a continuous deployment pipeline:

• Enforce policies in tooling, not documentation. Relying on engineers to check Slack before submitting code will fail. Encoding the lock in Gerrit makes the policy impossible to accidentally bypass.
• Canary deployments reduce blast radius. Routing a fraction of traffic to the new build before a full rollout limits the impact of any defect that slipped through testing.
• Small, frequent deploys are safer than large batched ones. Limiting in-flight commits to 3–4 at a time makes root cause analysis significantly faster.
• Deploy visibility keeps everyone informed. Real-time Slack notifications tie the deployment directly to the engineers responsible for the changes, enabling faster human intervention.
• Manual override must always be possible. Automation should accelerate the happy path, but engineers need clear, fast controls to stop or redirect a deployment when something goes wrong.

Known Limitations and Future Work

After running this system in production for over three months, the team identified two areas for improvement:

• Automated alert-triggered deployment halts: The team wanted alerts to automatically stop rollouts, but needed to solve the false-positive problem — for example, a transient network blip should not abort an otherwise healthy deploy.
• Reducing time from commit to canary: At the time of writing, sequential monolith builds meant it could take over 20 minutes from code submission to a canary deployment starting. Parallelizing build steps was a planned improvement to shrink this window.

Both challenges are common in CI/CD pipelines at scale: balancing automation confidence against alert reliability, and optimizing build times without sacrificing test coverage.

Summary

Thumbtack’s continuous deployment pipeline demonstrates a pragmatic, incremental approach: build on existing CI infrastructure, enforce deployment policies in the tools engineers already use, adopt canary rollouts to limit risk, and keep humans in the loop with clear notifications and manual override capabilities. The combination of Gerrit, Jenkins, and Slack notifications — with a custom repository locking plugin — produced a system that reduced deploy overhead and improved deployment safety without removing engineer agency.