This article is based on reporting originally published at searchitoperations.techtarget.com.
Why Continuous Delivery Matters for Microservices Teams
When a development team ships a handful of releases per month from a single monolithic codebase, a collection of hand-crafted scripts can hold everything together. Scale that to dozens of microservices being updated many times per day, and the same approach collapses quickly.
That is exactly the situation Ibotta, a Denver-based mobile rewards app provider, faced as it migrated from a monolithic application to a microservices architecture. At peak velocity the team was pushing as many as 45 code updates in a single day. Their existing script-based deployment process could not keep up, and the risk of a bad release reaching all users simultaneously was unacceptably high. Adopting a formal continuous delivery (CD) pipeline gave the team a repeatable, auditable path to production and a clear strategy for limiting the blast radius of any problematic change, according to Scott Bassin, engineering director at Ibotta.
Continuous Pipelines Rely on Orchestration, Not Just Automation
A common mistake when building a CD pipeline is treating it as a simple automation exercise — replacing manual clicks with scripts. In practice, continuous delivery requires orchestrating a chain of distinct tools: version control, build systems, test frameworks, secrets managers, artifact registries, deployment engines, logging platforms, and application performance monitoring (APM) tools.
“Often, you’re walking into a setup of tools that are half-tied into each other with funky scripts,” said Brandon Carroll, director of transformation, DevOps and cloud services at TEKsystems Inc. The goal is not to find a single omnipotent tool; it is to establish a coherent workflow that spans all the tools your team already relies on.
Ibotta already had continuous integration (CI) in place, giving developers fast feedback on code quality through automated testing and version control. When they set out to extend into CD, their criteria included minimal setup complexity and out-of-the-box integration with their existing validation, logging, and APM tooling. After evaluating options including Jenkins and AWS CodeDeploy, they implemented a CD engine called Harness, deployed as a service.
Steve Burton, DevOps evangelist at Harness, describes CI and CD as fundamentally different disciplines. A pipeline engine — whether gated for controlled delivery or fully open for continuous deployment — must handle far more than moving a build artifact from A to B. It needs to push code across heterogeneous infrastructure, verify deployments, roll back failed releases, manage secrets, and produce a full audit trail. These operational concerns are easy to underestimate when stitching together a homegrown pipeline.
Creating a Safe Deployment Environment
Automated testing within a CI pipeline is essential, but no test environment is a perfect replica of production. You cannot reproduce production traffic volumes, and you cannot legally or safely inject real customer data into a traditional staging setup.
“Things test fine and pass to production, then they go live and fall over,” said James Freeman, head of professional services at Quru, a consultancy specialising in open source technologies. Speaking at AnsibleFest 2018 in Austin, Texas, Freeman stressed that good process behind deployments is not optional — it is what bridges the gap that testing cannot fill.
Blue/Green Deployment
Blue/green deployment maintains two identical environments — one live (blue), one idle (green). A new release is deployed to the idle environment and tested before traffic is switched over. If the new version has a problem, traffic is routed straight back to the previous environment without rebuilding anything.
- Strengths: Fast rollback, reliable, well-suited to back-end services.
- Trade-offs: Resource-intensive (you maintain two full environments at all times), changes are exposed to all users simultaneously at switchover, and tight configuration management is required to keep both environments in sync.
Ibotta currently uses blue/green deployment to manage the high volume of microservices updates, with the blue/green changeover acting as the gate between test and production. As the microservices estate grows, Bassin plans to configure the CI pipeline to initiate production deployments automatically.
Canary Releases
Canary deployment takes its name from the practice of using a canary in a coal mine as an early warning system. Instead of switching all traffic at once, a small slice of users receives the new version first. Engineers monitor error rates, latency, and other key signals before gradually rolling the change out to everyone.
- Strengths: Limits the blast radius of a bad release to a small user cohort, gives engineers real production signal before full rollout, well-suited to front-end UX changes.
- Trade-offs: More complex to implement and monitor than blue/green; requires robust traffic-splitting and observability tooling.
In a project with UK retailer Sports Direct, Quru used automated Ansible playbooks to upgrade 60 nodes — but deliberately started with a single node to validate that the process and configuration were correct. Only after that canary node proved healthy did the team proceed with the remaining 59 nodes. This incremental approach provided confidence without requiring a full staging environment.
Ibotta plans to move from blue/green to canary releases for its microservices. The motivation is speed: canary rollouts can reach production faster than blue/green because they do not require a full staging switchover, and the smaller blast radius reduces risk in a microservices environment where each service is independently deployable.
Choosing the Right Deployment Pattern for Your Pipeline
Neither blue/green nor canary deployment is universally superior. Carroll recommends that teams model their entire operations toolchain first, then select the deployment pattern that best fits their infrastructure constraints, risk tolerance, and release cadence. From there, the CD pipeline should be iterated incrementally — adding automated tests, post-deployment monitoring, and feedback loops — until the team achieves its target cycle time.
| Pattern | Best Suited For | Rollback Speed | Resource Cost | Blast Radius |
|---|---|---|---|---|
| Blue/Green | Back-end services, database migrations | Very fast (traffic switch) | High (double infrastructure) | All users at switchover |
| Canary | Front-end features, UX changes | Fast (reroute traffic) | Moderate | Small subset of users |
Plan for Failure Before You Need To
Even with rigorous pre-production testing and a well-orchestrated pipeline, failures in production are inevitable. Unexpected edge cases, third-party dependency changes, and traffic patterns that no test suite anticipated will surface in live environments. A continuous delivery or deployment strategy is only as strong as its plan for detecting, containing, and recovering from those failures.
Key practices to build in from the start include:
- Automated rollback triggers: Define error-rate or latency thresholds that automatically revert a deployment without human intervention.
- Comprehensive observability: Logs, metrics, and distributed traces should be in place before you need them to diagnose an incident.
- Auditability: Every deployment should be traceable — who triggered it, what changed, and what the pipeline did in response to any anomaly.
- Runbooks for common failure modes: Document the manual recovery steps your team would take if automation fails, so on-call engineers are never starting from scratch.
Growing into a continuous delivery pipeline is less a one-time project and more an ongoing engineering discipline. The tooling, deployment patterns, and monitoring strategies should evolve alongside the architecture and the team’s confidence in the process.