Microservices architecture promises a lot: decoupled components, independent deployment cycles, and the ability to scale individual services on demand. In practice, however, many teams discover that the operational complexity of running CI/CD at microservices scale can quickly become a bottleneck in its own right.
This article draws on a talk by Dan Garfield, chief technology evangelist at Codefresh, delivered in episode five of the DevNetwork Dev Professional Series, and summarised on devops.com. Garfield walked through real-world pitfalls and practical strategies for scaling microservices delivery pipelines without losing your mind—or your budget.
Why Microservices? The Case Against the Monolith
To illustrate why tightly coupled, monolithic architectures create problems, Garfield used an analogy from the Apollo 13 mission. Apollo 13 flew with two separate modules—the command module and the lunar module—each built by different engineering teams with different design conventions. When the command module suffered an oxygen failure, the crew needed to repurpose oxygen filters from the lunar module. The filters were incompatible, forcing an improvised, high-stakes retrofit.
The analogy maps directly to software: monolithic teams build conventions that work internally but are incompatible with anything outside. Reuse becomes nearly impossible, code value decreases, and development effort increases.
“Being able to reuse components and scale independently is the benefit of microservices.”
— Dan Garfield, Chief Technology Evangelist, Codefresh
Expedia’s Microservices Journey: A Real-World Case Study
Garfield used Expedia as a concrete example of an organisation that moved from monolithic services to discrete microservices—one for car search, one for car sorting, one for bookings, one for airline search, and so on. Breaking the monolith improved elasticity and allowed individual services to be deployed and scaled independently.
However, Expedia’s microservices journey also surfaced a class of problems that the team had not fully planned for: how to manage CI/CD at scale.
Common Problems When CI/CD Meets Microservices at Scale
Most teams begin with one CI/CD pipeline per microservice. That approach works at ten services. It becomes unmanageable at hundreds or thousands. Garfield summarised the key failure modes:
1. Pipeline Sprawl
Each microservice gets its own pipeline, each pipeline connects to its own Git repository, and some services need separate pipelines for production, staging, and testing. The result is an enormous surface area to maintain, where a single policy change—say, adding a security scan step—must be manually applied across hundreds of pipeline definitions.
2. Network and Infrastructure Costs
Running full validation and integration test suites across thousands of separate pipelines consumes substantial network bandwidth and compute. The cost—both financial and in build time—can grow faster than the engineering team anticipates.
3. Shared Library Fragmentation
A common instinct is to consolidate common pipeline logic into shared libraries. But shared libraries introduce version-specific dependency conflicts. Expedia developers using Jenkins, for example, were forced to manually copy-paste pipeline code from one definition to the next. Different teams adopted different conventions, leading to shadow IT and inconsistent delivery practices.
“The scale of automation itself is where most people have the biggest bottlenecks.”
— Dan Garfield
Three Strategies for Scaling Microservices CI/CD
1. Use Container-Based Pipelines
Rather than relying on shared libraries baked into the CI server, Garfield recommended building each pipeline step as a container image. Each image is self-contained, carries its own runtime and dependencies, and can be versioned and pulled independently.
The practical advantages include:
- Language version isolation: Teams can run different language runtimes in the same pipeline without conflicts.
- Self-service: Developers pull the image version they need without waiting for a platform team to update a shared library.
- Reusable step images: Common CI tasks—cloning a repository, building a Docker image, running unit tests, linting, or executing a security scan—can be published as standardised images on a registry such as Docker Hub and reused across every pipeline without duplication.
This approach makes the pipeline itself composable in the same way that microservices make the application composable.
2. Consolidate to a Single Context-Aware Pipeline
Instead of maintaining a separate pipeline definition for every microservice, Garfield proposed converging to a single, parameterised pipeline that adapts its behaviour based on the context provided at trigger time.
The mechanism works as follows:
- Triggers carry metadata: When a pipeline run is initiated—by a Git commit, a push to an image registry, a scheduled job, or a pull request event—the trigger passes along contextual information: which repository, which branch, which service, which environment.
- The pipeline adapts: Downstream steps read that context to decide what to clone, which dependencies to fetch, which tests to run, and where to deploy.
- Filtering and observability: A single pipeline surface gives operators a unified view of commits and changes across all services, making it easier to audit and troubleshoot.
Garfield demonstrated this pattern within the Codefresh platform, which provides native Kubernetes integrations and a trigger model designed for exactly this use case. The trade-off is a modest reduction in per-service pipeline customisation, but the operational simplicity gained typically outweighs the constraint.
3. Adopt a Canary Release Strategy
When an organisation operates hundreds or thousands of microservices, spinning up full integration test environments for every service on every commit becomes cost-prohibitive. Garfield acknowledged that at sufficient scale, “testing early becomes less useful as infrastructure complexity arises.”
A canary release strategy addresses this by routing a small percentage of real production traffic to the new version of a service before a full rollout. Key benefits include:
- Reduced blast radius: If the new version has a defect, only a small fraction of users are affected.
- Real-world signal: Canary traffic exercises the service under actual production conditions—load patterns, data shapes, dependent services—that synthetic tests often miss.
- Faster feedback loops: Teams can validate behaviour quickly without waiting for heavyweight integration environments to spin up.
Canary releases work particularly well when paired with feature flags and automated rollback policies based on error-rate or latency thresholds.
The Core Insight: Make CI/CD Reusable, Just Like Microservices
The thread running through all three strategies is the same principle that makes microservices valuable in the first place: reusability through modularity. Containerised pipeline steps, a single parameterised pipeline, and progressive delivery techniques all reduce duplication and make the delivery system easier to reason about as the service count grows.
If your organisation is hitting the limits of one-pipeline-per-service, the starting point is not to add more pipelines—it is to redesign the pipeline architecture with the same discipline you applied to the application architecture.
Key Takeaways
- Pipeline sprawl is one of the most common and underestimated costs of a microservices migration.
- Shared libraries create version-conflict problems; container-based pipeline steps are a more maintainable alternative.
- A single, context-aware pipeline driven by rich trigger metadata can replace thousands of near-identical pipeline definitions.
- Canary releases help manage testing costs and risk as service count grows.
- The goal is to make your CI/CD system as modular and reusable as the microservices it deploys.