This article is based on a case study originally published on InfoQ, describing the application of Continuous Delivery Indicators at Siemens Healthineers across a distributed organization of 16 software delivery teams.
Key Takeaways
- Continuous Delivery (CD) is now a standard practice in modern software development, but systematically measuring it remains uncommon.
- Continuous Delivery Indicators, introduced by Steve Smith in Measuring Continuous Delivery, quantify CD in terms of stability and speed.
- Adopting CD Indicators encourages teams to view their deployment pipeline as a value stream to be actively optimized.
- CD Indicators provide a process optimization framework with clear stability and speed goals and visible progress tracking.
- Effective rollout requires a near-real-time tooling layer that collects raw data and computes indicators automatically and transparently.
- Treating indicator data as process signals—not people KPIs—is essential to maintaining data integrity.
Introduction: The Case for Data-Driven Development
Software delivery organizations increasingly ship complex systems at high frequency. The three core activities involved are Product Management, Development, and Operations—understood here as activities rather than siloed departments. Each activity demands rapid, well-informed decisions: Product Management decides which features to prioritize; Development focuses on process efficiency; Operations focuses on reliability.
Decisions can be made based on team experience alone, or they can be grounded in data. A data-driven approach produces more objective, transparent outcomes—particularly important as delivery velocity increases and the number of teams grows. Transparency through data reduces the need for time-consuming synchronization meetings and helps align everyone continuously.
This article focuses on how Development activities can be supported by Continuous Delivery Indicators, and how that data enables rapid, defensible prioritization decisions. It draws on the real-world implementation at Siemens Healthineers’ teamplay platform, a globally distributed organization comprising 16 software delivery teams across three countries.
Process Indicators, Not People KPIs
Before introducing any measurement framework, organizations must establish a clear principle: the data collected represents process signals, not evaluations of individual contributors. When metrics are used to assess people, those people are incentivized to game the numbers, destroying data quality and trust.
Leadership must explicitly set this expectation. When teams understand that CD Indicators reveal where the process is struggling—not who is failing—they engage honestly with the data and use it to improve rather than to defend themselves.
What Are Continuous Delivery Indicators?
Once a team knows what to build, the central development question becomes: how do we build it efficiently? The answer lies in analysing the team’s value stream: Code → Build → Deploy. This value stream is visible on the team’s deployment pipeline.
Continuous Delivery Indicators, as defined by Steve Smith in Measuring Continuous Delivery, quantify two dimensions of value-stream health:
- Stability – how reliably value flows through the pipeline without failures.
- Speed – how quickly value flows from commit to deployment.
The Speed definition here extends beyond simple lead time. Steve Smith defines speed as a combination of lead time for a change and the interval between changes—an extension of the lead-time-only framing found in Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim.
All indicators are designed so that lower values signal better performance, making graphs immediately readable: values trending close to the X-axis indicate healthy flow; high values indicate bottlenecks.
Stability Indicators
| Indicator | Components | Good Signal |
|---|---|---|
| Build Stability | Build Failure Rate, Build Failure Recovery Time | Low rate; low recovery time |
| Deployment Stability | Deployment Failure Rate, Deployment Failure Recovery Time | Low rate; low recovery time |
Speed (Throughput) Indicators
| Indicator | Components | Good Signal |
|---|---|---|
| Code Throughput | Master Branch Commit Lead Time, Master Branch Commit Frequency (interval) | Low lead time; low interval between commits |
| Build Throughput | Build Lead Time, Build Frequency (interval) | Low lead time; low interval between builds |
| Deployment Throughput | Deployment Lead Time, Deployment Frequency (interval) | Low lead time; low interval between deployments |
Visualizing these indicators over time gives teams an at-a-glance view of their value stream. Aligned with the Theory of Constraints, teams can identify their largest bottleneck and systematically invest effort to eliminate it before moving to the next constraint.
Reading the Graphs: Bottlenecks vs. Healthy Flow
When CD Indicator graphs are plotted over time:
- Values asymptotic to the X-axis → healthy behaviour, low friction in the value stream.
- Persistently high values → a bottleneck requiring attention.
- Upward trend over time → a worsening problem that should trigger investigation before it reaches crisis level.
This visual model means developers and product owners can engage with pipeline health without needing deep expertise in statistical analysis.
How CD Indicators Change Prioritization Behaviour
Without CD Indicators, teams tend to invest in development process improvements only after major outages or based on the loudest voice in the room. With CD Indicators, investment decisions become evidence-based:
- Test Automation – targeted where build failure rates are high.
- Deployment Automation – applied where deployment failure rates or lead times are elevated.
- TDD / BDD Process Improvement – directed at stages where commit lead times are long.
The three-way prioritization trade-off that product owners must navigate becomes explicit:
- Invest in new features to improve product effectiveness.
- Invest in development efficiency based on CD Indicator data.
- Invest in service reliability to reduce operational risk.
CD Indicators make the development efficiency dimension visible and quantifiable, so it can take its rightful place alongside feature and reliability priorities.
Tooling: Making Indicators Actionable
At Siemens Healthineers, the team developed an in-house proprietary tool to compute and visualize CD Indicators for all 16 development teams. Key design decisions included:
- Data source: Azure DevOps services—Azure Repos for code data, Azure Pipelines for build and deployment data.
- History retention: Default Azure DevOps data retention (30 days) was extended to 365 days to enable meaningful trend analysis.
- Layout: Value Stream Stability on the left; Value Stream Speed on the right, mirroring the indicator framework.
- Automated bottleneck detection: The tool identifies the largest stability and speed bottlenecks and highlights them in red, allowing teams to act without needing to interpret raw data tables.
- Automated suggestions: Contextual guidance for resolving detected bottlenecks is surfaced directly in the tool.
- Slack integration (planned): Monthly value stream snapshots delivered to team Slack channels, with deep links back to the tool for further analysis.
The near-real-time feedback loop the tool provides is essential: a team working heads-down on features has no natural opportunity to reflect on whether their deployment pipeline is slowing them down. The tool surfaces that awareness automatically.
Adoption Experience at Siemens Healthineers
The CD Indicators were rolled out across 16 teams working on teamplay, a global digital healthcare service. Introducing value stream thinking required deliberate change management. Most development teams do not habitually think of their deployment pipeline as an assembly line with measurable throughput and stability—the concept needed explanation and workshops to land effectively.
Teams provided substantial feedback during rollout that shaped the tool’s data collection logic:
- Count only Master Branch builds, not Pull Request builds, for throughput metrics.
- Track Pull Request lead times separately to identify review wait-time bottlenecks.
- Report Deployment Lead Time and Deployment Interval both in aggregate across all environments and broken out per environment.
A common early reaction was positive surprise: teams had never before seen their own value stream visualized in terms of stability and speed. The data revealed dynamics that were invisible during day-to-day development.
Real-World Example: Deployment Stability Improvement
One team’s Deployment Stability Indicator revealed a high Deployment Failure Rate alongside fast Deployment Failure Recovery Times. The combination indicated frequent, easily fixed failures—likely from a fixable configuration or scripting issue rather than fundamental instability. Once the team saw the data, they resolved the root cause within a few days, dramatically improving deployment reliability. The key enabler was simply awareness: the fix itself was straightforward once the problem was visible.
Real-World Example: Interpreting Trend Data
In another team’s drilldown, the Deployment Failure Rate trend showed a steady increase from September onward, reaching nearly 100% recently. Simultaneously, Deployment Failure Recovery Time spiked. This meant the pipeline was staying red almost continuously—a critical bottleneck blocking value from reaching production environments.
In contrast, the same team’s Master Branch Commit Lead Time and Build Interval trends showed recent improvement. The appropriate response: prioritize the deployment stability problem (the biggest constraint), while maintaining the process discipline that was already improving code and build throughput.
Future Directions
Two extensions to the CD Indicator framework are under exploration:
- Predictive analytics: Using machine learning to predict stability and speed in downstream pipeline environments based on upstream indicator data—potentially enabling earlier intervention before failures propagate.
- Automated deployment gates: Automatically blocking deployments into certain pipeline environments when CD Indicator thresholds are breached, enforcing quality standards without manual intervention.
Summary
Continuous Delivery Indicators give development teams a structured, data-driven lens for continuously improving how they build software. By quantifying value stream stability and speed, teams can move away from gut-feel and post-incident reactions toward deliberate, evidence-based investment in process improvement.
Critically, the approach depoliticizes development prioritization. When bottlenecks are visible in data rather than surfaced through anecdote or influence, the conversation shifts from opinion to evidence. This supports healthier team dynamics, clearer trade-off decisions, and ultimately better business outcomes—higher user engagement and more reliable revenue-generating software.
Acknowledgements
The CD Indicators Tool described in this article was implemented with instrumental contributions from Kiran Kumar Gollapelly, Krishna Chaithanya Pomar, and Bhadri Narayanan ARR, with initial funding from Frances Paulisch. Thanks to the entire teamplay team at Siemens Healthineers for introducing and adopting these methods.
This article is part of the Data-Driven Decision Making for Software Product Delivery Organizations Series, which covers data-driven approaches across Product Management, Development, and Operations.