This article is based on a conversation with Dr. Nicole Forsgren, then CEO and Chief Scientist at DORA (DevOps Research and Assessment) and co-author of Accelerate, originally published via informationweek.com in 2018.
Why Measurement Is the Foundation of DevOps Success
DevOps is, at its core, the art and science of continuous improvement. Organizations across every industry are racing to deliver better software faster—but speed without visibility is dangerous. Many teams still operate with manual processes, disconnected tool chains, or isolated pockets of automation separated by wide manual hand-offs. These gaps are exactly where measurement becomes indispensable.
You cannot improve what you cannot see. Before you can define where you want to go, you need an accurate picture of where you are right now. That means establishing baseline metrics from the very start of your DevOps journey—whether that is release frequency, lead time for changes, or mean time to recovery (MTTR). Baselines give you a statistical foundation to visualize where improvements are occurring and where bottlenecks persist.
“We capture metrics in an effort to strive to get better every year and not just set our sights on a destination. Because if we just try to point to a destination, once we arrive and we stop, the rest of the industry will continue to get better and they will pass us by.”
Dr. Nicole Forsgren, co-author of Accelerate and lead investigator on the State of DevOps Report
One important mindset shift: a bad baseline is not a failure—it is valuable data. Even unflattering numbers communicate the business case for investing in improvement. As Forsgren put it: embrace the bad. A poor metric gives you the evidence you need to justify resources, prioritize fixes, and demonstrate progress over time.
Choosing Metrics That Actually Matter
Not all metrics are created equal. Vanity metrics—numbers that look good in a slide deck but do not drive decisions—can actually mislead teams and waste measurement effort. The metrics worth tracking are those that connect directly to business outcomes and reveal the health of your software delivery pipeline.
Some widely adopted pipeline health metrics include:
- Deployment frequency: How often code is successfully released to production. Higher frequency generally correlates with lower risk per release.
- Lead time for changes: The time from a committed code change to that change running in production. Shorter lead times indicate a more efficient pipeline.
- Mean Time to Recovery (MTTR): How quickly a team can restore service after an incident or failed deployment. This is a critical measure of operational resilience.
- Change failure rate: The percentage of deployments that cause a degraded service or require a hotfix. Lower rates signal higher quality gates.
- Idle time / environment wait time: How long developers spend waiting for environments to be provisioned or available. Self-service, automated environment provisioning can dramatically reduce this waste.
These four pipeline metrics—deployment frequency, lead time, MTTR, and change failure rate—form the backbone of the DORA research framework and are detailed extensively in Accelerate. The right set for your organization may vary, but the guiding question is always: does this metric reflect something the business genuinely cares about?
The Challenge of Measuring in an Automated World
Automation accelerates delivery, but it also amplifies defects. A bug that exists in one place in your codebase may be replicated across many other locations; add automation and that bug can reach production at machine speed before anyone notices. This is why metrics and defined quality gates must be established before you automate, not after.
Even a metric that seems straightforward on paper—like MTTR—becomes significantly more complex in a highly automated environment. Automated rollbacks, canary deployments, and distributed tracing all change how you measure and interpret recovery time. Teams need to ensure their metric definitions keep pace with their tooling.
Equally important is the shift in testing culture. As continuous delivery pipelines mature, manual testing as a primary quality gate becomes a bottleneck. Teams increasingly need engineers who can write automated tests, not just execute manual test scripts. Automating test execution is one of the highest-leverage investments a DevOps team can make to maintain quality at speed.
Combining System Metrics with Human Insight
One nuanced point that Forsgren raised is the complementary relationship between system-based metrics and survey-based metrics. Automated tooling can capture precise, timestamped data about deployment events, error rates, and pipeline durations. But people—the engineers, operators, and product managers living inside these systems daily—can provide qualitative context that raw telemetry cannot.
“People can tell us if we’re not using version control. People can tell us how often we deploy. People can tell us approximately what our MTTR is. People can give us reasonable insights into the health status of our deployment pipeline. Survey-based measures can range from non-scientific to advanced scientific measures, and they can complement our system-based measures.”
Dr. Nicole Forsgren
DORA’s State of DevOps research uses scientifically validated survey instruments alongside system data, precisely because neither source alone tells the full story. If your tooling reports a three-minute MTTR but engineers feel like incidents drag on for hours, that discrepancy is itself a signal worth investigating. Structured surveys, retrospectives, and team health checks are legitimate measurement instruments—not soft alternatives to real data.
Treating Infrastructure and Pipelines as Code
A prerequisite for reliable metrics is a consistent, reproducible environment. If your environments are “snowflakes”—manually configured and subtly different from one another—your measurements will reflect configuration variance as much as actual process performance. Practices like infrastructure as code, pipeline as code, and configuration management bring the same version-control discipline to infrastructure that software engineers apply to application code.
When environments are defined in code, provisioned on demand, and torn down automatically, idle time drops substantially. Self-service environment provisioning is one of the most impactful changes a platform engineering team can make to reduce the time developers spend waiting rather than building.
Turning Telemetry into Action
Collecting metrics and visualizing pipeline telemetry is valuable—but it is only the first step. The real payoff comes when that data drives decisions. Fast feedback loops are the heart of both DevOps and Agile: short cycle times, early detection of failures, and rapid course correction.
To make metrics actionable, consider the following practices:
- Define your questions before you collect data. Know what you are trying to learn. Dashboards full of uninterpreted numbers are noise. Start with a specific question—”Where is the most time lost in our release process?”—and instrument accordingly.
- Set targets, not just baselines. A baseline tells you where you are. A target—agreed upon by engineering and business stakeholders—tells the team what success looks like and creates accountability.
- Review metrics in retrospectives. Integrate metric review into your sprint retrospectives or weekly team syncs. Trends matter more than point-in-time snapshots.
- Act on leading indicators, not just lagging ones. MTTR and change failure rate are lagging indicators—they tell you about past events. Test coverage, code review turnaround time, and build stability are leading indicators that predict future outcomes.
- Celebrate incremental progress. Small improvements compounded over many iterations produce significant results. If your MTTR dropped by 10% this quarter, that is a win worth communicating.
Key Takeaways
- Establish baseline metrics immediately—even an unflattering baseline is useful for justifying investment and tracking progress.
- Focus on metrics that connect to business outcomes: deployment frequency, lead time, MTTR, and change failure rate are proven starting points.
- Automation amplifies both speed and defects; quality gates and metric definitions must be built into the pipeline from the start.
- Combine system-generated telemetry with structured human feedback for a complete picture of pipeline health.
- Infrastructure as code and self-service environments reduce measurement noise and idle time simultaneously.
- Define the questions you want to answer before diving into data sets—metrics without questions produce dashboards, not insights.
The competitive landscape in software delivery moves fast. Teams that instrument their pipelines, interpret the data honestly, and act on what they learn will continue to improve. Those that do not will find themselves passed by organizations that do.