There is a well-worn business adage: “We manage what we measure.” In most engineering disciplines, that is straightforward enough. In DevOps, it is anything but. Software delivery blends technical automation, cultural change, and organizational structure — a combination that makes agreeing on meaningful metrics genuinely difficult.
This article draws on a 2018 interview with Tim Buntel, then DevOps Advocate at XebiaLabs (published originally on adtmag.com), and expands on the core frameworks he described. The underlying measurement principles remain relevant regardless of tooling choices.
Why DevOps Metrics Are Uniquely Hard
Manufacturing has clear, countable outputs: units produced, defect rate per thousand, cycle time per unit. Software delivery is harder to quantify. A feature shipped is not the same as a car assembled — its value depends on user adoption, system stability, and business context. When you layer in the cultural transformation that DevOps demands (shared responsibility, cross-functional teams, blameless postmortems), purely technical metrics tell only part of the story.
Organizations making real investments in DevOps — new toolchains, reorganizations, training programs — inevitably reach the same question: How do we know if this is working? The answer is not a single number. It is a small set of complementary global measurements evaluated together, over time, and without pitting teams against one another.
The Four Metrics That Actually Matter
The following four measurements were popularized by DevOps Research and Assessment (DORA) and have been validated by large-scale surveys of software delivery organizations. They focus on outcomes — speed and stability — rather than activity.
1. Deployment Frequency
Deployment frequency measures how often your team successfully releases to production. The goal is not to achieve the fastest possible cadence for its own sake. Rather, it is to reach a state where deploying is a low-risk, routine event — so that you can release when the business needs it, whether that is multiple times per day or once a month.
A high-performing team is one that can deploy on demand, not necessarily one that always deploys at maximum frequency. Chasing a mythical “five-minute release cycle” without business justification is a distraction.
2. Lead Time for Changes
Lead time for changes measures the elapsed time from a developer committing code to that code running in production and accessible to users. It is often the most automation-friendly of the four metrics because a well-built CI/CD pipeline makes it observable and predictable.
Every manual gate between commit and deployment is a candidate for automation. Tracking this metric longitudinally reveals whether process improvements are actually shortening the delivery cycle or just moving bottlenecks around.
3. Mean Time to Recovery (MTTR)
MTTR measures how quickly a team can restore service after an incident or failed deployment. It is often confused with Mean Time Between Failures (MTBF) — a metric that tries to minimize how often failures occur. MTBF sounds appealing but has a dangerous side effect: it incentivizes teams to avoid change, which stifles experimentation and slows innovation.
MTTR takes a more realistic stance: systems will fail. The question is how fast you can recover. Teams that know they can recover quickly are more willing to experiment, ship smaller changes, and iterate — all core DevOps behaviors. Optimizing for fast recovery is culturally liberating in a way that optimizing for zero failure is not.
4. Change Failure Rate
Change failure rate measures the percentage of deployments that result in a service degradation, incident, or rollback. It is the stability counterbalance to deployment frequency. Moving fast is only valuable if what you ship is reliable enough. A low change failure rate signals that your testing, code review, and release processes are working. A high rate signals systemic risk that speed alone will not fix.
Together, these four metrics form a balanced scorecard: two measure speed (deployment frequency, lead time) and two measure stability (MTTR, change failure rate). Organizations that improve on all four simultaneously are operating at a genuinely higher level of maturity.
A Critical Rule: Measure Globally, Not Locally
None of these metrics are useful if applied to individual developers or small teams in isolation. Their purpose is to reveal systemic patterns across the organization. When metrics are applied locally — or used to rank teams against one another — they produce exactly the wrong behaviors: gaming, sandbagging, and internal competition instead of collaboration.
Signs you are using metrics badly include: teams competing over dashboard rankings, developers inflating story point estimates to protect their velocity numbers, or managers pressuring individuals to hit targets that depend partly on factors outside their control. These are symptoms of measuring the wrong things, or measuring right things the wrong way.
Common Metrics That Do More Harm Than Good
Two metrics appear repeatedly in engineering organizations and consistently cause problems when used as DevOps success indicators.
Lines of Code
Lines of code (LOC) is seductively easy to measure and nearly meaningless as a quality or productivity signal. A developer shipping thousands of lines of code might be producing elegant, maintainable software — or generating bloat that creates long-term maintenance debt. Conversely, a developer who drastically reduces a codebase by removing dead code and refactoring may be delivering enormous value with a negative LOC count.
The better question to ask developers is not “how much did you write?” but “did you solve the business problem with the most efficient code possible?” LOC cannot answer that question. Business outcomes can.
Velocity
Velocity, in the Agile sense, is a capacity-planning tool: it helps a team estimate what they can deliver in a given sprint based on historical throughput. It was never designed to be a performance metric, and using it that way breaks it almost immediately.
The problems with velocity as a DevOps metric include:
- It is not comparable across teams. A backend team, a front-end team, a microservices team, and an API team will define and measure story points differently. Comparing their velocities is meaningless.
- It is easily gamed. When developers know their performance is judged by velocity, they inflate story point estimates to make targets easier to hit — destroying the accuracy of the planning signal velocity was meant to provide.
- It encourages local optimization. Teams optimize for their own velocity numbers rather than contributing to shared organizational outcomes.
Velocity belongs in sprint planning discussions, not on a DevOps performance dashboard.
Putting It Into Practice
Starting a DevOps measurement program does not require sophisticated tooling on day one. A practical approach:
- Define baselines first. Before optimizing anything, establish where you are today on the four DORA metrics. Even rough estimates are useful starting points.
- Automate data collection. Manual metric collection is error-prone and slow. Instrument your CI/CD pipeline to emit deployment events, failure signals, and recovery timestamps automatically.
- Review trends, not snapshots. A single data point is noise. Look at how metrics move over weeks and quarters — that is where the signal lives.
- Use metrics to open conversations, not close them. A spike in change failure rate should prompt a blameless retrospective, not a blame session. Metrics surface problems; teams solve them.
- Report globally. Present metrics at the organizational level in leadership reviews. Use team-level data privately within teams for self-improvement, never for public ranking.
Summary
Measuring DevOps effectively means resisting the temptation to count things that are easy to count (lines of code, velocity, deployment counts) and instead tracking outcomes that reflect the health of the whole delivery system. The four DORA metrics — deployment frequency, lead time for changes, mean time to recovery, and change failure rate — provide a balanced, outcome-focused view of software delivery performance. Applied globally and used to foster collaboration rather than competition, they give engineering leadership the intelligence it needs to guide real improvement.
Original interview source: adtmag.com (2018), featuring Tim Buntel, XebiaLabs.