This article draws on reporting and expert commentary originally published by InformationWeek, based on a survey conducted by Interop ITX and InformationWeek in February 2018.
The Real Risk of Measuring the Wrong Things
The story of a large retailer’s decline offers a cautionary tale for engineering teams: obsessing over internal operational metrics while losing sight of what customers actually need can be fatal. Optimising server response times, reducing ticket queues, and improving internal deployment pipelines are all valuable — but only if they translate into tangible improvements for the people your software serves.
This tension between internal efficiency metrics and external customer outcomes sits at the heart of one of the most persistent challenges in DevOps: knowing what success actually looks like.
The Survey Data: DevOps Teams Are Looking Inward
According to a February 2018 survey conducted by Interop ITX and InformationWeek of approximately 150 technology professionals:
- 49% of respondents said they measure DevOps success primarily through internal factors — lower costs, fewer application defects, improved application performance, better ROI, improved efficiencies, and better cross-department collaboration. This figure was up from 35% in 2017.
- Only 29% of respondents primarily measured success using external business factors such as increased revenue, faster time-to-market, improved competitive positioning, and enhanced customer experience — down one percentage point from 2017.
The trend is moving in the wrong direction. More teams are turning inward at a time when the competitive landscape demands a sharper focus on customer-facing outcomes.
Why This Is a Problem: Expert Perspectives
Andi Mann, Chief Technology Advocate for Splunk at the time, described the survey findings as “disturbing.”
“Looking at how modern businesses are disrupting enterprises, it’s about delivering services. As a customer, I don’t care about server response time — I care about how quickly I get a movie.”
Andi Mann, Chief Technology Advocate, Splunk
The point is critical. Internal metrics such as deployment pipeline speed or mean defect rates are useful engineering signals, but they are proxies. The ultimate goal of any DevOps initiative is to deliver better, faster, more reliable value to customers. If your metrics don’t trace back to that goal, you risk optimising for the wrong outcomes entirely.
The Four Metrics That Actually Matter
Robert Reeves, co-founder and CTO of Datical (now Liquibase), pointed to the DevOps Research and Assessment (DORA) framework as the industry-standard approach for measuring DevOps performance in a way that connects engineering work to business results. The four DORA metrics are:
- Deployment frequency — How often does your team successfully release to production? Higher frequency indicates a healthier, lower-risk delivery pipeline.
- Lead time for changes — How long does it take for a commit to reach production? Shorter lead times mean faster delivery of value to customers.
- Mean time to recovery (MTTR) — When a failure occurs, how quickly can your team restore service? This directly affects the customer experience during incidents.
- Change failure rate — What percentage of changes result in a degraded service or require remediation? Lower rates signal higher-quality deployments.
These four metrics have become the de facto standard for high-performing engineering organisations because they measure outcomes rather than activity. A team can merge hundreds of pull requests a week and still score poorly on lead time or change failure rate — activity alone tells you nothing about delivery quality.
Why Teams Default to Internal Metrics: The Silo Problem
Mann identified organisational silos as the root cause of inward-facing measurement. Historically, IT operations teams were not directly tied to business outcomes — they were measured on uptime, ticket resolution, and infrastructure costs. DevOps is designed to break down those walls, but the cultural change takes time.
“A lot of organisations are still in these functional silos. They measure what they can directly affect, and unfortunately those silos can’t affect outcomes.”
Andi Mann, Splunk
A separate survey conducted by Micro Focus on enterprise DevOps adoption found that 50% of respondents identified IT silos as a major impediment to application development. When teams are siloed, they naturally optimise for the slice of the value stream they control — even if that slice doesn’t correspond to anything a customer cares about.
Leverage Your Toolchain Data
Rosalind Radcliffe, a Distinguished Engineer at IBM, noted that many of the tools already used in a DevOps transformation naturally generate the data needed to populate meaningful metrics. CI/CD pipelines, incident management platforms, deployment trackers, and observability tools all produce signals that can feed directly into DORA-style measurement.
“Many of the tools that people adopt as part of their DevOps transformation help generate data that can help drive numbers into these metrics.”
Rosalind Radcliffe, Distinguished Engineer, IBM
The implication is practical: you probably don’t need to build new measurement infrastructure from scratch. Audit the data your existing tools produce and map it to the metrics that matter. Tools like GitHub, GitLab, Jira, PagerDuty, and purpose-built DORA dashboards (such as those offered by Google Cloud’s DevOps Research capabilities) can surface these numbers with relatively little additional configuration.
Who Is Responsible for Tracking DevOps Success Metrics?
Knowing what to measure is only part of the challenge — someone needs to own the measurement and act on it. Mann’s guidance here varies by organisation size:
- In larger organisations: The head of application development, a Project Management Office (PMO), or a dedicated project manager for application development should be accountable for evaluating the DevOps team’s value stream and success metrics.
- In smaller organisations: A Scrum Master or engineering lead can take on responsibility for tracking and communicating DevOps success metrics.
Mann’s broader point is that everyone on the team should care about these outcomes — but without a clear owner, metrics tend to get reviewed infrequently and acted on even less often.
Practical Steps to Reorient Your DevOps Metrics
- Audit your current metrics. List everything your team currently tracks. Categorise each metric as internal (efficiency, cost, defect rates) or external (customer experience, revenue impact, time-to-market).
- Implement the four DORA metrics. If you’re not already tracking deployment frequency, lead time for changes, MTTR, and change failure rate, start now. Most CI/CD and incident management tools can supply this data.
- Connect metrics to customer outcomes. For each DORA metric, document a direct line to a customer experience impact. For example: lower MTTR → shorter outage windows → fewer customers affected by downtime.
- Assign ownership. Designate a specific role — PMO lead, engineering manager, or Scrum Master — to review and report on success metrics on a regular cadence.
- Break down silos. Share DevOps metrics with product, business, and customer success teams. Making the data visible across functions helps everyone understand how engineering performance translates to business results.
Key Takeaway
Internal metrics are not useless — they are useful diagnostic signals. But they should never be the primary definition of DevOps success. The teams and organisations that consistently deliver value are those that keep customer outcomes at the centre of how they measure their own performance. The DORA four-metric framework offers a proven, research-backed starting point for any team ready to make that shift.