Most software teams still rely on quality metrics designed for waterfall-era delivery cycles: number of automated tests, test case coverage, pass/fail rates. In a world of continuous integration and continuous delivery (CI/CD), these numbers give development and operations teams an incomplete—and sometimes misleading—picture of release risk.
To understand which quality metrics are actually most valuable in a DevOps context, Tricentis commissioned Forrester Consulting to survey DevOps practitioners worldwide. The resulting 55-page report—DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts—provides rankings, heat maps, quadrant mappings, and regional analysis. The findings below are drawn from that research.
Why Classic Quality Metrics Fall Short in DevOps
In traditional software delivery, quality gates were designed around infrequent, large releases. A test pass rate of 95% or a count of automated test cases made sense when releases happened quarterly. In a CI/CD pipeline, where small increments ship continuously, the same metrics fail to answer the question that actually matters: does this release candidate carry an acceptable level of risk for production?
Every other component of the DevOps toolchain—pipelines, infrastructure-as-code, observability—has been redesigned around speed and continuous feedback. Quality measurement has largely been left behind.
Global Findings: Three Key Takeaways
Across all regions surveyed, DevOps experts consistently ranked risk- and coverage-oriented metrics above raw counting metrics. Three patterns emerged clearly:
- Business risk framing drives metric selection. Organizations that reframed quality in terms of business risk shifted their measurement practices accordingly—tracking metrics that quantify risk exposure rather than test volume.
- Experts favour contextual metrics; laggards favour counting metrics. DevOps leaders gravitated toward metrics such as requirements coverage and risk coverage. Less mature teams continued to rely on metrics like number of tests executed—numbers that are easy to collect but difficult to act on.
- End-to-end user experience metrics separate leaders from laggards. High-maturity teams measured quality across full user journeys and end-to-end transactions. Lower-maturity teams measured quality at the application or team boundary, creating blind spots at integration points.
The overarching conclusion: DevOps quality transformation is not primarily a tooling problem. It requires a cultural and organisational shift toward shared ownership of business risk across development, testing, and operations.
Europe: Higher Measurement Commitment and Stronger Risk Focus
European respondents stood out for the breadth and depth of their quality metric measurement practices. Three regional trends were notable:
1. Broader Adoption of Quality Metrics Across the Board
For almost all metrics surveyed, European usage rates were at least 6% higher than the global average. For metrics related to time, coverage, risk, effectiveness, and efficiency, the gap widened to more than 14%. This reflects a culture of continuous process scrutiny—particularly around resource utilisation and delivery efficiency.
2. Risk and Coverage Metrics Valued More Highly
European respondents ranked risk and coverage quality metrics approximately 21% higher than the global average. The survey sample skewed toward Financial Services, Insurance, Healthcare, and Government—all heavily regulated sectors where risk measurement is a compliance requirement as much as an engineering discipline. The elevated scores may also reflect a stronger emphasis on protecting brand reputation in markets with sophisticated, demanding consumers.
3. Test Data Preparation Time Is a Distinctive Concern
European teams were 18% more likely to measure Time Spent Preparing Test Data than the global average, and valued the metric 23% more highly. The most plausible explanation is the General Data Protection Regulation (GDPR), which came into effect in May 2018 and placed significant restrictions on the use of production data in test environments. Teams forced to adopt data masking and synthetic data generation naturally began tracking the time cost of these new processes and monitoring their impact on overall pipeline throughput.
Asia Pacific: End-to-End Testing, API Quality, and a Widening Maturity Gap
The Asia Pacific region showed a distinct pattern: a stronger emphasis on end-to-end and API quality metrics, combined with a wider gap between DevOps leaders and laggards than seen elsewhere.
1. End-to-End Testing Metrics Are a Regional Priority
Asia Pacific respondents measured and valued end-to-end testing metrics more than their global peers. Percent of Automated End-to-End Tests was measured by 47% of regional organisations compared to 36% globally, and highly valued by 84% versus 70% globally. Risk Coverage showed an even larger gap: measured by 49% of Asia Pacific respondents versus 34% globally, and highly valued by 71% versus 59% globally. This focus aligns with the region’s strong emphasis on digital transformation and delivering seamless user experiences across complex, multi-step transactions.
2. API Testing Metrics Reflect Open Banking Trends
Overall, API quality metrics were measured by 16% more organisations in Asia Pacific than in the global sample. The highest-valued API metrics were API Test Coverage (63% regionally vs. 39% globally) and API Risk Coverage (79% vs. 62% globally). The majority of Asia Pacific respondents identified as working in Financial Services and Insurance—a sector undergoing rapid transformation through API-driven open banking initiatives, which places API reliability and risk coverage at the centre of quality strategy.
3. A Larger Gap Between DevOps Leaders and Laggards
The study classified respondents as DevOps leaders or laggards based on process maturity indicators. Asia Pacific had a lower proportion of DevOps leaders than the global average (18% vs. 26%). Regional leaders measured quality metrics at rates comparable to their global counterparts; regional laggards, however, measured quality metrics at substantially lower rates than laggards elsewhere. This bimodal distribution suggests that organisations that have genuinely invested in DevOps maturity in the region are competitive globally, while those that have not are falling further behind relative to both regional leaders and global peers.
Practical Implications for DevOps Teams
The global and regional findings converge on a consistent set of recommendations for teams that want to modernise their quality measurement practices:
- Audit your current metrics. Identify which metrics are counting artifacts (tests run, defects logged) and which are contextual (risk coverage, requirements coverage, end-to-end pass rate). Shift investment toward the latter.
- Define risk in business terms. Work with product owners and stakeholders to articulate what a bad release means for end users and revenue. Translate that definition into measurable quality signals.
- Expand coverage to integration and API boundaries. Team-level or application-level metrics hide failure modes that only appear at system integration points. Include end-to-end and API-level metrics in your go/no-go criteria.
- Address test data as a pipeline dependency. Especially in regulated industries or regions with data privacy obligations, test data preparation time is a real pipeline constraint. Measure it, and invest in synthetic data or masking tooling to reduce it.
- Use maturity benchmarks to set priorities. If your team is in the laggard category, the fastest path to improvement is adopting the measurement practices that leaders already use—not building entirely new tooling.
Ultimately, delivering better software faster requires more than CI/CD pipelines and test automation frameworks. It requires agreement—across engineering, product, and business stakeholders—on what quality and acceptable risk actually mean. The metrics follow from that agreement, not the other way around.
Source: Tricentis / Forrester Consulting, DevOps Quality Metrics that Matter (originally reported via cio.com, 2019).