Which Quality Metrics Matter Most for DevOps? Key Findings from Industry Research

This article summarises findings originally reported on devops.com, drawing on research commissioned by Tricentis and conducted by Forrester.

The Quality Metrics Problem in DevOps

Software delivery has been radically reshaped by DevOps — shorter release cycles, microservices, CI/CD pipelines, cross-functional teams — yet the quality metrics most organisations rely on have barely changed. Many were designed for waterfall-era release schedules and manual testing workflows. Applying them to a continuous delivery pipeline can obscure more than they reveal.

Every other part of the delivery process has been examined and optimised for speed and reliability. Quality measurement deserves the same scrutiny. The core question is: which metrics actually tell you whether a release candidate carries an acceptable level of risk, and which ones are just historical baggage?

The Research Methodology

To answer this question, Tricentis commissioned Forrester to study which of 75 quality metrics matter most for DevOps success. The methodology proceeded in five steps:

1. Survey scope: 603 global enterprise leaders responsible for DevOps strategy were surveyed.
2. Identify high performers: From those 603 respondents, 157 met Forrester’s criteria for mature and successful DevOps adoption. These are referred to as “DevOps experts” or “DevOps leaders” throughout the research.
3. Measure usage and value: Experts reported which metrics they regularly measure and how highly they rated each one’s value.
4. Rank each metric: Each of the 75 metrics was scored on two axes — usage frequency and perceived value — to produce heat maps and quadrant charts.
5. Identify differentiators: Metrics where DevOps leaders measured significantly more than DevOps laggards were flagged as “DevOps differentiators.”

The Four Categories of Quality Metrics

The 75 metrics were grouped into four stages of the delivery pipeline:

• Build — metrics captured during code compilation, static analysis, and unit testing.
• Functional Validation (FV) — metrics that assess whether individual features behave as specified.
• Integration Testing (IT) — metrics focused on how components and APIs interact.
• End-to-End Regression Testing (E2E) — metrics that verify full system behaviour before release.

Understanding the Quadrant Framework

For each category, Forrester plotted metrics on a quadrant with usage on one axis and value on the other, producing four segments:

• Value Added: Used frequently and consistently rated as valuable. These are your core DevOps quality indicators.
• Hidden Gem: Not used frequently, but rated as highly valuable by the teams that do use them. These represent the biggest opportunity for improvement.
• Overrated: Used frequently but not rated as particularly valuable by teams that measure them. Worth reconsidering whether they earn their overhead.
• Distraction: Low usage and low value. Candidates for removal from your reporting dashboards entirely.

This quadrant model is useful beyond the specific research findings — it provides a reusable lens for auditing any set of metrics your team currently tracks.

Hidden Gems: High Value, Low Adoption

The following metrics are not commonly used — even among DevOps experts — but are ranked as extremely valuable by the teams that do measure them. These represent the strongest opportunity to improve your quality signal without dramatically increasing measurement overhead:

• New defects (Integration Testing)
• Critical defects (Functional Validation)
• Automated tests prioritised by risk (Build)
• Code coverage (Build)
• Test cases executed (Build)
• Static analysis results (Build)
• Variance from baselines of percent of test cases passed (End-to-End)
• Release readiness (End-to-End)

The strong showing of Build-stage metrics here is notable. Risk-prioritised automated tests and static analysis results both appear as hidden gems, suggesting many teams are missing early quality signals that could reduce downstream defects and rework.

Top DevOps Differentiators

These are the metrics that DevOps leaders measure significantly more than DevOps laggards. If you want to understand what separates high-performing DevOps organisations from average ones, this list is a strong starting point:

• Automated tests prioritised by risk (Build)
• Percent of automated E2E test cases (End-to-End)
• Risk coverage (Integration Testing)
• Release readiness (Functional Validation, Integration Testing, End-to-End)
• Test efficiency (Functional Validation, Integration Testing)
• Requirements covered by tests (Build, Functional Validation, Integration Testing, End-to-End)
• Test case coverage (Build, End-to-End)
• Static analysis results (Build)
• Variance from baseline of percent of test cases passed (End-to-End)
• Test effectiveness (Functional Validation, Integration Testing, End-to-End)

Several themes emerge here. Risk-based thinking — tracking risk coverage, release readiness, and risk-prioritised test automation — distinguishes mature DevOps programmes from those still measuring activity rather than outcomes. Requirements traceability across all four pipeline stages also appears repeatedly, suggesting that linking tests back to business requirements is a hallmark of high-performing teams.

Most Used Metrics by DevOps Experts

These are the metrics most frequently tracked overall by DevOps leaders, ranked by usage:

1. Test case coverage (End-to-End)
2. Pass/fail rate (Functional Validation)
3. API pass/fail rate (Integration Testing)
4. Number of tests executed (End-to-End)
5. API bug density (Integration Testing)
6. Requirements covered by tests (Functional Validation)
7. Requirements covered by tests (End-to-End)
8. Blocked test cases (Functional Validation)
9. Percent of automated E2E test cases (End-to-End)
10. Successful code builds (Build)

The presence of API-specific metrics (API pass/fail rate, API bug density) in the most-used list reflects the shift toward microservices and API-driven architectures. Teams that have adopted these patterns clearly feel the need to instrument their integration layer more carefully.

Most Valued Metrics by DevOps Experts

Usage frequency and perceived value do not always align. These are the metrics rated most highly by the DevOps experts who track them — in other words, the metrics teams consider most worth the effort:

1. Requirements covered by API tests (Integration Testing)
2. Percent of automated E2E tests (End-to-End)
3. Requirements covered by tests (End-to-End)
4. Requirements covered by tests (Functional Validation)
5. Count of critical functional defects (Functional Validation)
6. Total number of defects discovered in test (End-to-End)
7. Number of test cases executed (End-to-End)
8. Pass/fail rate (Functional Validation)
9. New API defects found (Integration Testing)
10. Automated tests prioritised by risk (Build)

Requirements traceability dominates the most-valued list, appearing in three separate entries across different pipeline stages. This strongly suggests that the ability to answer “does this release satisfy our stated requirements?” is the quality question DevOps leaders care most about — not just whether the build passed.

Practical Takeaways for DevOps Teams

The research surfaces several actionable conclusions for teams looking to improve their quality measurement practices:

• Audit your current metrics against the quadrant model. Are you spending time collecting metrics that fall into the “Overrated” or “Distraction” quadrants? Reallocating that effort toward hidden gems could improve decision quality with little additional cost.
• Adopt risk-based thinking early. Automated tests prioritised by risk and risk coverage both appear as differentiators and hidden gems. Shifting left on risk assessment — rather than treating all tests equally — is a characteristic of mature DevOps quality programmes.
• Track release readiness as a first-class metric. It appears as both a hidden gem and a differentiator across multiple pipeline stages. If you cannot answer “is this release candidate ready?” with a single data point, you likely lack this metric in a usable form.
• Instrument your API layer. API-specific metrics appear in both the most-used and most-valued lists, reflecting the increasing centrality of service integration to application quality.
• Connect tests to requirements. Requirements traceability is the single most consistent theme across both the differentiator list and the most-valued list. If your test results cannot be mapped back to business requirements, you are missing a critical quality signal.

Where to Go Deeper

The full analysis — including heat maps and quadrant charts for all four pipeline stages, per-metric breakdowns, and the complete ranking of all 75 metrics — was published in an eBook titled “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics — Ranked by Industry Experts,” produced by Tricentis. The original coverage appeared on devops.com. Readers seeking the full dataset and visualisations should search for the eBook directly on the Tricentis or Forrester websites.