This article summarizes findings reported by SD Times based on “The Definitive Software Quality Metrics For Agile+DevOps,” a Forrester report commissioned by Tricentis and released in July 2018. Quotes are attributed to Diego Lo Giudice (Forrester) and Wayne Ariola (Tricentis) as reported in that coverage.
The Core Shift: From Counting Metrics to Contextual Metrics
Organizations that have seen the greatest returns from Agile and DevOps adoption share a common trait: they have stopped treating counting metrics — such as the total number of tests executed — as the primary indicators of software quality. Instead, they rely on contextual metrics that reflect whether software actually meets real-world user experience requirements and business risk thresholds.
This conclusion emerged from a survey of 603 enterprise Agile and DevOps specialists. The respondents identified as “DevOps leaders” were measurably better at quantifying success and were significantly further along in end-to-end test automation — a prerequisite for sustaining the rapid delivery cadences that modern software cycles demand.
The Bottleneck: Manual Testing
Despite general progress in the industry, a clear bottleneck persists. Fifty-nine percent of survey respondents identified manual testing as the primary factor holding their organizations back. While this is not a new observation, the report underscores that lagging organizations tend to underinvest in automation and continue to rely on manual gates at critical points in their delivery pipelines.
Five Practices of DevOps Testing Leaders
The report identified five practices almost universally adopted by high-performing DevOps organizations:
- Automated end-to-end functional testing — prioritized over purely manual regression cycles.
- Properly allocated testing budgets — combined with active investment in upgrading team testing skills.
- Continuous testing — implemented to match release frequency and support continuous delivery pipelines.
- Integrated testers — testers embedded within delivery teams rather than siloed in a separate QA function.
- Shift-left testing — defect detection moved earlier in the development lifecycle to reduce the cost of remediation.
Automation Without Measurement Is a Risk Multiplier
Rushing to automate without establishing accurate quality measurement is itself a significant risk. Forrester’s report put it plainly:
“Automating the software development life cycle is an imperative for accelerating the speed and frequency of releases. However, without an accurate way to measure and track quality throughout the software development lifecycle, automating the delivery pipeline could increase the risk of delivering more defects into production. And if organizations cannot accurately measure business risk, then automating development and testing practices can become a huge danger to the business.”
Forrester, “The Definitive Software Quality Metrics For Agile+DevOps” (2018)
In other words, a fast pipeline that consistently ships defects is worse than a slower, more deliberate one. Speed amplifies whatever quality posture already exists — good or bad.
The Overconfidence Problem
Perhaps the most striking finding concerns self-assessment. Eighty percent of all respondents believed they “always or often” deliver customer-facing software within an acceptable level of business risk. Yet only 29 percent of DevOps laggards considered delivering within acceptable business risk to be “very important,” and even among DevOps leaders, that figure was only 50 percent.
Forrester characterized this as likely mass overestimation:
“Given that most firms, even the ones following continuous testing best practices, admit that their software testing processes have risk gaps and do not always give accurate measures of business risk, it stands to reason that the 80% who say they always or often deliver within acceptable risk may be overestimating their capabilities.”
Forrester
Diego Lo Giudice, Vice President and Principal Analyst at Forrester and the lead on this report, described the gap as more perceptual than deliberate:
“I think it’s more intentional than truth — they get it that business risk is of paramount importance and have the illusion of addressing it. It’s a misinterpretation of what real business risk is perhaps? I think it’s hard to keep testing and quality focus on business risk.”
Diego Lo Giudice, Forrester
The “Geek Gap”: Misaligned Definitions of Risk
Wayne Ariola, Chief Marketing Officer at Tricentis, identified a specific organizational dynamic driving overconfidence — a fundamental misalignment between business and technical definitions of risk:
“This is your classic geek gap. Business leaders assume that the definition of risk is aligned to a business-oriented definition of risk, but the technical team has it aligned to a very different definition of risk. This mismatch is your primary cause of overconfidence. For example, assume a tester saw that 100 percent of the test suite ran with an 80 percent pass rate. This gives you no indication of the risk associated with the release. The 20 percent that failed could be an absolutely critical functionality, like security authentication, or it could be trivial, like a UI customization option that’s rarely used.”
Wayne Ariola, Tricentis
This is a concrete and important distinction. An 80% pass rate sounds reassuring until you realize it says nothing about which tests failed. A test suite where all critical-path tests pass and only cosmetic edge cases fail is fundamentally different from one where core security or payment flows are broken. Raw pass/fail ratios without risk weighting are misleading signals.
Why Counting Metrics Lose Value in DevOps
In a traditional, time-boxed testing phase at the end of a waterfall cycle, counting metrics made sense. The question being answered was: Are we done testing? Under that model, tracking the number of tests run, passed, failed, and outstanding was a reasonable proxy for readiness.
In a continuous delivery context, that question is no longer the right one. As Ariola explained:
“‘Counting’ metrics lost their value because the question changed for DevOps. It was no longer about how much testing you completed in the allotted time. The focus shifted to whether you could release and what business risks were associated with the release. If you can answer this question with five tests, that’s actually better than answering it with 5,000 tests that aren’t closely aligned with business risk. Count doesn’t matter — what’s important is — one, the ability to assess risk and two, the ability to make actionable decisions based on test results.”
Wayne Ariola, Tricentis
The new question is: Does this release carry an acceptable level of business risk? That question cannot be answered by counting. It requires risk-weighted, outcome-oriented metrics.
Productivity Metrics Are Not Quality Metrics
Lo Giudice flagged productivity as another example of a metric that loses relevance as Agile maturity increases:
“Measuring productivity is one example. Teams focus on the amount of output, or read code, not necessarily valuable outcomes for the business or the right code. Agile focuses on both building the right things and evaluating things right, so value metrics become more important than productivity of teams, and so related productivity metrics lose relevance. Quality, however, remains always of paramount importance.”
Diego Lo Giudice, Forrester
High throughput of the wrong features, or fast delivery of defect-prone code, is not a success metric. Agile specifically demands a dual focus: building the right thing and building it well.
Over-Testing Is a Real Problem
Counterintuitively, the report also surfaced a problem at the opposite extreme: organizations running excessively large test suites against minor changes, creating unnecessary delays with no meaningful quality benefit. Ariola described the fix:
“We’ve seen many organizations that are severely over-testing. They’re running massive test suites against minor changes — self-inflicting delays for no good reason. To start, organizations must take a step back and really assess the risks associated with each component of their application set. Once risk is better understood, you can identify practices that will mitigate those risks much earlier in the process. This is imperative for speed and accuracy.”
Wayne Ariola, Tricentis
Risk-based test selection — running only the tests relevant to the components that have changed — is a practical technique for addressing this. It requires knowing your application’s risk profile at a component level, which itself is a metric worth building.
Key Takeaways for Engineering Teams
- Reframe the release question. Stop asking “Did enough tests pass?” and start asking “What is the business risk level of this release?”
- Weight your tests by business criticality. A pass rate figure without risk weighting is not a reliable signal of release readiness.
- Align business and technical risk definitions. Close the “geek gap” by ensuring that engineers and business stakeholders share a common definition of what constitutes unacceptable risk.
- Don’t automate before you can measure. Automation amplifies your current quality posture. If your measurement framework is inaccurate, automation will deliver defects faster and at higher volume.
- Shift left — but focus on risk coverage, not test count. Earlier testing only pays off when those tests are targeted at the highest-risk components.
- Audit your test suite for over-testing. Running thousands of tests against a small change is waste. Understand your application risk profile and size test execution accordingly.
The transition from counting metrics to risk-based quality metrics is not just a tooling change — it requires a cultural and organizational shift in how engineering and business teams define, communicate, and act on software quality signals.