This article summarizes findings from “The Definitive Software Quality Metrics For Agile+DevOps,” a Forrester report commissioned by Tricentis and covered by SD Times (Ian C. Schafer, October 2018).
From Counting Tests to Measuring Business Risk
Organizations that have seen the greatest return from Agile and DevOps adoption share a common quality-assurance characteristic: they have moved away from counting metrics — such as total tests executed or pass/fail ratios — toward contextual metrics that assess whether software genuinely meets user experience requirements and delivers acceptable business risk.
This finding comes from a Forrester research report commissioned by testing platform vendor Tricentis, titled “The Definitive Software Quality Metrics For Agile+DevOps.” The study surveyed 603 enterprise Agile and DevOps practitioners and segmented respondents into “DevOps leaders” and “DevOps laggards” based on their maturity and outcomes.
What Separates DevOps Leaders from Laggards
According to the report, DevOps leaders are significantly further along in end-to-end test automation — a prerequisite for the release frequency and delivery speed that modern software pipelines demand. The study identified five practices almost universally employed by these leaders:
- Automated end-to-end functional testing as a core quality gate, not an afterthought.
- Properly allocated testing budgets paired with a deliberate focus on upgrading testing skills across teams.
- Continuous testing integrated into the pipeline to match release frequency and support continuous delivery.
- Testers embedded in integrated delivery teams rather than siloed in a separate QA function.
- Shift-left testing — moving testing earlier in the development lifecycle to catch defects before they compound.
Fifty-nine percent of all survey respondents identified manual testing as the primary bottleneck slowing their organizations down relative to top performers.
Automation Without Risk Measurement Is Dangerous
The report is explicit that automation alone is not the answer. Forrester warned:
“Automating the software development life cycle is an imperative for accelerating the speed and frequency of releases. However, without an accurate way to measure and track quality throughout the software development life cycle, automating the delivery pipeline could increase the risk of delivering more defects into production. And if organizations cannot accurately measure business risk, then automating development and testing practices can become a huge danger to the business.”
Forrester, The Definitive Software Quality Metrics For Agile+DevOps
In other words, automating a flawed or poorly measured process at scale amplifies defect risk rather than reducing it. The pipeline moves faster, but so does the exposure.
The Overconfidence Problem: 80% Think They’re Delivering Safely
One of the report’s most striking findings is a widespread overconfidence in risk management. Eighty percent of all respondents said they “always or often deliver customer-facing products within acceptable business risk” — yet the underlying data tells a different story.
Only 29% of DevOps laggards consider delivering within acceptable business risk to be “very important,” and even among leaders the figure reaches only 50%. Forrester concluded that the 80% self-assessment figure represents a significant overestimation of actual capability:
“Given that most firms, even the ones following continuous testing best practices, admit that their software testing processes have risk gaps and do not always give accurate measures of business risk, it stands to reason that the 80% who say they always or often deliver within acceptable risk may be overestimating their capabilities.”
Forrester, The Definitive Software Quality Metrics For Agile+DevOps
Diego Lo Giudice, vice president and principal analyst at Forrester and lead author of the report, interpreted the discrepancy this way:
“I think it’s more intentional than truth — they get it that business risk is of paramount importance and have the illusion of addressing it. It’s a misinterpretation of what real business risk is perhaps? I think it’s hard to keep testing and quality focus on business risk.”
Diego Lo Giudice, VP and Principal Analyst, Forrester
The “Geek Gap”: When Technical and Business Risk Don’t Align
Wayne Ariola, Chief Marketing Officer at Tricentis at the time of the report, offered a concrete explanation for the overconfidence gap:
“This is your classic geek gap. Business leaders assume that the definition of risk is aligned to a business-oriented definition of risk, but the technical team has it aligned to a very different definition of risk. This mismatch is your primary cause of overconfidence. For example, assume a tester saw that 100 percent of the test suite ran with an 80 percent pass rate. This gives you no indication of the risk associated with the release. The 20 percent that failed could be an absolutely critical functionality, like security authentication, or it could be trivial, like a UI customization option that’s rarely used.”
Wayne Ariola, CMO, Tricentis
This distinction is fundamental. A high overall pass rate is meaningless if the failing tests cover high-criticality user journeys. Without tagging tests to business impact, pass rates become vanity metrics.
Why Counting Metrics Have Lost Their Value
Ariola articulated clearly why the old-guard metrics no longer serve DevOps environments:
“‘Counting’ metrics lost their value because the question changed for DevOps. It was no longer about how much testing you completed in the allotted time. The focus shifted to whether you could release and what business risks were associated with the release. If you can answer this question with five tests, that’s actually better than answering it with 5,000 tests that aren’t closely aligned with business risk. Count doesn’t matter — what’s important is one, the ability to assess risk, and two, the ability to make actionable decisions based on test results.”
Wayne Ariola, CMO, Tricentis
Lo Giudice reinforced the point from an Agile perspective, arguing that productivity metrics — lines of code, story points delivered, tests run — can actively mislead teams when the real goal is valuable, working software:
“Teams focus on the amount of output, or raw code, not necessarily valuable outcomes for the business or the right code. Agile focuses on both building the right things and evaluating things right, so value metrics become more important than productivity of teams, and so related productivity metrics lose relevance. Quality, however, remains always of paramount importance.”
Diego Lo Giudice, VP and Principal Analyst, Forrester
Over-Testing: A Hidden Drag on Delivery Speed
Ariola also raised a less commonly discussed failure mode — running too many tests in the wrong places:
“We’ve seen many organizations that are severely over-testing. They’re running massive test suites against minor changes — self-inflicting delays for no good reason. To start, organizations must take a step back and really assess the risks associated with each component of their application set. Once risk is better understood, you can identify practices that will mitigate those risks much earlier in the process. This is imperative for speed and accuracy.”
Wayne Ariola, CMO, Tricentis
The implication: right-sizing your test suite to the actual risk profile of each change is as important as automating it. Broad, undifferentiated test coverage against low-risk changes burns pipeline time without reducing meaningful risk.
The Right Question to Ask Before Every Release
Ariola framed the fundamental shift in DevOps quality thinking in terms of which question teams should be asking at release time:
“In the past, when software testing was a timeboxed activity at the end of the cycle, we focused on answering the question, ‘Are we done testing?’ When this was the primary question, counting metrics associated with the number of tests run, incomplete tests, passed tests, failed tests, etc. drove the process and influenced the release decision. As you can imagine, these metrics are highly ineffective in understanding the actual quality of a release. In today’s world, we have to ask a different question: ‘Does the release have an acceptable level of risk?'”
Wayne Ariola, CMO, Tricentis
Practical Takeaways for DevOps and QA Teams
The Forrester report’s findings translate into concrete actions for engineering and QA leadership:
- Map your tests to business risk. Every test in your suite should be traceable to a business capability or user journey. If it isn’t, you cannot make a risk-informed release decision.
- Replace pass/fail counts with risk coverage scores. Track what percentage of your high-criticality paths are covered by automated tests, not just total test volume.
- Implement continuous testing end-to-end. Testing must run throughout the pipeline, not just at the end of a sprint or before a production push.
- Shift left — but do it deliberately. Earlier testing only adds value if the tests being shifted are the right ones for the stage of development.
- Audit your test suite for over-testing. Rationalize suites regularly. Running large numbers of low-value tests against small changes is a self-inflicted slowdown.
- Close the geek gap. QA and engineering teams need a shared vocabulary for business risk with product and business stakeholders. Risk tiers for application components can help create that alignment.
- Invest in tester skills alongside tooling. Automation tooling without skilled practitioners to maintain and evolve test strategy will quickly produce a brittle, noisy pipeline.
The shift from “are we done testing?” to “is this release safe to ship?” is not just a philosophical one — it requires changing how teams instrument their pipelines, how test results are reported to stakeholders, and how release decisions are governed. Organizations that make that shift, the Forrester data suggests, are the ones pulling ahead.
Source: SD Times, Ian C. Schafer, October 2018. Original research: Forrester Consulting, “The Definitive Software Quality Metrics For Agile+DevOps,” commissioned by Tricentis.