This article is based on an interview with Nicole Forsgren, CEO and Chief Scientist of DevOps Research and Assessment (DORA), conducted by Jeff Frick of theCUBE at PagerDuty Summit in San Francisco. Original coverage appeared on SiliconAngle.
The Problem with DevOps Adoption Statistics
Industry surveys love to report adoption numbers. According to VersionOne’s 2017 State of Agile Report, 71 percent of businesses had adopted some form of DevOps to streamline software development and deployment. A separate RightScale report cited Docker container technology as the most popular DevOps tool, used by 35 percent of respondents.
These numbers make for compelling slide decks. But according to Nicole Forsgren, CEO and Chief Scientist of DevOps Research and Assessment (DORA), they are almost useless as a guide to strategy.
“We don’t say things like, ’27 percent of the industry is using configuration management,'” Forsgren said. A peek into competitors’ playbooks can be “interesting,” she acknowledged, but knowing what tools others have purchased tells you nothing about whether those organizations are actually succeeding. “Those other reports cannot say what is predictive of improvement.”
DORA’s approach is different. Instead of simply cataloguing tool adoption, DORA studies the specific practices companies use and—critically—measures their outcomes. The goal is to identify which DevOps capabilities reliably predict better business results.
Why Adoption Rates Are Not a Roadmap
The gap between adoption and success is well-documented across technology disciplines. Forsgren pointed to research from Gartner indicating that roughly 70 percent of big data initiatives fail. If an organization’s strategy amounts to surveying what tools the market is buying and then purchasing the same software, the probability of success remains low regardless of how popular a given tool may be.
DevOps is no exception. Despite high adoption rates, many organizations struggle to realize the friction-free software delivery pipeline the methodology promises. A survey by Shippable found that 68 percent of DevOps practitioners cited poor connectivity in the DevOps toolchain as a significant obstacle, while 27 percent pointed to siloed departments as a barrier to the cross-functional collaboration DevOps requires.
High adoption rates, in other words, mask a wide variance in actual performance. Knowing that a majority of companies use a given practice or tool does not help an organization understand whether adopting it will improve their own delivery speed, stability, or business outcomes.
DORA’s Scientific Approach to DevOps Research
DORA was founded on the premise that DevOps research should meet the same standards of rigor applied in academic disciplines. The organization partnered with Puppet Labs to produce the annual State of DevOps Report, differentiating the tools and methods that produce measurable results from those that generate noise.
The data DORA collects must pass rigorous review before informing its conclusions. Compelling anecdotes about a technology or an operational method do not qualify as evidence. Every capability included in DORA’s model must demonstrably predict organizational performance.
“It lets us say, ‘If we focus on core capabilities, they will predict organizations’ ability to develop and deliver quality software with both speed and stability,'” Forsgren explained. DORA frames DevOps not as a collection of tools, but as a discipline grounded in lean and agile principles combined with technology and automation.
Debunking the Speed-vs-Stability Trade-off
One of DORA’s most significant findings challenges a widely held assumption: that faster software delivery necessarily comes at the cost of system stability. For years, this trade-off was treated as an engineering fact of life—to ship faster, you accept more instability.
DORA’s research found no evidence for this trade-off in high-performing organizations. “We thought that in order to get stability, you had to slow down. It doesn’t show up anywhere in the data,” Forsgren said. “High performers get both.”
This is consistent with the broader principle that counterintuitive findings sometimes emerge when you measure outcomes rigorously rather than assuming causal relationships. High-performing engineering organizations achieve high deployment frequency and low change-failure rates simultaneously—a combination that survey-based research built on adoption counts alone would be unlikely to surface.
Applying Research: Assessments and Practical Consulting
DORA’s work is not purely academic. The organization offers assessments and consultations that apply its research findings to specific organizational contexts.
“We offer an assessment using these strong, scientifically based measures that we have prepared and refined over four years of rigorous academic research,” Forsgren said. Assessments gather data from all relevant roles within an organization—development, testing, operations, quality assurance, information security, vendors, contractors, and consultants—and then benchmark results against industry data.
The output is specific and actionable: “I can—algorithmically—tell you what your bottleneck is, what your constraint is, where you should start to accelerate your performance,” Forsgren said.
Forsgren cited Capital One as an example of a DORA client. According to her account, Capital One conducted an assessment across more than a dozen lines of business. DORA focused the organization on two capabilities out of a possible twenty. The result, as reported by Forsgren, was a 20x improvement in deployment frequency within two months.
Key Takeaways for DevOps Practitioners
- Adoption statistics are descriptive, not prescriptive. Knowing what percentage of organizations use a given tool or practice does not tell you whether adopting it will improve your outcomes.
- Outcome measurement matters. Effective DevOps research measures deployment frequency, lead time, change-failure rate, and mean time to restore—not just tool adoption rates.
- The speed-stability trade-off is a myth for high performers. DORA’s data shows that elite organizations achieve both high velocity and high stability simultaneously.
- Focus beats breadth. Rather than adopting every available DevOps practice, identifying the specific capabilities that act as bottlenecks in your organization and improving those yields disproportionate results.
- Scientific rigor applies to operations. Applying the same standards to DevOps strategy that you would apply to a technical hypothesis—forming measurable predictions and testing them—produces better decisions than following industry trend reports.
Nicole Forsgren was featured in theCUBE’s Women in Tech series at the time of this interview. DORA was subsequently acquired by Google in 2018.