The Real Risk of Measuring the Wrong Things

The story of a large retailer’s decline offers a cautionary tale for engineering teams: obsessing over internal operational metrics while losing sight of what customers actually need can be fatal. Optimising server response times, reducing ticket queues, and improving internal deployment pipelines are all valuable — but only if they translate into tangible improvements for the people your software serves.

This tension between internal efficiency metrics and external customer outcomes sits at the heart of one of the most persistent challenges in DevOps: knowing what success actually looks like.

The Survey Data: DevOps Teams Are Looking Inward

According to a February 2018 survey conducted by Interop ITX and InformationWeek of approximately 150 technology professionals:

• 49% of respondents said they measure DevOps success primarily through internal factors — lower costs, fewer application defects, improved application performance, better ROI, improved efficiencies, and better cross-department collaboration. This figure was up from 35% in 2017.
• Only 29% of respondents primarily measured success using external business factors such as increased revenue, faster time-to-market, improved competitive positioning, and enhanced customer experience — down one percentage point from 2017.

The trend is moving in the wrong direction. More teams are turning inward at a time when the competitive landscape demands a sharper focus on customer-facing outcomes.

Why This Is a Problem: Expert Perspectives

Andi Mann, Chief Technology Advocate for Splunk at the time, described the survey findings as “disturbing.”

“Looking at how modern businesses are disrupting enterprises, it’s about delivering services. As a customer, I don’t care about server response time — I care about how quickly I get a movie.”

Andi Mann, Chief Technology Advocate, Splunk

The point is critical. Internal metrics such as deployment pipeline speed or mean defect rates are useful engineering signals, but they are proxies. The ultimate goal of any DevOps initiative is to deliver better, faster, more reliable value to customers. If your metrics don’t trace back to that goal, you risk optimising for the wrong outcomes entirely.

The Four Metrics That Actually Matter

Robert Reeves, co-founder and CTO of Datical (now Liquibase), pointed to the DevOps Research and Assessment (DORA) framework as the industry-standard approach for measuring DevOps performance in a way that connects engineering work to business results. The four DORA metrics are:

1. Deployment frequency — How often does your team successfully release to production? Higher frequency indicates a healthier, lower-risk delivery pipeline.
2. Lead time for changes — How long does it take for a commit to reach production? Shorter lead times mean faster delivery of value to customers.
3. Mean time to recovery (MTTR) — When a failure occurs, how quickly can your team restore service? This directly affects the customer experience during incidents.
4. Change failure rate — What percentage of changes result in a degraded service or require remediation? Lower rates signal higher-quality deployments.

These four metrics have become the de facto standard for high-performing engineering organisations because they measure outcomes rather than activity. A team can merge hundreds of pull requests a week and still score poorly on lead time or change failure rate — activity alone tells you nothing about delivery quality.

Why Teams Default to Internal Metrics: The Silo Problem

Mann identified organisational silos as the root cause of inward-facing measurement. Historically, IT operations teams were not directly tied to business outcomes — they were measured on uptime, ticket resolution, and infrastructure costs. DevOps is designed to break down those walls, but the cultural change takes time.

“A lot of organisations are still in these functional silos. They measure what they can directly affect, and unfortunately those silos can’t affect outcomes.”

Andi Mann, Splunk

A separate survey conducted by Micro Focus on enterprise DevOps adoption found that 50% of respondents identified IT silos as a major impediment to application development. When teams are siloed, they naturally optimise for the slice of the value stream they control — even if that slice doesn’t correspond to anything a customer cares about.

Leverage Your Toolchain Data

Rosalind Radcliffe, a Distinguished Engineer at IBM, noted that many of the tools already used in a DevOps transformation naturally generate the data needed to populate meaningful metrics. CI/CD pipelines, incident management platforms, deployment trackers, and observability tools all produce signals that can feed directly into DORA-style measurement.

“Many of the tools that people adopt as part of their DevOps transformation help generate data that can help drive numbers into these metrics.”

Rosalind Radcliffe, Distinguished Engineer, IBM

The implication is practical: you probably don’t need to build new measurement infrastructure from scratch. Audit the data your existing tools produce and map it to the metrics that matter. Tools like GitHub, GitLab, Jira, PagerDuty, and purpose-built DORA dashboards (such as those offered by Google Cloud’s DevOps Research capabilities) can surface these numbers with relatively little additional configuration.

Who Is Responsible for Tracking DevOps Success Metrics?

Knowing what to measure is only part of the challenge — someone needs to own the measurement and act on it. Mann’s guidance here varies by organisation size:

• In larger organisations: The head of application development, a Project Management Office (PMO), or a dedicated project manager for application development should be accountable for evaluating the DevOps team’s value stream and success metrics.
• In smaller organisations: A Scrum Master or engineering lead can take on responsibility for tracking and communicating DevOps success metrics.

Mann’s broader point is that everyone on the team should care about these outcomes — but without a clear owner, metrics tend to get reviewed infrequently and acted on even less often.

Practical Steps to Reorient Your DevOps Metrics

1. Audit your current metrics. List everything your team currently tracks. Categorise each metric as internal (efficiency, cost, defect rates) or external (customer experience, revenue impact, time-to-market).
2. Implement the four DORA metrics. If you’re not already tracking deployment frequency, lead time for changes, MTTR, and change failure rate, start now. Most CI/CD and incident management tools can supply this data.
3. Connect metrics to customer outcomes. For each DORA metric, document a direct line to a customer experience impact. For example: lower MTTR → shorter outage windows → fewer customers affected by downtime.
4. Assign ownership. Designate a specific role — PMO lead, engineering manager, or Scrum Master — to review and report on success metrics on a regular cadence.
5. Break down silos. Share DevOps metrics with product, business, and customer success teams. Making the data visible across functions helps everyone understand how engineering performance translates to business results.

Key Takeaway

Internal metrics are not useless — they are useful diagnostic signals. But they should never be the primary definition of DevOps success. The teams and organisations that consistently deliver value are those that keep customer outcomes at the centre of how they measure their own performance. The DORA four-metric framework offers a proven, research-backed starting point for any team ready to make that shift.

Why DevOps Failures Are Normal—and What They Tell You

Failure in a DevOps context is not a sign the approach is wrong. It is usually a signal that the implementation was incomplete, rushed, or misunderstood. According to research from the DevOps Research and Assessment (DORA) group, elite DevOps teams do not avoid failure—they recover from it faster than their peers. The goal is not a zero-failure environment; it is a learning organization that treats failure as data.

The Most Common Causes of DevOps Failure

1. Treating DevOps as a Technology Problem Only

The most pervasive mistake is assuming that buying the right tools—CI/CD pipelines, container orchestration, monitoring platforms—is sufficient. It is not. A successful DevOps program is fundamentally a cultural shift. Collaboration, psychological safety, shared ownership, and transparent communication must be addressed explicitly. The tooling exists to support the culture, not replace it.

Culture-related considerations include team values, management style, meeting cadences, workspace design, work-life balance expectations, transparency norms, and how success is defined and celebrated. Skipping this work is a reliable path to DevOps failure.

2. Ignoring the Technology Change DevOps Actually Requires

The opposite mistake is also common: assuming DevOps is purely a people-and-process initiative that requires no change to existing tooling or infrastructure. DevOps depends on observable, fast feedback loops. If your build system takes hours to produce results, or your monitoring produces no actionable signal, you cannot iterate quickly enough to benefit from DevOps practices. Cloud-compatible infrastructure, automated testing, and metrics-based observability are prerequisites, not optional extras.

3. Limiting DevOps to Developers and Operations Teams

DevOps does not begin with developers and end with operators. If security, database administration, network engineering, and project management are excluded, you are not eliminating bottlenecks—you are relocating them. Every stakeholder involved in the software delivery lifecycle needs to understand and participate in DevOps principles: shared ownership, self-service tooling, continuous learning, and collaborative problem-solving.

Security is a particularly common omission. Bolting security review onto the end of a pipeline that was designed without it creates exactly the kind of friction DevOps is supposed to eliminate. Integrating security early—sometimes called DevSecOps—is a sign of a mature program.

4. Setting Unrealistic Timelines for Results

DevOps transformations take time. Organizations that expect measurable improvement within weeks of starting a program often become disillusioned and abandon the initiative before it has had a chance to mature. Set realistic, measurable goals tied to leading indicators—deployment frequency, mean time to recovery, change failure rate—rather than lagging business metrics alone. Be explicit with leadership about the timeline for seeing returns.

5. Demanding Process Perfection From the Start

No DevOps program launches perfectly. Expecting otherwise creates risk aversion that undermines the experimentation DevOps depends on. Instead:

• Start DevOps experiments in low-risk areas with limited blast radius.
• Distinguish between productive failure (fast, small, cheap, reversible) and systemic failure (repeated, expensive, preventable).
• Build explicit error budgets or failure budgets so that teams have permission to iterate without fear of punishment.
• Run blameless postmortems after incidents to extract learning rather than assign fault.

The objective is not to prevent all failure—it is to fail in ways that teach you something and are not repeated.

6. Creating a New Silo Instead of Breaking Old Ones

A common organizational response to DevOps adoption is to form a dedicated “DevOps team” that sits alongside existing development and operations teams. This is widely recognized as an anti-pattern. It creates a new silo rather than dissolving the boundaries between existing ones.

A dedicated DevOps team can serve a legitimate purpose as a temporary pilot or center of enablement—a group that builds internal tooling, establishes standards, and coaches product teams. But it should never be the end state. The goal is for every product team to be capable of planning, building, shipping, and running its own software, aligned to a business service or objective.

7. Losing End-to-End Visibility

Automation and self-organization can paradoxically reduce visibility if they are not instrumented properly. When teams cannot see what is happening across the full delivery pipeline—from code commit to production—they cannot identify bottlenecks, respond quickly to incidents, or align their work to business goals.

Effective DevOps programs make the delivery process visible to all stakeholders through shared dashboards, common tooling, and standardized reporting. End-to-end observability is what allows developers, operators, and business stakeholders to work as a coherent system rather than a collection of disconnected hand-offs.

What to Do After a DevOps Failure

When something goes wrong—a deployment causes an outage, a pipeline change delays a release, or a team abandons a new practice—the response matters as much as the failure itself.

1. Contain the immediate impact. Restore service before conducting any analysis. Rollback capability should be built into your pipeline from day one.
2. Run a blameless postmortem. Focus on systemic causes, not individual mistakes. Document what happened, why it happened, and what will change.
3. Identify the failure category. Was this a process gap, a tooling gap, a cultural gap, or a communication gap? The category determines the remedy.
4. Make a specific, testable change. Vague resolutions do not prevent recurrence. Change one thing, measure the effect, and iterate.
5. Share the learning broadly. Postmortems should be visible across the organization. Other teams will face similar problems; give them the benefit of your experience.

Building a More Resilient DevOps Program

Preventing future failure requires sustained investment in three areas:

• People: Invest in training, hiring for collaborative mindset, and creating psychological safety. Engineers who fear blame do not report problems early.
• Process: Document your delivery process end-to-end, identify constraints, and apply continuous improvement systematically rather than reactively.
• Technology: Maintain your toolchain. Outdated or poorly integrated tools create friction and reduce visibility, both of which increase failure rates.

DevOps is not a product you install and run. It is an ongoing organizational capability that requires continuous attention. The organizations that treat it as a transformational program—with executive sponsorship, clear metrics, and a genuine commitment to cultural change—are the ones that eventually see the efficiency and reliability gains DevOps promises.

Summary

DevOps failure is not a question of if—it is a question of when and how badly. The teams that succeed long-term are not the ones that never fail; they are the ones that fail small, learn fast, and systematically eliminate the conditions that caused the failure. Treat each failure as an investment in a more robust engineering organization, and you will find that DevOps delivers exactly what it promises.

The main goal of DevOps is quite simple: ship software updates frequently, reliably, and with better quality. This goal is somewhat “motherhood and apple pie,” since almost every organization will agree that they want to get there. Many will tell you they’ve already embarked on the DevOps journey by following some commonly followed frameworks, such as “CALMS.”

However, very few will express complete satisfaction with the results. After speaking to 200+ DevOps professionals at various stages of the adoption lifecycle, we found that organizations generally fall in one of three categories:

We were most interested in groups two and three since they were actually in the middle of their DevOps journey. When asked to better explain the challenges and roadblocks, here is what we found:

• 68% said that the lack of connectivity between the various DevOps tools in their toolchain was the most frustrating aspect
• 52% said that a large portion of their testing was still manual, slowing them down
• 38% pointed out that they had a mix of legacy and modern applications, i.e. a brownfield environment. This created complexity in terms of deployment strategies and endpoints, toolchain, etc.
• 27% were still struggling with siloed teams that could not collaborate as expected
• 23% had limited access to self-service infrastructure
• Other notable pain points included finding the right DevOps skill set, difficulty managing the complexity of multiple services and environments, lack of budget and urgency, and limited support from executive leadership

Let us look at each of these challenges in greater detail.

#1: Lack of connectivity in the DevOps toolchain
There are many DevOps tools available that help automate different tasks like CI, infrastructure provisioning, testing, deployments, config management, release management, etc. While these have helped tremendously as organizations start adopting DevOps processes, they often do not work well together.

As a classic example, a Principal DevOps engineer whose team uses Capistrano for deployments told us that he still communicates with Test and Ops teams via JIRA tickets whenever a new version of the application had to be deployed, or whenever a config change had to be applied across their infrastructure.

All the information required to run Capistrano scripts was available in the JIRA ticket, which he manually copied over to his scripts before running them. This process usually took several hours and needed to be carefully managed since the required config was manually transferred twice: once when entered into JIRA, and again when he copied it to Capistrano.

This is one simple example, but this problem exists across the entire toolchain.

Smaller organizations get around this problem by writing custom scripts that glue their toolchain together. This works fine for a couple of applications, but quickly escalates to spaghetti hell since these scripts aren’t usually written in a standard fashion. They are also difficult to maintain and often contain tokens, keys and other sensitive information. Worse still, these scripts are highly customized for each application and cannot be reused to easily scale automation workflows.

For most serious organizations, it is an expensive and complex effort to build this homegrown “DevOps glue,” and unless they have the discipline and resources of the Facebooks and Amazons of the world, it ultimately becomes a roadblock for DevOps progress.

Continuous Delivery is very difficult to achieve when the tools in your DevOps toolchain cannot collaborate and you manage dependencies manually or through custom scripts.

Challenge #2: Lack of test automation
Despite all the focus on TDD, most organizations still struggle with automating their tests. If the testing is manual, it is almost impossible to execute the entire test suite for every commit, becoming a barrier for Continuous Delivery. Teams try to optimize this by running a core set of tests for every commit and running the complete test suite only periodically. This means that most bugs are found later in your software delivery workflow and are much more expensive to find and fix.

Test automation is an important part of the DevOps adoption process and hence needs to be a top priority.

Challenge #3: Brownfield environments
Typical IT portfolios are heterogeneous in nature, spanning multiple decades of technology, cloud platform vendors, private and public clouds in their labs and data centers, all at the same time. It is very challenging to create a workflow that spans across these aspects since most tools work with specific architectures and technologies. This leads to toolchain sprawl as each team uses the toolchain best serving their needs.

The rise of Docker has also encouraged many organizations to develop microservices-based applications. This has also increased the complexity for DevOps automation since an application now needs hundreds of deployment pipelines for heterogeneous microservices.

Challenge #4: Cultural problems
Applications evolve across functionals silos. Developers craft software, which is stabilized by QA, and then deployed and operated by IT Operations. Even though all these teams are expected to work together and collaborate, they often have conflicting interests.

Developers are driven to move as fast as they can and build new stuff. QA and Release management teams are driven to be as thorough as possible, making sure no software errors can escape past their watchful eyes. Both teams are often gated by SecOps and Infrastructure Ops, who are incentivized to ensure production doesn’t break.

Governance and compliance also plays a role in slowing things down. Cost centers are under pressure to do more with less, which leads to a culture that opposes change, since change introduces risk and destabilizes things, which means more money and resources are required to manage the impact.

This breakdown across functional silos leads to collaboration and coordination issues, slowing down the flow of application changes.

Some organizations try to address this by making developers build, test and operate software. Though this might work in theory, developers are bogged down by production issues, and a majority of time is spent on operating what they built last month as opposed innovating on new things. Most organizations try to get all teams involved across all phases of the SDLC, but this approach still relies on manual collaboration.

Automation is the best way to broker peace and help Dev and Ops collaborate. But as we see in other challenges, ad-hoc automation itself can slow you down and introduce risk and errors.

Challenge #5: Limited access to self-service infrastructure and environments
For many organizations, virtual machines and cloud computing transformed the process of obtaining the right infrastructure on-demand. What previously took months could now be achieved in a few minutes. IaaS providers like AWS have hundreds of machines with flexible configurations and many options for pre-installed OS and other tools. Tools like Ansible, Chef, Puppet help represent infrastructure-as-code, which further speeds up provisioning and re-provisioning of machines.

However, this is still a problem in many organizations, especially those running their own data centers or those that haven’t embraced the cloud yet.

We Need more from DevOps
A popular DevOps framework describes a CALMS approach, consisting of Culture, Automation, Lean, Measurement and Sharing. The DevOps movement started as a cultural movement, and even today, most implementations focus heavily on culture.

While culture is an important part of any DevOps story, changing organizational culture is the hardest thing of all. Culture forms over a period of time due to ground realities. Ops teams don’t hate change because they are irrational or want to be blockers. Over the years, they’ve taken the heat every time an over-enthusiastic Dev team tried to fast-track changes to production without following every step along the way.

Seating them with the developers might help make the work environment a little friendlier but it doesn’t address the root cause, no matter how many beers they have together.

There Is No Step-by-Step Guide to DevOps Culture

Adopting a DevOps culture is not something an organization can mandate in a single meeting. Declaring “we are doing DevOps now” changes nothing on its own, because the transformation is fundamentally cultural rather than procedural. It requires ongoing conversation, iterative adjustment, and genuine commitment from every level of the business — starting with executive leadership and extending all the way to individual contributors on development and operations teams.

Because every organization has a unique history, structure, and set of incentives, there is no universal roadmap. What works at a cloud-native startup will not map directly onto a decades-old enterprise. The cultural shift must be shaped around the specific realities of each company.

Learning from Failure: A Core DevOps Principle

Among the many cultural shifts that DevOps demands, one stands out as both the most important and the most difficult to internalize: treating failure as a learning opportunity rather than a liability. If an organization does not place genuine value on understanding failures in its processes, tooling, and team practices — and on using those failures to drive continuous improvement — its DevOps initiatives will stall regardless of which tools it adopts or which methodologies it follows.

This is precisely why the phrase “DevOps culture” comes up so frequently in the industry. The tooling — CI/CD pipelines, infrastructure as code, monitoring and observability platforms — is the relatively easy part. The hard part is building teams that can honestly examine what went wrong, share that information openly, and act on it without fear of punishment.

Why Fear of Failure Slows Organizations Down

It is entirely understandable that businesses want to minimize outages, bugs, and service disruptions. Incidents carry real costs: direct financial impact from downtime, customer churn, and reputational damage. But a culture that prioritizes preventing all failure above learning from it creates a subtle and damaging trap.

• Slower iteration: Teams that cannot tolerate failure become risk-averse. They deploy less frequently, run fewer experiments, and fall behind competitors who iterate quickly.
• Hidden problems: When failure is punished, engineers hide mistakes. Small issues that could be caught and corrected early go unreported until they become major incidents.
• Stagnation: Genuine improvement — in software quality, in system reliability, in team capability — requires confronting what is not working. A culture that avoids failure effectively avoids growth.

Attempting to prevent every mistake is not only exhausting — it is practically impossible in complex software systems. The goal should be resilience and rapid recovery, not the illusion of perfection.

Psychological Safety: Creating Space for Honest Conversation

For a failure-positive culture to exist, teams need a safe environment in which to be honest. This concept is sometimes called psychological safety — the shared belief that team members will not be punished or humiliated for speaking up about mistakes, questions, or concerns.

Psychological safety does not mean tolerating carelessness. It means that when something goes wrong, the instinctive response is curiosity rather than blame. It means engineers feel comfortable saying “I made a mistake, and here is what I learned” instead of quietly hoping no one notices.

Building this environment is a leadership responsibility. If leaders respond to incidents with recrimination, the message is clear: silence is safer than honesty. If leaders instead model curiosity — asking “what can we learn from this?” rather than “whose fault is this?” — that tone cascades through the organization.

Blameless Postmortems: Structuring the Learning Process

One of the most practical tools DevOps teams use to institutionalize learning from failure is the blameless postmortem. A postmortem is a structured review conducted after an incident or project milestone, focused on understanding what happened and what can be improved — not on identifying individuals to hold responsible.

Key characteristics of an effective blameless postmortem include:

• A focus on systems, not people: Most failures are the result of process gaps, tooling limitations, or unclear communication — not individual incompetence. Postmortems should examine the system that allowed the failure to occur.
• A clear timeline: Reconstructing what happened, in what order, helps teams understand contributing factors and decision points.
• Documented action items: A postmortem that produces no follow-up actions is a missed opportunity. Concrete, owned action items are what turn analysis into improvement.
• Wide sharing: Postmortems should be readable by the broader team, not locked in a private document. Shared knowledge prevents the same mistakes from recurring elsewhere in the organization.

Rewarding Transparency Over Silence

In organizations where a “heads will roll” mentality prevails, employees quickly learn that the safest behavior is to say nothing when something goes wrong. This incentivizes cover-ups, delays problem resolution, and compounds technical debt. The organization loses the opportunity to learn from every incident that goes unreported.

Some organizations that have successfully embedded DevOps principles take the opposite approach: they actively recognize and reward engineers who surface failures, document them clearly, and propose solutions. This is not about rewarding the failure itself — it is about rewarding the transparency and initiative that turn a failure into a systemic improvement.

When surfacing a problem is professionally safe — or even professionally beneficial — the information that flows to engineering and leadership becomes far richer and more actionable.

Trust Is the Foundation

Everything described above depends on trust. Trust that leadership will respond constructively to bad news. Trust that colleagues will engage with postmortems honestly rather than defensively. Trust that the organization values improvement over the appearance of infallibility.

Trust of this kind cannot be mandated — it is built over time through consistent behavior. Leaders who model vulnerability, who publicly acknowledge their own mistakes and what they learned from them, do more to build a failure-positive culture than any policy document could.

Key Takeaways

• DevOps culture cannot be installed; it must be grown through ongoing behavioral change at every level of the organization.
• Learning from failure is a prerequisite for continuous improvement — not an optional nice-to-have.
• A culture that punishes failure incentivizes silence, which makes systems less reliable over time.
• Blameless postmortems are a practical mechanism for turning incidents into institutional knowledge.
• Psychological safety and organizational trust are the foundations on which all of this depends, and both start with leadership.
• Teams empowered to learn from failure adapt faster, build more reliable systems, and contribute more directly to business outcomes.