This article is based on reporting from TechTarget covering presentations at the Velocity Conference 2017 in New York.
The Problem with Root Cause Analysis in a Microservices World
For decades, root cause analysis (RCA) has been the default framework for IT incident response. When an outage occurred in a monolithic application environment, teams would trace the failure back to a specific line of code, a failed switch port, or a faulty hard drive — one identifiable trigger that set off a domino effect. Fix the root cause, close the ticket.
That model is increasingly mismatched with how modern software is built. As applications decompose into microservices distributed across complex network infrastructure, the idea that a single root cause exists — and can be quickly identified during an active incident — breaks down. That was a central theme at the Velocity Conference 2017 in New York, where multiple DevOps engineers and engineering leaders called for a fundamental rethink of incident response philosophy.
Mitigation First, Diagnosis Later
Kristopher Beevers, founder and CEO of hosted DNS provider NS1, offered a direct reframing: focus on mitigation, not root cause, when a crisis begins.
“Identify and troubleshoot the service impact and worry about doing a diagnosis later.”
— Kristopher Beevers, NS1
This is a meaningful shift in priority ordering. Traditional RCA encourages responders to understand why something broke before restoring service. The emerging view separates those two activities: restore service first, investigate cause afterward — and accept that the investigation may not yield a single definitive answer.
Start at the Top of the Stack, Not the Bottom
Sarah Wells, principal engineer at the Financial Times, described a practical implementation of this philosophy. Her team has deliberately tuned out low-level infrastructure noise in favor of business-relevant service signals.
“We’ve turned off alerts for service-level errors, status codes and response times. We start [our analysis] at the top of the stack.”
— Sarah Wells, Financial Times
Her example was concrete: if the homepage list of top articles on the Financial Times website is broken, that is what matters to users and the business. That is where incident response should begin — not with a deep dive into infrastructure metrics that may have no direct relationship to visible service degradation.
This top-down approach aligns alert prioritization with user experience rather than system internals, which is especially valuable when system internals are themselves distributed across dozens or hundreds of microservices.
APIs as Boundaries of Responsibility and Observation
Mark McBride, founder of Turbine Labs and a former engineer at Twitter, Nest Labs, and Google, offered a structural argument for why microservice architectures naturally reduce the importance of traditional RCA.
At scale, members of the same IT organization become consumers of microservices offered by other teams via APIs. No single engineer understands everything behind every API their service calls. Nor do they need to — what they can observe and reason about is the behavior of the API itself: its latency, error rate, and response patterns.
This has two practical effects on incident response:
- Reduced anxiety: Individual responders are not expected to diagnose every upstream dependency. They own their API surface and can observe it directly.
- Common control plane: APIs provide a shared point of observation and remediation that spans different programming languages and runtimes, enabling global mitigations without requiring deep knowledge of every subsystem.
As McBride noted, this structure diminishes the need to find the specific element of infrastructure behind an API that malfunctioned — because the API boundary itself becomes the unit of recovery.
The Human Cost of Root Cause Analysis
Beyond the technical mismatch, several speakers highlighted a cultural problem with RCA: it encourages blame.
“It’s common for a human to be blamed as the root cause of a problem. Then companies end up firing people instead of improving things, and ‘cover your [butt]’ becomes the priority instead of solving problems.”
— Baron Schwartz, co-founder and CEO, VividCortex
When the answer to “what was the root cause?” becomes “an engineer made a mistake,” the process has failed organizationally even if it succeeded analytically. The resulting culture discourages transparency, slows learning, and pushes engineers toward self-protection rather than systemic improvement.
This is not a new observation — blameless postmortems have been a pillar of Site Reliability Engineering culture for years — but the Velocity speakers connected it directly to the argument against RCA as a primary incident response methodology.
From Postmortem to Incident Review: A Change in Language and Mindset
Peter Nealon, a solutions architect at Runkeeper (a mobile running app owned by ASICS), articulated the shift in post-incident analysis goals:
“You often don’t arrive at a single thing, and it’s more important to understand the context, contributing factors, and how to mitigate and remediate the issue.”
— Peter Nealon, Runkeeper
Nealon also flagged a symbolic detail: some DevOps engineers are abandoning the term postmortem in favor of incident review. The language is intentional.
“We’re learning how to improve versus figuring out what died and why. We aren’t aiming for root cause analysis, but to reduce our mean time to remediation.”
— Peter Nealon, Runkeeper
Mean time to remediation (MTTR) as the primary metric — rather than the ability to name a root cause — reorients the entire incident lifecycle toward business outcomes: restoring service quickly and preventing recurrence through systemic improvement.
Incident Commander Model: Structured Roles Under Pressure
Nealon described a concrete organizational change his team was implementing at the time: an incident commander role to direct troubleshooting during active incidents.
In the Runkeeper model:
- Application developers handle hands-on troubleshooting tasks, operating on the assumption that most incidents originate in application code rather than infrastructure.
- Site reliability engineers (SREs) act as subject-matter experts on the underlying platform, available to consult but not the default first responders.
- The incident commander coordinates communication, assigns tasks, and maintains situational awareness across the team.
This separation of concerns mirrors patterns from other high-stakes operational environments and reduces the cognitive load on any single responder during a crisis.
Key Takeaways for DevOps Teams
The arguments made at Velocity 2017 coalesce around a few actionable principles:
- Prioritize service restoration over root cause identification during an active incident. Diagnosis can wait; user impact cannot.
- Align alerting with business-critical services, not low-level infrastructure metrics. Start at the top of the stack.
- Use API boundaries as units of observation and recovery in microservice environments. You do not need to diagnose what is behind an API to remediate through it.
- Replace blame-oriented RCA with blameless incident reviews that focus on context, contributing factors, and systemic improvements.
- Track mean time to remediation as the primary post-incident metric, not the ability to name a single root cause.
- Define clear incident roles — such as an incident commander — to reduce chaos and duplicate effort during response.
These practices do not mean abandoning post-incident investigation entirely. They mean separating the when and the goal of that investigation: after service is restored, with the purpose of improving systems rather than assigning fault.