This article is based on a perspective originally published on TechTarget.com, exploring how the DevOps movement should orient itself around software resilience rather than traditional reliability.
The Reliability Trap
Many organisations have historically prided themselves on rigorous internal controls for software delivery. Developers would raise work orders, get every approval box checked, and document every step required to move code into production. On the surface, this sounds disciplined—even responsible. In practice, it creates a serious bottleneck.
Because every change required the same heavyweight process, teams were incentivised to batch changes together into large projects and deploy infrequently. When something did go wrong—and it always does—systems went down, and any emergency fix still had to pass through the same slow, documented process. The single-minded focus on reliability (preventing failure) paradoxically made systems more fragile: bigger batches meant bigger blast radii, and slower recovery meant longer outages.
What DevOps Actually Changes
DevOps is often associated with CI/CD pipelines and automation tooling, but the more fundamental shift is cultural: development and operations teams align around a shared goal. The question is what that goal should be.
The example above makes it clear that the goal cannot simply be reliability in its traditional sense—defined as the absence of failure. Failures will happen. Networks drop. Deployments introduce bugs. Third-party services time out. A goal of zero failures is not only unattainable; chasing it leads to the batch-and-freeze anti-pattern described above.
The better goal is software resilience: designing systems so that the failure of any individual component does not bring the whole system down.
Reliability vs. Resilience: A Practical Distinction
Software testing advocate Noah Sussman has argued that software systems should shift their focus from reliability to resilience. The distinction is important:
- Reliability is concerned with preventing failure. It asks: how do we stop things from going wrong?
- Resilience is concerned with containing failure. It asks: when something goes wrong—and it will—how do we ensure the system as a whole keeps functioning?
Reliability thinking often leads to risk aversion, infrequent releases, and large change batches. Resilience thinking leads to smaller, safer deployments, better monitoring, and architectures designed to tolerate partial failure.
Resilience in Practice: The E-Commerce Example
Consider how a large e-commerce homepage is typically constructed. Rather than one monolithic page rendered by a single service, the page is assembled from many independent components: a navigation bar, a personalised recommendations widget, a recently browsed items list, promotional banners, and so on.
Each component makes its own call to a backend service. If one of those services is temporarily unavailable—perhaps because a new deployment is rolling out at that exact moment—only that one component fails to render. The rest of the page loads normally. A customer may not even notice the missing widget.
This is software resilience in action. No single service failure cascades into a full site outage. The architecture is designed around the assumption of partial failure, not the hope of preventing it.
The Five Principles of Software Resilience for DevOps Teams
Translating this mindset into engineering practice produces a concrete set of principles:
- Stop optimising purely for reliability. Accept that failures will occur, and design accordingly rather than trying to prevent every possible failure mode.
- Build systems that are resilient by design. Use patterns such as circuit breakers, bulkheads, graceful degradation, and timeouts to isolate failure domains.
- Make smaller, more frequent changes. Small changes are easier to reason about, easier to test, and—critically—easier to roll back when something goes wrong.
- Deploy incrementally. Roll out changes to a small percentage of traffic or a subset of servers at a time. Blue/green deployments, canary releases, and feature flags all support this approach.
- Design for component-level failure. No single component failure should halt the system. Decouple services, use asynchronous communication where appropriate, and define clear fallback behaviours.
Resilience Requires Observability
Shifting to a resilience mindset also changes how teams think about monitoring. If failures are expected, you need to detect and respond to them quickly. Mean time to recovery (MTTR) becomes a more meaningful metric than mean time between failures (MTBF).
This drives investment in observability: structured logging, distributed tracing, alerting on symptoms rather than just causes, and dashboards that surface component-level health in real time. Teams that can detect a partial failure within seconds and deploy a fix within minutes are far more resilient than teams that prevent failures by deploying quarterly.
The Virtuous Cycle DevOps Creates
When DevOps teams orient around resilience rather than reliability, the vicious cycle described at the start reverses:
- Smaller changes reduce deployment risk, so teams deploy more often.
- Deploying more often reduces the size of each change, making rollbacks trivial.
- Better monitoring surfaces problems faster, reducing the impact of any failure that does occur.
- Resilient architectures contain failures, so the pressure to make every deployment perfect decreases.
- Lower pressure means fewer bureaucratic controls, faster delivery, and more experimentation.
This is the practical payoff of adopting software resilience as the real goal of DevOps: faster delivery, safer systems, and engineering teams that can respond to reality as it is—not as a work order imagined it would be.
Key Takeaways
- Traditional reliability thinking—focused on preventing failure—often leads to infrequent, high-risk deployments.
- Software resilience reframes the goal: design systems that tolerate component failures gracefully.
- Small, incremental deployments and resilient architectures (circuit breakers, graceful degradation) are the core engineering practices.
- MTTR is a more actionable metric than MTBF in a resilience-oriented culture.
- DevOps succeeds when dev and ops align on resilience as their shared objective—not just on tooling.