Testing in Production: From Developer Joke to Cloud-Native Best Practice

Not long ago, a developer who told their manager “we test in production” was guaranteed to end the conversation immediately. The phrase was so obviously reckless it functioned as a joke. Today, for teams running cloud-native, microservices-based applications, testing in production is increasingly treated as a rational—and sometimes unavoidable—part of the software delivery lifecycle.

Why Monolithic Testing Assumptions No Longer Hold

For most of the past two decades, the Continuous Integration/Continuous Delivery (CI/CD) pipeline was the canonical place to validate software changes. This worked well when applications were largely monolithic. The scope of change was bounded, the desired end-state was well-defined, and automated tests in the pipeline could verify it before anything reached users.

Microservices architectures break that assumption. When an application is composed of dozens or hundreds of independent services, the pipeline cannot build and test all of them simultaneously. At any given moment, some services are being rewritten, some are deploying, and others are already running in production—each at a different version. The concept of a single, stable “desired state” for the whole system becomes difficult to define, let alone test for.

This is not just a tooling problem. It is a fundamental shift in where change validation has to happen.

Production as the Only Place Where the Full System Exists

In a microservices environment, the production environment is often the only place where all services are running together in their current state. Pre-production environments are expensive to keep fully representative, and they typically lag behind the real thing. This makes production the most realistic—sometimes the only realistic—environment for certain classes of testing.

That does not mean shipping untested code and hoping for the best. It means combining several disciplines:

• Observability: Instrumenting services so that failures surface quickly and can be understood in context, not just detected.
• Progressive delivery: Techniques such as feature flags, canary deployments, and traffic shadowing limit blast radius when something goes wrong.
• Fast rollback: The ability to revert or re-route traffic quickly reduces the cost of a failure that does reach users.
• Small, frequent changes: Deploying smaller increments makes it easier to isolate the cause of a regression and reduces the risk of any single deployment.

With the right combination of these practices, production testing is not reckless—it is disciplined risk management applied to an environment that cannot be fully replicated elsewhere.

Observing State vs. Validating Change

Two complementary philosophies are converging here, and it helps to keep them distinct.

Observability: Understanding What Is Happening Now

Tools and vendors in the observability space—such as Honeycomb—argue that in sufficiently complex distributed systems, most failures result from the convergence of multiple simultaneous conditions rather than a single identifiable bug. Traditional monitoring, which checks for known failure modes, cannot catch these emergent problems. Observability tooling instead gives engineers the ability to ask arbitrary questions of their running system and understand its internal state from its external outputs.

This is observing state: understanding what the system is doing right now. It is powerful, but it does not by itself tell you whether a specific change you just deployed is safe.

CI/CD: Validating Change as It Moves Through the System

CI/CD pipelines are built to validate change: each commit triggers a sequence of builds, automated tests, and deployment steps that provide confidence before code reaches users. Rob Zuber, CTO of CircleCI, described this distinction in an eBook published in early 2020, arguing that the CI/CD tool is well-positioned at the center of the delivery stack precisely because it understands change—what changed, when, and what the test results were.

According to Zuber, CircleCI itself uses its own CI/CD platform alongside complementary tools for crash reporting, observability, and centralised logging. No single tool covers the full picture; the CI/CD system acts as the anchor that ties change context to the operational data coming from observability tools.

Putting Change and State Together

The mature position, reflected in both the observability camp and the CI/CD camp, is that neither approach alone is sufficient. Effective quality management for cloud-native applications requires:

1. Good coding and review practices to prevent avoidable defects from entering the pipeline at all.
2. Source control management that gives every change a clear audit trail.
3. Automated testing in CI/CD to catch regressions before deployment.
4. Progressive delivery mechanisms to limit exposure when untested interactions emerge in production.
5. Observability in production to detect and diagnose problems that only manifest under real traffic and real data.
6. Fast incident response and rollback to contain blast radius.

These are not competing approaches. They are layers of a single strategy, each addressing the risks the others cannot fully cover.

What This Means for DevOps Teams

If your organisation is moving toward microservices or cloud-native architectures, the testing strategy that served you well with monoliths will develop gaps. Some practical steps worth evaluating:

• Audit your pre-production environments. How representative are they of production? Where are the gaps, and what risks do those gaps create?
• Invest in distributed tracing and structured logging. These are the foundation of meaningful observability in a microservices environment.
• Adopt feature flags. Decoupling deployment from feature release gives you a fast, low-risk lever when something goes wrong in production.
• Make deployments smaller and more frequent. Counter-intuitively, deploying more often—in smaller increments—reduces the risk of each individual deployment and makes problems easier to isolate.
• Instrument your CI/CD pipeline to emit change context. When your observability tooling can correlate a spike in errors with a specific deployment, mean time to resolution drops significantly.

Conclusion

Testing in production is no longer a euphemism for negligence. For teams operating distributed, cloud-native systems, it is an acknowledgement that production is the only environment where the full system—with its real dependencies, real data volumes, and real user behaviour—can be observed. The goal is not to skip testing; it is to extend testing into the environment where the most important unknowns actually live, with the observability and delivery engineering discipline to do so safely.

Cloud-native development is transforming every part of the software delivery lifecycle. Testing is one of the most consequential areas of that transformation, and teams that adapt their quality strategy accordingly will be better positioned to move fast without accumulating unmanageable risk.