This article is based on ideas originally published on TechTarget. The arguments below represent an independent explanation and expansion of those concepts for a DevOps engineering audience.
The Myth of Complete Continuous Testing
The popular image of continuous testing is an always-on build server that does far more than compile code. The ideal is a system that spins up ephemeral environments, runs end-to-end scenarios, simulates real user behaviour, and catches every defect before it reaches production. It sounds compelling — and parts of it are genuinely valuable — but the underlying assumption is flawed: complete automated testing is impossible.
Software systems have too many states, too many user paths, and too many environmental variables for any automated suite to cover exhaustively. Treating automation as a substitute for good engineering judgment is, at best, naïve.
The Problem with Mass Inspection
When teams rely heavily on broad, automated test sweeps — sometimes called mass inspection — they are often treating a symptom rather than the underlying disease. The implicit logic is: “We ship imperfect code, so we need a giant safety net to catch the worst problems before they reach users.” That logic has several costly consequences:
- It is expensive. Large, comprehensive test suites require significant infrastructure, maintenance effort, and engineering time to keep meaningful and green.
- It encourages lower first-time quality. When developers know a massive automated suite will catch mistakes, there is less pressure to write well-crafted, resilient code in the first place.
- Automation becomes change detection, not testing. Once a suite matures, it largely detects whether anything changed relative to a previous baseline. But the job of every developer on the team is to introduce change. The two goals are in direct tension.
Mass inspection does not raise software quality — it measures how far it has fallen.
Balanced DevOps Testing: Regression vs. New Feature Testing
The bulk of what most teams call “continuous testing” is actually regression testing: verifying that a code change has not broken existing behaviour. Regression testing is necessary, but it is not sufficient. It answers the question “Did we break something we already had?” — not “Does this new thing actually work correctly?”
Improving first-time code quality — through practices associated with the Software Craftsmanship movement, such as clean architecture, meaningful code review, and pair programming — can reduce the frequency of regressions. But it does not eliminate the need to validate new functionality before it ships.
A practical, balanced approach looks like this:
- Invest in code resilience and craftsmanship to reduce the surface area that regression tests need to cover.
- Automate a targeted subset of new-feature scenarios — the most critical paths — and incorporate those checks into the deployment pipeline.
- Use human exploratory testing for new features early in the development cycle, before automation is written and before code is merged.
Why New Feature Testing Deserves Its Own Practice
New feature testing is qualitatively different from regression testing. A regression test checks a known specification against a known implementation. A new feature test explores unknown territory: Does the feature behave the way the product team intended? Does it handle edge cases a specification document never anticipated? Does it interact with the rest of the system in unexpected ways?
Automated tools are poor at answering these questions early in the lifecycle, because the specification itself may still be evolving. A human tester — one who can read browser developer tools, inspect network requests, and reason about user intent — is far better suited to this stage.
If new feature testing is skipped or done poorly, bad code reaches production. If DevOps simply means “deliver buggy software to production faster,” most teams are already doing it. That is not a standard any engineering organisation, product manager, or customer would accept.
Practical Recommendations for DevOps Teams
1. Involve Human Testers Early
Assign a technically capable tester to work alongside developers during feature development — not after a feature is merged. This person does not need to be a senior developer, but should be comfortable using browser developer tools, reading basic logs, and articulating what “correct” looks like for a user. Some high-performing teams use a pairing model: one person focused mostly on programming, one focused mostly on exploratory testing, working on the same feature together.
2. Distinguish Exploration from Regression
Keep exploratory new feature testing conceptually separate from your regression suite. Exploratory testing is wide-ranging and human-driven. Regression testing is narrow, repeatable, and automated. Conflating the two leads teams to automate too early (before the behaviour is stable) or to rely on humans for work that tools handle efficiently.
3. Automate Critical Paths, Not Everything
After a new feature is validated by human testers and the behaviour is understood and stable, identify the handful of scenarios that represent the highest business risk. Automate those specific paths and add them to the pipeline. Resist the urge to achieve exhaustive automation coverage — the maintenance burden usually outweighs the benefit.
4. Still Run End-to-End Checks Before Production
Even with strong new feature testing and good craftsmanship practices, end-to-end user simulation tests before rollout remain valuable. The goal is not to eliminate them but to make them leaner and more targeted, so they confirm integration rather than compensate for unknown functionality.
The Bigger Picture: Test the Right Things the Right Way
The rush toward automation and continuous testing can obscure the fundamental purpose of testing: to ship better software. When testing is treated as a monolithic, always-on process, teams lose sight of why each type of testing exists and when it adds the most value.
Breaking testing into its component parts — new feature testing, exploratory testing, regression testing, integration testing, performance testing — makes it easier to ask: “Are we applying the right technique at the right stage for the right reason?” New feature testing, in particular, deserves deliberate, human attention early in the cycle. Give it that attention, and the rest of your testing pipeline becomes leaner, faster, and more meaningful.
Original ideas sourced from an article published on TechTarget (techtarget.com).