This article is based on a talk delivered by Ben Riley, DevOps specialist at CA Technologies, and Richard Jordan, practice manager at Nationwide Building Society, at Computing’s annual DevOps Summit. Original source: computing.co.uk.
Testing Has Not Changed — But DevOps Demands More of It
Ben Riley opened his talk on continuous testing with a simple observation: “Everyone in this room has tested something.” The fundamentals of testing — defining an idea, writing requirements, and validating behaviour — have remained largely unchanged for 30 years. What has changed is the pace at which software must be delivered, and the volume of changes that must be validated.
At Nationwide Building Society, the formal DevOps journey began roughly 18 months before the summit, though Richard Jordan noted that informal DevOps practices had been in place for at least five years. The key insight from their experience: adopting DevOps did not reduce the need for testing. It increased it. Continuous testing became the primary method of measuring delivery progress.
Two Testing Models: Ice Creams vs. Volcanoes
Riley and Jordan used two memorable shapes to describe contrasting testing philosophies:
The Ice Cream Model (Traditional)
The traditional testing pyramid is often described as an inverted cone — wide at the top and narrow at the bottom. Riley and Jordan call this the “ice cream” shape. It is characterised by:
- A large base of manual UI and integration testing
- A small proportion of automated unit tests at the narrow end
- Slow feedback cycles because manual testing dominates
- Poor repeatability and scalability at the UI layer
Traditional manual UI testing is inherently slow and difficult to run consistently at scale. When defects are found late in the cycle, they are significantly more expensive to fix.
The Volcano Model (Automation-First)
The “volcano” is the ice cream flipped upside down. The wide base represents a large suite of automated unit tests, with progressively smaller layers of:
- Unit tests — the vast majority, fast and cheap to run
- Component tests — validating individual services or modules in isolation
- API and integration tests — verifying interactions between components
- GUI tests — limited automated checks at the interface layer
- Manual session-based testing — the narrow peak, reserved for exploratory and edge-case scenarios
This approach encourages development teams to write and own their automated tests — “quick to build, stable to run” — so that failures surface early, when bugs are fast and cheap to fix. The guiding principle is fail fast and fail early.
Why the Volcano Model Suits DevOps
Continuous integration and continuous delivery pipelines depend on rapid, reliable feedback. A test suite dominated by slow manual steps creates a bottleneck that undermines the entire DevOps value stream. The volcano model aligns directly with DevOps principles:
- Speed: Automated unit and component tests run in seconds or minutes, not hours or days.
- Repeatability: Automated tests produce consistent results across environments, reducing flakiness and human error.
- Shift-left: Pushing testing responsibility toward developers and earlier pipeline stages means defects are caught before they compound.
- Scalability: Thousands of automated tests can run in parallel on every commit; manual testing cannot scale the same way.
The Cultural Challenge: Getting Developers to Test
Moving from ice cream to volcano is not purely a tooling problem — it is a cultural one. At Nationwide, Jordan encountered pushback from developers who were now being asked to write and maintain their own automated tests, a responsibility that had traditionally belonged to a separate QA team.
Jordan’s approach combined incentive and expectation:
- Carrot: Offering developers new, modern testing tools that make the work less painful
- Stick: Being clear that the old model is no longer viable and that this change is non-negotiable
Embedding an automated-testing mindset from the start of a project — rather than retrofitting it — is essential. At the same time, Nationwide’s team was actively addressing technical and design debt that made legacy systems harder to test automatically.
There Is No Big Bang — Only Incremental Progress
Nationwide’s transition away from the ice cream model was, by Jordan’s own admission, far from complete at the time of the summit. “There won’t be a big bang, but we’ll chip away at it,” he said. This is a realistic and honest assessment that most organisations will recognise.
Shifting a testing culture in a large financial institution takes time. Legacy test suites, existing toolchains, and established team habits do not change overnight. The practical advice from Riley and Jordan is to make incremental improvements: replace the next manual test with an automated one, instrument the next new service from day one, and steadily move the balance of effort from the top of the ice cream to the base of the volcano.
Key Takeaways
- Automated testing is not a nice-to-have in DevOps — it is the feedback engine that makes continuous delivery possible.
- The “volcano” model (automation-heavy at the unit level, manual testing reserved for exploratory work) is better suited to DevOps pipelines than the traditional manual-heavy “ice cream” model.
- Shifting testing left means developers must own automated tests. This requires both cultural change and the right tooling support.
- Incremental progress beats waiting for a perfect solution. Start automating the next test, not all tests at once.
- Technical debt and testability debt must be addressed together, or automated coverage will remain limited.