The Disaster That Started the Conversation

In April 2018, TSB migrated billions of customer records onto a new core-banking platform. The project had been in planning for over 18 months and involved more than 1,000 professionals. Within 20 minutes of the cutover, customers began reporting missing savings balances and incorrect transaction histories. Weeks of service disruption followed, costing the bank an estimated £330 million and driving approximately 80,000 customers to switch to competitors.

The post-incident analysis pointed to a single, preventable root cause: insufficient rigorous testing before go-live.

TSB is not an isolated case. Across industries, digital leaders facing executive pressure and fixed delivery deadlines routinely compress or skip testing phases—especially when complex legacy systems are involved. The result is almost always the same: defects surface in production, where fixing them is far more expensive and damaging than catching them earlier.

Why Testing Gets Treated as an Afterthought

In traditional waterfall delivery models, testing sits at the end of the pipeline. By the time testers receive a build, the delivery deadline is imminent and any defects discovered are unlikely to be fully resolved before go-live. This creates a self-defeating cycle:

• Testers flag issues late in the cycle.
• Business stakeholders override the concerns because timelines are fixed.
• The product ships with known or unknown defects.
• Incidents occur in production at far greater cost than earlier remediation would have required.

Testing is perceived as the “troublemaker”—the discipline that causes delays rather than the one that prevents them. Reframing that perception is the first step toward more resilient delivery.

How Agile Transforms the Testing Mindset

Agile methodology fundamentally changes when and how testing happens. Rather than a single testing phase at the end of a project, testing is distributed across the entire software development lifecycle (SDLC):

• Testing in sprints: Testers are embedded within cross-functional Agile teams and participate in every sprint, not just the final one.
• Continuous feedback: Each sprint’s test results directly inform the scope and priorities of the next sprint, creating a tightening feedback loop.
• Shared ownership: The whole-team approach means every team member—developer, business analyst, tester—is responsible for product quality, not just the QA function.
• Broader tester skills: Testers in Agile environments need strong communication, collaboration and exploratory-testing skills alongside their technical expertise, because they are active participants in sprint planning and retrospectives.

This shift means that defects are found and resolved when they are cheapest to fix: during development, not after deployment.

Test Automation: The Critical Enabler

Embedding testers into Agile teams is necessary but not sufficient. The volume of testing required to support continuous delivery at pace makes manual-only testing unsustainable. Test automation is what allows teams to move quickly without sacrificing coverage.

What to Automate

• Regression testing: Running the full suite of existing tests against every new build manually is prohibitively slow. Automated regression suites catch regressions overnight or within a CI/CD pipeline run.
• Smoke testing: A fast automated smoke test validates that a new build is stable enough for deeper testing before human effort is invested.
• API and integration tests: Automated contract and integration tests verify that services communicate correctly as codebases evolve.

Freeing testers from repetitive manual regression work allows them to focus on higher-value exploratory testing—the kind of creative, scenario-driven testing that automated scripts cannot easily replicate.

The ROI of Automation

According to the original case study, a large telecoms provider previously required two full-time testers and two weeks to execute 8,000 tests at a cost of approximately £7,000 per cycle. After implementing automated testing, the same test suite ran overnight at a fraction of that cost. The two testers were redeployed to process improvement and exploratory testing—adding direct value to the delivery pipeline rather than executing repetitive checks.

Note: This specific figure comes from the original devopsonline.co.uk source and has not been independently verified.

Test automation requires upfront investment in people, tooling and process design. It also demands sustained discipline: automation suites rot if they are not actively maintained alongside the product they test. Teams that treat automation as a one-time project rather than an ongoing engineering discipline consistently see their suites degrade in value over time.

Dynamic Test Automation: Letting Data Drive the Tests

Standard test automation typically executes test cases in a fixed, predetermined order. Dynamic Test Automation takes a different approach: the test data itself carries indicators that determine which tests run and what the expected outcome should be. This surfaces hidden dependencies that linear test execution would miss.

“If we always work through our automated tests in the order 1, 2, 3, 4, 5 we will never find out what happens if we test 1, 2, 3, 5, 4.”

Adam Pettman, 2i

Because unknown dependencies are more likely to be exposed when test execution order varies, dynamic approaches tend to find defect categories that traditional scripts would miss entirely.

Generating Test Data

Dynamic automation requires quality test data. Live production data is usually unsuitable for test environments for privacy and compliance reasons. Two practical approaches exist:

Approach 1 – Anonymisation

Live data is transformed so that personally identifiable information (PII) is replaced with realistic but fictional equivalents. For example, “Bob Smith, Accountant” becomes “James Brown, Finance Associate.” The data relationships and realistic value distributions are preserved, improving test coverage compared with purely synthetic data. The limitation is scale: anonymising data consistently across large relational databases with many foreign-key relationships is complex and may not be feasible in every context.

Approach 2 – Generating Data from Scratch

Test data is created synthetically, working with Product Owners and business stakeholders to ensure all key scenarios are represented. Brief workshops to define the required data rows require an initial time investment, but that investment pays back quickly: synthetic data that covers edge cases catches defects before User Acceptance Testing (UAT), dramatically reducing the number of surprises at that stage.

Breaking the Data Intentionally

A complete test strategy must verify not only that valid inputs produce correct outputs (the “happy path”) but also that the system handles invalid or unexpected inputs gracefully (the “sad path”). Introducing deliberately malformed or boundary-violating data in a controlled way—sometimes called negative testing or fault injection—validates that the system catches errors correctly rather than silently producing wrong results.

Case Study: Dynamic Test Automation for an Eco-Tech Start-Up

Testing firm 2i partnered with a cloud-native start-up to embed Dynamic Test Automation into their delivery pipeline. The engagement spanned the full team, from CTO to junior developer. The technology choices made in this implementation illustrate how the theory translates into practice.

Generating Test Data with Python

Test data was generated from scratch using Python scripts that produced a series of JSON files ready for upload through the application. This gave the team full control over the data’s structure, volume and variation without any dependency on production data or anonymisation tooling.

Breaking the Data with Python

A separate Python script randomly injected invalid values into the JSON files. Files with injected values were tagged so that the test framework knew to expect failures when processing them—providing a clean mechanism for sad-path testing without manual intervention.

UI Test Automation with Cypress and Cucumber

Cypress was selected as the browser automation framework to interact with the web application and drive test execution. All test cases were written in BDD (Behaviour-Driven Development) format using Cucumber, which allowed test scenarios to be expressed in plain language (Gherkin syntax) while the underlying implementation remained hidden from business stakeholders. This enabled non-technical Product Owners to read, review and even contribute to test case design.

Test cases covered both happy paths (valid JSON files processed successfully) and sad paths (files tagged as “Broken” rejected as expected). The full suite formed the basis of the ongoing regression pack.

Embedding in the CI/CD Pipeline

The complete Dynamic Test Automation suite was integrated into the team’s GitOps CI/CD pipeline. Every commit triggers the test suite automatically, giving the engineering team continuous confidence that new code has not broken existing functionality—without slowing down their deployment velocity.

Where Quality Assurance Is Heading

Test automation will continue to expand its scope. Several trends are already visible:

• Low-code and no-code automation tools are lowering the barrier to entry, enabling teams without deep scripting expertise to build and maintain automated test suites.
• AI-assisted testing is beginning to augment test case generation, defect prediction and visual regression detection—though this area is still maturing.
• Shift-left security testing (integrating security checks earlier in the pipeline) is following the same pattern that functional testing has undergone with Agile.

However, automation does not eliminate the need for skilled testers. Human expertise remains essential for:

• Defining relevant test scenarios and acceptance criteria.
• Exploratory testing that probes the edges of expected behaviour.
• Overseeing automation suite quality and preventing test rot.
• Translating business requirements into testable conditions.

The organisations that will deliver the most reliable software at the highest velocity are those that combine strategic test planning, well-designed test cases, disciplined automation and skilled QA engineers—all embedded directly in the delivery pipeline from the first sprint, not bolted on at the end.

Original article by Adam Pettman, first published on devopsonline.co.uk. Implementation referenced was delivered by 2i.

Automation and QA: Not the Same Thing

A common misconception in DevOps is that rising levels of test automation will eventually make QA engineers redundant. The logic sounds reasonable on the surface: if machines can run thousands of test cases faster and more consistently than humans, why maintain a dedicated QA function at all?

The reality is more nuanced. Automation handles the mechanical execution of tests, but it cannot replace the strategic thinking, contextual judgment, and user empathy that skilled QA engineers bring to the software development lifecycle.

“Test automation may largely eliminate the need for manual testing in some scenarios, but it will never eliminate the need for QA.”

— Chris Marsh, Director of Technology, AKQA

Marsh points out that automation becomes one tool in a QA engineer’s toolkit rather than a wholesale replacement for the role. Unit tests are relatively cheap to write and execute, making them strong candidates for automation. Integration tests and UI tests, however, often require more manual oversight because they touch complex, context-sensitive user interactions that are difficult to fully script in advance.

The Zero-Defect Trap

Traditional QA philosophy often aims to eliminate every defect before a release goes live. While that goal is well-intentioned, it carries a real cost: delaying the feedback loop with actual users and reducing return on investment. No production software is truly defect-free, and chasing that standard can prevent teams from shipping working features that deliver real value.

QA engineers need to be embedded in the build pipeline from the start, advising on quality standards across the entire project lifecycle rather than acting as a final gate at the end of the process. Shifting quality thinking earlier — sometimes called “shift-left testing” — allows teams to catch issues when they are cheapest to fix.

A Test Is Only as Good as Its Design

Automation raises a fundamental dependency: the quality of your automated tests determines the quality of your feedback. An automated suite that tests the wrong things, or that has grown stale as the codebase evolved, provides a false sense of confidence rather than genuine assurance.

“Because technology development is a complex, dynamic process, automated QA is doomed to fail unless someone is able to make sure that the tests are current — or new bugs and vulnerabilities will not be detected.”

— Greg Hoffer, VP of Engineering, Globalscape

Hoffer highlights an important failure mode: a serious security vulnerability can slip through an automated QA process if no test was ever written to check for that specific condition. Automated tests only cover the scenarios their authors anticipated. Novel bugs, edge cases, and security issues that emerge from unexpected usage patterns require human investigation to uncover.

There is also a usability dimension that pure automation cannot address. A fully automated test suite can confirm that software behaves exactly as specified while that software remains entirely unusable by real people. Automated checks validate functional correctness against predefined criteria — they cannot assess whether the product actually meets human needs or delivers a coherent user experience.

“Automated QA, continuous integration (CI), and continuous deployment (CD) are all great advances in the efficiency of DevOps. But we should not expect them to be perfect. It is still incumbent on the developer community to be vigilant.”

— Greg Hoffer, VP of Engineering, Globalscape

QA May Actually Become More Strategic, Not Less Relevant

Rather than shrinking the QA function, automation is reshaping it. As CI/CD pipelines absorb routine regression testing, QA engineers can focus on higher-value work: exploratory testing, risk analysis, test strategy, and quality advocacy across the organization.

“As confidence in CI and automation increases, there is a very likely scenario of customers using Continuous Delivery for selective parts of their applications, although it still is not something that is completely reliable.”

— Rupinder Singh, Senior Vice President, Expert Services, Software AG

Singh observes that as automation handles manual test cycles, the QA role gains visibility and importance in technical communities. The engineer who can design a sustainable, maintainable automated test strategy — and know when to supplement it with human judgment — becomes a critical asset rather than a legacy overhead.

Human Creativity Remains the Missing Ingredient

Automated tests are excellent at verifying that known paths through an application still work and that new code has not broken existing behavior. What they cannot do is invent new test scenarios, reason about unexpected user behavior, or ask the question: “What happens if someone uses this feature in a way we never anticipated?”

“It still takes human creativity and ingenuity to explore those paths, and then write automated tests against expected outputs. Companies must — and should — continue to employ QA teams, and they need to invest in training and software licenses for the automation platforms.”

— Mark Doyle, Software Architect, Collabroscape

Doyle also identifies a compounding benefit of well-designed automated testing: when failed tests automatically create entries in a defect tracking system, QA engineers can use that data to build more comprehensive and targeted test plans over time. Automation feeds information back to humans, who then improve the automation — a virtuous cycle that requires active human participation to sustain.

Practical Benefits of Test Automation in a QA-Led Pipeline

• Faster build validation: Running an automated suite before manual QA begins saves engineers from investing time in a broken build.
• Consistent regression coverage: Automated tests reliably re-check existing functionality every time code changes, catching regressions that manual spot-checking might miss.
• Freed capacity for exploratory work: Removing routine test execution from human queues allows QA engineers to focus on complex, judgment-intensive scenarios.
• Automatic defect logging: Integration between CI pipelines and issue trackers reduces manual overhead and ensures failures are captured immediately.
• More stable software over time: Consistent automated coverage enforces quality gates that accumulate stability across releases.

The Bottom Line

Test automation is a force multiplier for QA, not a replacement for it. It handles repetitive, well-defined test scenarios at scale and speed no human team can match. But the strategic design of what to test, the maintenance of test suites as software evolves, the evaluation of usability, and the exploration of unanticipated failure modes all require human expertise.

DevOps teams that treat automation as a substitute for QA judgment risk shipping software that passes every test yet still fails its users. The more mature approach is to invest in QA engineers who understand automation deeply — and know precisely where automation ends and human insight must begin.