This article is based on a case-study piece originally published on devopsonline.co.uk, authored by Adam Pettman of 2i. It examines why testing is frequently deprioritised during digital transformation programmes, how Agile and test automation address that problem, and what a real-world implementation looks like.
The Disaster That Started the Conversation
In April 2018, TSB migrated billions of customer records onto a new core-banking platform. The project had been in planning for over 18 months and involved more than 1,000 professionals. Within 20 minutes of the cutover, customers began reporting missing savings balances and incorrect transaction histories. Weeks of service disruption followed, costing the bank an estimated £330 million and driving approximately 80,000 customers to switch to competitors.
The post-incident analysis pointed to a single, preventable root cause: insufficient rigorous testing before go-live.
TSB is not an isolated case. Across industries, digital leaders facing executive pressure and fixed delivery deadlines routinely compress or skip testing phases—especially when complex legacy systems are involved. The result is almost always the same: defects surface in production, where fixing them is far more expensive and damaging than catching them earlier.
Why Testing Gets Treated as an Afterthought
In traditional waterfall delivery models, testing sits at the end of the pipeline. By the time testers receive a build, the delivery deadline is imminent and any defects discovered are unlikely to be fully resolved before go-live. This creates a self-defeating cycle:
- Testers flag issues late in the cycle.
- Business stakeholders override the concerns because timelines are fixed.
- The product ships with known or unknown defects.
- Incidents occur in production at far greater cost than earlier remediation would have required.
Testing is perceived as the “troublemaker”—the discipline that causes delays rather than the one that prevents them. Reframing that perception is the first step toward more resilient delivery.
How Agile Transforms the Testing Mindset
Agile methodology fundamentally changes when and how testing happens. Rather than a single testing phase at the end of a project, testing is distributed across the entire software development lifecycle (SDLC):
- Testing in sprints: Testers are embedded within cross-functional Agile teams and participate in every sprint, not just the final one.
- Continuous feedback: Each sprint’s test results directly inform the scope and priorities of the next sprint, creating a tightening feedback loop.
- Shared ownership: The whole-team approach means every team member—developer, business analyst, tester—is responsible for product quality, not just the QA function.
- Broader tester skills: Testers in Agile environments need strong communication, collaboration and exploratory-testing skills alongside their technical expertise, because they are active participants in sprint planning and retrospectives.
This shift means that defects are found and resolved when they are cheapest to fix: during development, not after deployment.
Test Automation: The Critical Enabler
Embedding testers into Agile teams is necessary but not sufficient. The volume of testing required to support continuous delivery at pace makes manual-only testing unsustainable. Test automation is what allows teams to move quickly without sacrificing coverage.
What to Automate
- Regression testing: Running the full suite of existing tests against every new build manually is prohibitively slow. Automated regression suites catch regressions overnight or within a CI/CD pipeline run.
- Smoke testing: A fast automated smoke test validates that a new build is stable enough for deeper testing before human effort is invested.
- API and integration tests: Automated contract and integration tests verify that services communicate correctly as codebases evolve.
Freeing testers from repetitive manual regression work allows them to focus on higher-value exploratory testing—the kind of creative, scenario-driven testing that automated scripts cannot easily replicate.
The ROI of Automation
According to the original case study, a large telecoms provider previously required two full-time testers and two weeks to execute 8,000 tests at a cost of approximately £7,000 per cycle. After implementing automated testing, the same test suite ran overnight at a fraction of that cost. The two testers were redeployed to process improvement and exploratory testing—adding direct value to the delivery pipeline rather than executing repetitive checks.
Note: This specific figure comes from the original devopsonline.co.uk source and has not been independently verified.
Test automation requires upfront investment in people, tooling and process design. It also demands sustained discipline: automation suites rot if they are not actively maintained alongside the product they test. Teams that treat automation as a one-time project rather than an ongoing engineering discipline consistently see their suites degrade in value over time.
Dynamic Test Automation: Letting Data Drive the Tests
Standard test automation typically executes test cases in a fixed, predetermined order. Dynamic Test Automation takes a different approach: the test data itself carries indicators that determine which tests run and what the expected outcome should be. This surfaces hidden dependencies that linear test execution would miss.
“If we always work through our automated tests in the order 1, 2, 3, 4, 5 we will never find out what happens if we test 1, 2, 3, 5, 4.”
Adam Pettman, 2i
Because unknown dependencies are more likely to be exposed when test execution order varies, dynamic approaches tend to find defect categories that traditional scripts would miss entirely.
Generating Test Data
Dynamic automation requires quality test data. Live production data is usually unsuitable for test environments for privacy and compliance reasons. Two practical approaches exist:
Approach 1 – Anonymisation
Live data is transformed so that personally identifiable information (PII) is replaced with realistic but fictional equivalents. For example, “Bob Smith, Accountant” becomes “James Brown, Finance Associate.” The data relationships and realistic value distributions are preserved, improving test coverage compared with purely synthetic data. The limitation is scale: anonymising data consistently across large relational databases with many foreign-key relationships is complex and may not be feasible in every context.
Approach 2 – Generating Data from Scratch
Test data is created synthetically, working with Product Owners and business stakeholders to ensure all key scenarios are represented. Brief workshops to define the required data rows require an initial time investment, but that investment pays back quickly: synthetic data that covers edge cases catches defects before User Acceptance Testing (UAT), dramatically reducing the number of surprises at that stage.
Breaking the Data Intentionally
A complete test strategy must verify not only that valid inputs produce correct outputs (the “happy path”) but also that the system handles invalid or unexpected inputs gracefully (the “sad path”). Introducing deliberately malformed or boundary-violating data in a controlled way—sometimes called negative testing or fault injection—validates that the system catches errors correctly rather than silently producing wrong results.
Case Study: Dynamic Test Automation for an Eco-Tech Start-Up
Testing firm 2i partnered with a cloud-native start-up to embed Dynamic Test Automation into their delivery pipeline. The engagement spanned the full team, from CTO to junior developer. The technology choices made in this implementation illustrate how the theory translates into practice.
Generating Test Data with Python
Test data was generated from scratch using Python scripts that produced a series of JSON files ready for upload through the application. This gave the team full control over the data’s structure, volume and variation without any dependency on production data or anonymisation tooling.
Breaking the Data with Python
A separate Python script randomly injected invalid values into the JSON files. Files with injected values were tagged so that the test framework knew to expect failures when processing them—providing a clean mechanism for sad-path testing without manual intervention.
UI Test Automation with Cypress and Cucumber
Cypress was selected as the browser automation framework to interact with the web application and drive test execution. All test cases were written in BDD (Behaviour-Driven Development) format using Cucumber, which allowed test scenarios to be expressed in plain language (Gherkin syntax) while the underlying implementation remained hidden from business stakeholders. This enabled non-technical Product Owners to read, review and even contribute to test case design.
Test cases covered both happy paths (valid JSON files processed successfully) and sad paths (files tagged as “Broken” rejected as expected). The full suite formed the basis of the ongoing regression pack.
Embedding in the CI/CD Pipeline
The complete Dynamic Test Automation suite was integrated into the team’s GitOps CI/CD pipeline. Every commit triggers the test suite automatically, giving the engineering team continuous confidence that new code has not broken existing functionality—without slowing down their deployment velocity.
Where Quality Assurance Is Heading
Test automation will continue to expand its scope. Several trends are already visible:
- Low-code and no-code automation tools are lowering the barrier to entry, enabling teams without deep scripting expertise to build and maintain automated test suites.
- AI-assisted testing is beginning to augment test case generation, defect prediction and visual regression detection—though this area is still maturing.
- Shift-left security testing (integrating security checks earlier in the pipeline) is following the same pattern that functional testing has undergone with Agile.
However, automation does not eliminate the need for skilled testers. Human expertise remains essential for:
- Defining relevant test scenarios and acceptance criteria.
- Exploratory testing that probes the edges of expected behaviour.
- Overseeing automation suite quality and preventing test rot.
- Translating business requirements into testable conditions.
The organisations that will deliver the most reliable software at the highest velocity are those that combine strategic test planning, well-designed test cases, disciplined automation and skilled QA engineers—all embedded directly in the delivery pipeline from the first sprint, not bolted on at the end.
Original article by Adam Pettman, first published on devopsonline.co.uk. Implementation referenced was delivered by 2i.