This article is based on a post originally published by The Guardian’s engineering team describing how they adapted their quality assurance function for a microservices and continuous deployment environment.
The QA Problem in a Microservices World
Modern software is rarely a single, self-contained application. It is a constellation of cooperating services, each with its own lifecycle, deployment cadence, and failure modes. Teams must now contend with distributed timing issues, cache invalidation, data consistency across service boundaries, and horizontal scaling — challenges that simply did not exist in a monolithic world.
Yet despite this fundamental shift in how software is built, the role of the traditional Quality Assurance team has been slow to change. QA has historically meant sitting in front of a user interface — or pointing Selenium at it — and systematically exercising input combinations to trigger error conditions. That approach tests only the visible tip of the iceberg, while the underlying distributed infrastructure remains largely unexamined.
As the Guardian’s engineering team put it: the iceberg has not just grown — it has melted, exists concurrently in multiple locations, and is also on fire.
Rethinking Shared Responsibility for Quality
One response to this problem is to treat quality as everyone’s concern rather than a specialised team’s gatekeeper function. At the Guardian, each product team is expected to take responsibility for supporting its own platform. Rather than maintaining separate, dedicated DevOps or Site Reliability Engineering (SRE) teams, they distribute that responsibility across product teams — a culture of shared ownership.
This philosophy forces the QA function to evolve. If quality is pervasive in everything the organisation does, then the people responsible for quality must be equally pervasive and technically versatile.
Introducing the Software Engineer (Tools and Infrastructure)
The Guardian’s answer to this challenge is the Software Engineer (Tools and Infrastructure) — internally abbreviated as SETI. The title is borrowed from Google’s engineering lexicon, where it described engineers who come from a strong technical background but focus on improving the development process rather than shipping end-user features.
The distinction from a traditional Software Engineer in Test (SET) is important:
- A SET improves quality by testing the finished product — validating the output.
- A SETI improves quality by improving how software is built, deployed, and monitored — working on the process itself.
What a SETI Does
- Designs and maintains Continuous Integration and Continuous Deployment (CI/CD) pipelines
- Builds internal tools that help developers diagnose issues faster
- Creates production monitoring and alerting for live applications
- Carries out performance testing at meaningful scale
- Refactors code to improve maintainability and testability
- Collaborates with product engineers to solve systemic quality problems
What a SETI Does Not Do
- Act as a release gatekeeper or bottleneck
- Stamp software changes with a QA seal of approval
- Write unit tests on behalf of developers all day
At the Guardian, developers peer-review each other’s work and write tests as appropriate. The SETI role exists to improve the conditions and tooling around that work, not to own quality unilaterally.
Embedding SETIs in Product Teams
Each SETI is embedded within an existing product team rather than sitting in a centralised QA department. They are expected to work independently to identify, prioritise, and resolve quality issues — but they spend the majority of their time collaborating directly with other engineers on the team.
Crucially, SETIs also meet regularly across team boundaries to share knowledge: comparing problems they have faced, solutions they have implemented, and identifying opportunities to build reusable tooling or establish shared standards. With enough SETIs distributed across enough teams, the organisation gains a coherent, organisation-wide view of engineering health.
Case Study: Monitoring Features Using Production Data
The Guardian’s mobile team runs large beta programmes to maintain quality in their iOS and Android apps. Despite broad coverage of user-facing changes, some categories of bugs can slip into production and remain undetected for extended periods. One example: a bug that prevented users from seeing dynamic subscription messaging was not caught during internal testing or by beta testers.
To address this class of problem, the SETI team built a monitoring service that allows developers to define automated, periodic checks on how specific features are performing in production. The service is implemented as an AWS Lambda function that queries a Data Lake using the AWS Athena SDK. Although SETIs led the build, they worked alongside product engineers to explain the tooling and encourage adoption for a broader range of features.
Key Takeaway
Production data is a first-class testing signal. Monitoring features continuously against real usage patterns catches entire categories of bugs that pre-release testing cannot.
Case Study: Production-Quality Performance Feedback at Pull Request Stage
When releasing changes to theguardian.com, several categories of automated tests require the full application to be instantiated and run. These include:
- Validating pages against the Google AMP specification
- Server-side load testing
- Browser performance checks, such as Time to Interactive and Lighthouse scores
These tests are too resource-intensive and time-consuming for developers to run locally as part of a unit test suite, and running them on developer laptops produces results that are not representative of production conditions.
The SETI team’s solution was a tool called PRBuilds.
How PRBuilds Works
- PRBuilds triggers automatically every time a GitHub Pull Request is updated.
- It builds and runs the application in a production-like configuration inside a sandboxed environment, isolated from real user traffic.
- The configured test suite runs against this ephemeral environment.
- Results are posted back to the Pull Request as a comment, giving the author immediate, contextual feedback.
The result is that developers receive production-quality performance and conformance feedback at the earliest possible stage — before code is merged, and certainly before it reaches users.
Key Takeaway
Shifting performance and integration testing left — into the Pull Request lifecycle — surfaces regressions early without requiring developers to run heavyweight test suites locally.
Goals and Outcomes of the SETI Model
The Guardian’s SETI model is designed to achieve three things:
- Embed quality into culture and tooling from the ground up. By sitting inside product teams, SETIs help engineers build quality practices into their daily workflows rather than treating QA as an external checkpoint.
- Gain organisation-wide insight. By coordinating across teams, SETIs can identify systemic quality problems, surface common pain points, and develop reusable solutions and standards that benefit the whole engineering organisation.
- Avoid becoming a release bottleneck. The model explicitly rejects the gatekeeper model. SETIs are there to accelerate quality, not to slow down delivery.
What This Means for Your Organisation
The Guardian’s approach is not the only valid model, but it illustrates some broadly applicable principles for engineering organisations operating in a microservices and continuous deployment context:
- Quality ownership must be distributed. No single team can test a distributed system comprehensively from the outside. Quality must be built into each team’s development process.
- Test signals should match the deployment environment. Developer laptops are not production. Invest in tooling that provides realistic, automated feedback against production-like environments as early as possible in the development lifecycle.
- Production monitoring is a quality tool. Observability of live features using real usage data catches classes of bugs that pre-release testing routinely misses.
- Specialist engineers who focus on the development process — not just the product — add disproportionate value. Investing in internal tooling, CI/CD infrastructure, and developer experience has compounding returns across every team that benefits from it.
Original source: The Guardian Engineering Blog (theguardian.com). The examples and role descriptions in this article are attributed to The Guardian’s engineering team.