This article summarises a presentation by Redgate Evangelist Steve Jones at Day 1 of PASS Summit 2017, as reported by InfoQ. Original coverage can be found at infoq.com.
Why Databases Need DevOps Too
DevOps has transformed how teams deliver application code, but databases are often left out of the conversation. They carry state, require careful migration scripts, and can break production in ways that a simple application rollback cannot fix. Steve Jones, an evangelist at database tooling company Redgate, made the case at PASS Summit 2017 that every technique used to accelerate and safeguard application delivery can—and should—be applied to database change management.
To illustrate the pace modern infrastructure demands, Jones cited Amazon’s reported 2016 deployment cadence of a production change approximately every 12 seconds. Whether or not your organisation targets that frequency, the underlying point holds: effective change-management processes are what make high-velocity delivery safe, not just fast.
The Scale Problem: Why Getting Database Changes Right Matters
Databases underpin nearly every production system, and the volume of data they manage continues to grow. Jones pointed to Microsoft collecting hundreds of terabytes of telemetry from Azure users every day as a concrete example of the scale involved. At that level, even a poorly tested schema migration or index change can cascade into hours of downtime or data loss.
Gene Kim’s Three Principles as the Foundation
Jones grounded his recommendations in the three core principles of DevOps articulated by Gene Kim (co-author of The Phoenix Project and The DevOps Handbook):
- Systems Thinking – Optimise for the whole delivery pipeline, not individual silos. A database team that throws scripts over the wall to an operations team is not thinking in systems.
- Amplify Feedback Loops – Shorten the time between making a change and knowing whether it worked. Automated testing and monitoring of database changes are essential here.
- Culture of Experimentation and Learning – Treat failures as learning opportunities. Safe-to-fail environments encourage teams to try improvements without fear.
Practical Techniques for Database DevOps
1. Version Control Everything
The single most important foundational step is placing all database code under version control—Git, SVN, or any other VCS your team already uses. Schema definitions, stored procedures, functions, views, seed data scripts, and migration scripts should all live in source control, with the same discipline applied to commit messages and branching strategies as your application code.
2. Treat Database Code Like Application Code
SQL scripts, migration files, and schema definitions are code. They deserve pull requests, peer review, and automated linting or static analysis just as much as any TypeScript or Java file. Whether you store database objects in the same repository as your application or in a dedicated repo is a team decision—both approaches are valid—but the discipline applied to them must be identical.
3. Code Reviews for Database Changes
Mandatory, meaningful code reviews catch errors that automated tools miss—missing indexes, implicit type conversions, or a DROP statement targeting the wrong table. Jones emphasised that reviews must be taken seriously, not rubber-stamped, because the cost of a bad database change in production is typically much higher than a bad application change.
4. Use Curated Test Data Sets
A common mistake is testing database migrations against a subset of production data. Jones recommends a different approach: start with an empty database in each pre-production environment (DEV, UAT, STG) and populate it with a small, deliberately curated set of rows—typically 10 to 40—that represent the meaningful edge cases and use cases found in production.
This approach has several advantages:
- No risk of accidentally exposing real user data in lower environments.
- Tests remain fast and deterministic.
- Edge cases are explicitly documented in the seed data rather than hoped to appear in a random production sample.
5. Enforce Consistent, Documented Deployment Processes
Every deployment to production should follow an identical, repeatable process. Where manual steps cannot be eliminated, document them thoroughly in a runbook. Consistency ensures that no critical step—such as running unit tests, verifying primary keys are defined on new tables, or confirming backups—is accidentally skipped under pressure.
6. Write and Test Rollback Scripts Before Deploying
Rolling back a failed database change is notoriously harder than rolling back application code, because schema changes and data migrations alter state. Jones recommends writing the rollback script at the same time as the forward migration—and testing that rollback script in a pre-production environment before the change is deployed to production. If something goes wrong, you want a working, tested script ready to execute, not one being written under incident pressure.
7. Use Dark Deploys and Feature Flags
A dark deploy (sometimes called a dark launch) involves deploying code and schema changes to production without making the new functionality visible or active for users. The feature is hidden behind a feature flag and is only enabled once the team is confident the system can handle it—often after load testing in the production environment itself under realistic conditions.
Jones cited Facebook’s rollout of Facebook Messenger as an example of this technique being used to validate that infrastructure could absorb the anticipated user load before the feature was made generally available. Feature flags decouple deployment from release, giving teams a safety valve when database-level changes cannot easily be reversed.
Key Takeaways
- Version control is non-negotiable. No other database DevOps practice works reliably without it.
- Treat SQL as first-class code. Apply the same review and testing rigour you apply to application code.
- Curate your test data. Small, deliberate data sets are safer and more reliable than production samples.
- Plan for rollback before you deploy. Write and test rollback scripts as part of the change process, not as an afterthought.
- Decouple deployment from release using feature flags to reduce the blast radius of any single change.
Applying these principles consistently will not eliminate database incidents, but it will dramatically reduce their frequency, shorten their duration, and make each one a learning opportunity rather than a crisis.