This article is based on a commentary originally published on devops.com.
Speed Is the Point — Until It Becomes the Problem
One of the core promises of DevOps is accelerated delivery. Whether your team ships weekly, daily, or multiple times per day, the expectation is that software reaches users faster than it did before. That acceleration is genuinely valuable. But speed also creates pressure, and pressure cuts corners — and cut corners create lapses that can be far more expensive than any delayed release.
The uncomfortable truth is that some of the most damaging, and most avoidable, incidents in recent years have come not from exotic attack vectors or complex failures, but from basic hygiene oversights: misconfigured cloud storage buckets, missing access controls, absent rollback plans. These are not unsolvable problems. The tools and practices to prevent them exist. The challenge is working them into a delivery process that already feels maxed out.
Security and Testing: The Easiest Things to Skip, the Costliest to Ignore
Security and testing are the first casualties when delivery schedules tighten. This is well understood in theory, yet it keeps happening in practice. Publicly exposed cloud storage is a canonical example. Cloud providers and third-party vendors offer automated scanning tools that flag open or misconfigured buckets as part of routine environment checks. The warnings are there — but organisations have to be using the tools and acting on the output.
A blameless post-mortem culture is an important part of psychological safety inside a team, but it offers no protection from external consequences. Regulatory bodies, customers, and the market at large will impose their own penalties when user data is exposed. That reality places the responsibility to prevent such failures squarely on the engineering and operations organisation.
Roll Forward vs. Roll Back: You Need Both
Most mature DevOps teams optimise for rolling forward — detecting an issue quickly, pushing a fix, and continuing. This strategy works well the vast majority of the time and aligns with a culture of continuous improvement. However, there are failure scenarios where rolling forward is not an option: data corruption, breaking schema migrations, third-party dependency failures, or incidents that require immediate restoration of a known-good state.
A credible rollback capability goes well beyond pointing to a Git tag. A complete rollback plan must account for:
- Database state: Schema changes and data migrations may need to be reversed or compensated.
- Infrastructure configuration: Infrastructure-as-code changes (Terraform state, Kubernetes manifests) must be versioned and reversible.
- Dependent services: Downstream consumers of an API may break if a version is reverted without coordination.
- Feature flags: Feature flag systems allow functionality to be disabled instantly without a code deployment, providing a fast, low-risk fallback layer.
- Secrets and configuration: Environment-specific configuration must be captured alongside code versions.
Building and testing rollback procedures before you need them is what separates teams that recover gracefully from teams that recover slowly and painfully.
“There Isn’t Time” Is Not a Risk Management Strategy
The most common objection to investing in preventive practices is that the delivery pipeline leaves no room for them. This is understandable — faster delivery creates appetite for even faster delivery, and it can feel like adding any friction will break the momentum. But consider the alternative: if a security breach or a total systems failure occurs, does the team have time to respond to that? Incident response, customer communication, regulatory reporting, and service restoration are all extremely time-consuming. Prevention is almost always cheaper.
When a serious incident occurs, accountability typically falls on the IT and engineering organisation, not on business stakeholders or application owners. That means the engineering team has both the most to lose and the most ability to prevent the outcome.
Practical Steps to Protect Your Delivery Velocity
The goal is not to slow down delivery but to make delivery sustainably fast. A few concrete practices to consider:
- Integrate cloud security scanners into your CI/CD pipeline. Automated tools that check for misconfigurations — open storage buckets, overly permissive IAM policies, unencrypted resources — should run on every deployment, not just on quarterly audits.
- Use feature flags to decouple deployment from release. Deploying code and enabling it for users are separate events. Feature flags give you a fast, low-blast-radius way to disable a change without rolling back the entire deployment.
- Require a second reviewer on infrastructure and security changes. The person who creates a storage bucket should not be the only person who verifies its access policy. A lightweight peer-review step catches a large class of configuration errors.
- Document and rehearse your rollback procedure. Write it down, version it, and run a drill. Discover the gaps before an incident forces you to.
- Build disaster recovery into sprint capacity, not just into aspirations. If DR and security hardening only happen when there is spare capacity, they will rarely happen. Treat them as non-negotiable delivery items.
Conclusion
DevOps teams are delivering real value — more frequently, with greater functionality, and with demonstrably faster feedback loops. None of that progress should be put at risk by preventable failures in security posture or operational readiness. The tools to address these gaps already exist. The work is integrating them into the process as standard practice, not as afterthoughts. Getting the basics right is not the opposite of moving fast — it is what makes sustained speed possible.