This article is based on a talk given by Mike Roberts, co-founder of consultancy Symphonia Cloud, at Continuous Lifecycle London. The original report was published by The Register.
The Core Problem: Ideas Are Expensive to Test
Software development often feels painfully slow — not because engineers can’t write code fast enough, but because organizations struggle to validate whether the right things are being built in the first place. According to Mike Roberts, co-founder of cloud consultancy Symphonia Cloud, the solution isn’t to write code faster. It’s to test ideas more rapidly and learn from the results continuously.
Speaking at Continuous Lifecycle London, Roberts made a blunt observation he attributed to software consultant Jeff Patton: “Most of our ideas suck.” But that’s not a reason for despair — it’s a call to action. If you can shorten the feedback loop enough to test more ideas in less time, you improve your odds of discovering the ones that genuinely help customers.
From Continuous Delivery to Continuous Experimentation
Most DevOps practitioners are familiar with continuous delivery (CD) — the practice of keeping software in a releasable state and automating the path from commit to production. CD already accelerated development cycles considerably, compressing timelines that once stretched to years down to weeks.
Continuous experimentation takes that further. Instead of simply deploying faster, the goal is to use each deployment as a structured learning opportunity — measuring outcomes, feeding data back into product decisions, and iterating in days or even hours rather than weeks.
Roberts framed it clearly: “There’s no point in having continuous experimentation if we’re not continuously learning.” Velocity without feedback is just expensive motion.
The Numbers Game of Innovation
The strategic logic behind continuous experimentation is probabilistic. If most feature ideas, product hypotheses, and technical approaches are destined to underperform, then the team that can safely test the most ideas in a given time period has a structural advantage. They’ll surface the winning ideas faster and discard the losing ones before sinking deep resources into them.
This framing resets how engineering success is measured. A failed experiment isn’t wasted effort — it’s data. The waste lies in building something over six months only to discover it doesn’t move the needle for users.
“If we can try enough of these ideas out, we can play a numbers game. We can find the ideas that will really help our customers.”
Mike Roberts, Symphonia Cloud
Three Categories of Obstacles
Roberts identified three distinct types of barriers that prevent engineering organizations from reaching a state of frictionless, continuous experimentation.
1. Technical Challenges
- Incidental complexity: Code and infrastructure that grows complex not because the problem is complex, but because of accidental architectural decisions. This slows every future change.
- Infrastructure overhead: Long provisioning times, large upfront commitments, and costly systems raise the cost of each experiment and discourage frequent iteration.
- Ecosystem fragmentation: A patchwork of incompatible tools and services creates a technical tar pit — teams spend more time managing integrations than shipping value.
Roberts argued that serverless architecture directly addresses several of these issues. By abstracting away infrastructure management and eliminating idle-resource costs, serverless lowers the barrier to spinning up new experiments. Crucially, he emphasized that cost savings are only part of the benefit:
“If you’re only using serverless for the cost savings, I think you’re missing a big trick.”
Mike Roberts, Symphonia Cloud
The deeper value is in accelerating the development cycle — reducing the friction between idea and deployed test.
2. Organizational and Cultural Challenges
- Deployment lag: Even with fast CI/CD pipelines, manual approvals, risk-averse change management processes, and siloed teams can extend the time from code-complete to production by days or weeks.
- A culture that doesn’t value learning: Organizations that treat post-mortems as blame sessions, or that bury experiment results, destroy the feedback loop that makes experimentation valuable.
- Centralized product ownership: When only product managers or leadership can initiate ideas, engineers are reduced to what Roberts called “code robots” — executing specs rather than contributing domain expertise to product decisions.
Spreading ownership of product innovation across teams — including engineers — unlocks a broader pool of hypotheses to test and gives engineers a genuine stake in outcomes.
3. Safety Challenges
- Psychological safety to fail: Teams won’t experiment if failed experiments result in blame or career risk. Organizations must make it structurally safe to run an experiment that doesn’t work.
- Budget guardrails: Rapid experimentation can lead to runaway cloud costs or resource sprawl if not managed carefully. Automated cost controls and experiment timeboxing are essential.
- Security and data integrity: Moving fast increases the risk of shipping experiments that expose sensitive data or introduce vulnerabilities. Security must be embedded in the experiment pipeline, not bolted on at the end.
What This Means for Engineering Teams in Practice
Adopting continuous experimentation isn’t simply a tooling decision — it requires rethinking how work is scoped, measured, and valued. Some practical implications:
- Design features as testable hypotheses. Rather than specifying what to build, frame work as: “We believe [change X] will produce [outcome Y] for [user segment Z].” This forces upfront thinking about how success will be measured.
- Instrument everything from day one. An experiment without measurement is just a deployment. Build observability in from the start — feature flags, event tracking, and A/B testing infrastructure.
- Shorten the blast radius of experiments. Use canary releases, percentage rollouts, and feature flags to limit exposure while an experiment is live.
- Establish a standard for killing experiments. Define in advance what a failed experiment looks like, and be disciplined about stopping experiments that aren’t meeting thresholds rather than letting them linger in production.
- Share results broadly. Post experiment outcomes — wins and losses — in a shared channel or wiki. This builds institutional knowledge and normalizes failure as part of the process.
The Lead Time Opportunity
Roberts noted that over the past two decades, lead times for software delivery have fallen dramatically — from years to weeks in many organizations. The next frontier is pushing that down to days or hours for discrete experiments. That’s not a fantasy for elite tech companies alone; it’s increasingly achievable for any team willing to invest in the right architecture, tooling, and culture.
The bottleneck is rarely the code. It’s the process surrounding the code — approval gates, infrastructure provisioning, unclear ownership, and fear of failure. Address those, and the speed follows.
Key Takeaways
- Most product and feature ideas fail to deliver the expected value — this is normal, not exceptional.
- Continuous experimentation extends continuous delivery by making learning, not just deployment, the goal of each cycle.
- Serverless architecture can accelerate experimentation by reducing infrastructure friction, not just cutting costs.
- Cultural and organizational barriers — fear of failure, centralized ownership, slow deployment processes — are often harder to fix than technical ones.
- Safety in experimentation means psychological safety, budget controls, and embedded security — all three matter.