What Is DataOps?

DataOps is a practice that combines Agile development, DevOps culture, and statistical process control to improve the speed and accuracy of data analytics. Where traditional data management tends to be siloed — with data engineers, data scientists, and business stakeholders working in relative isolation — DataOps encourages these groups to collaborate continuously throughout the data lifecycle.

Core capabilities that DataOps targets include:

• Faster, more reliable data pipeline delivery
• Automated data quality monitoring and testing
• Streamlined data integration and orchestration
• Reproducible model deployment
• End-to-end data governance

In essence, DataOps strives to make data teams as agile and productive as high-performing software engineering teams.

Why Does Data Management Need Reinventing?

Many organizations still struggle with a familiar set of data problems: requests for reports that arrive late or answer the wrong question; pipelines that break silently; data trapped in departmental silos; and governance frameworks that slow down rather than enable analysis.

A common example is the report request cycle. A business stakeholder asks for a new report. The request is passed to a data engineer who interprets the requirement through a technical lens. The resulting output may not match what the stakeholder actually needed. Several revision cycles later, deadlines are missed and frustration mounts on both sides. DataOps addresses this by establishing clearer communication channels, shared tooling, and iterative feedback loops — much as Agile does for software development.

Key Focus Areas for Implementing DataOps

There is no single prescribed implementation path for DataOps. Organizations typically begin by focusing on several foundational areas.

1. Data Democratization

Data democratization means making data accessible to the people who need it, not just to a small team of specialists. As data volumes grow and business decisions increasingly depend on timely information, limiting access to data becomes a competitive liability. DataOps encourages building self-service data access mechanisms — governed catalogs, curated datasets, and role-based access controls — so that analysts and business users can find and use data without always relying on engineering bottlenecks.

2. Leverage the Right Data Platform

A mature DataOps practice requires a platform stack that supports the full data lifecycle. This typically includes:

• Development environments — notebooks (Jupyter, for example), version control (Git), and support for languages such as Python and R
• Orchestration tools — workflow schedulers (such as Apache Airflow or Prefect) to coordinate pipeline execution
• Data integration layers — ETL/ELT tools and connectors to move data between sources and destinations reliably
• Observability tooling — monitoring and alerting on pipeline health, data freshness, and data quality metrics

The specific tools will vary by organization, but the principle is a unified platform where data engineers, scientists, and analysts can collaborate without constantly switching contexts.

3. Automate for Speed and Reliability

Manual steps are the enemy of fast, reliable analytics. DataOps borrows heavily from CI/CD (continuous integration and continuous delivery) to automate repetitive tasks such as:

• Pipeline testing and validation on every code change
• Automated data quality checks (schema validation, null checks, range assertions)
• Deployment of updated models or transformation logic to production
• Alerting when pipeline SLAs are breached

Automation reduces the cycle time between an idea and a deployed, trusted data product.

4. Governance That Enables Rather Than Blocks

Traditional data governance — centralized committees, heavyweight approval processes — often moves too slowly for modern data teams. DataOps favors a more federated model: clear ownership of data assets, automated policy enforcement (for example, tagging PII columns and blocking unauthorized exports automatically), and governance that is embedded into pipelines rather than bolted on as a separate audit process.

Getting governance right is one of the harder challenges when adopting DataOps, especially in regulated industries. Organizations should expect to iterate on their governance model rather than waiting for a perfect framework before starting.

5. Foster Cross-Functional Collaboration

None of the above works without genuine collaboration between data engineers, data scientists, analytics engineers, and business stakeholders. Shared tooling helps, but cultural change matters just as much. Practices borrowed from Agile — short sprints, shared backlogs, regular retrospectives, and demo sessions — can help data teams align with business priorities and ship iteratively rather than in long, opaque project cycles.

How DataOps Relates to DevOps

The analogy between DataOps and DevOps is intentional and useful. DevOps broke down the wall between software development and IT operations. DataOps attempts to do something similar for the data world: breaking down the wall between the people who build data infrastructure and the people who use it to make decisions.

Both disciplines share a common set of values:

• Automation over manual process
• Continuous improvement through feedback loops
• Shared ownership rather than handoffs
• Measuring outcomes, not just outputs

Where DataOps differs is in the domain-specific challenges it must address: data quality is harder to test than application behavior, data lineage and compliance requirements add complexity, and the consumers of data products (business analysts, executives) are often less technical than application end users.

Does DataOps Actually Revolutionize Data Management?

“Revolution” is a strong word, and the reality is that DataOps is an evolution of existing good practices — Agile, DevOps, data quality engineering — applied deliberately to the data domain. Organizations that adopt DataOps principles typically report faster analytics cycle times, higher data quality, and better alignment between data teams and business goals. But realizing those benefits requires sustained investment in culture, tooling, and process — not just adopting a new label.

For enterprises sitting on increasingly large and complex data estates, DataOps offers a practical framework for managing that complexity without sacrificing speed. Whether that constitutes a revolution depends on how far the starting point was from good practice.

Getting Started with DataOps

A pragmatic starting point for most teams:

1. Audit your current pipeline reliability — understand where failures occur and how long they take to detect and fix.
2. Introduce automated data quality tests on your most critical datasets.
3. Version-control your transformation code and treat data pipeline changes like application code changes.
4. Run a short sprint with a cross-functional team — engineer, scientist, and business analyst together — to deliver one analytics use case end to end.
5. Establish data ownership so every key dataset has a named owner responsible for its quality and documentation.

DataOps is not a product you deploy — it is a set of practices you adopt incrementally. Starting small with measurable goals is far more effective than waiting until the entire organization is ready for a wholesale transformation.

Why Silos Are the Enemy of Speed

Customer expectations around delivery speed have risen sharply. Markets are more dynamic, global competition is intensifying, and the gap between organisations that ship quickly and those that do not is widening. DevOps addresses this directly by replacing rigid, department-based structures with cross-functional, outcome-oriented teams.

The core promise of DevOps is straightforward: combine agile methods, automated processes, and collaborative culture to shorten development cycles, improve product quality, and reduce time to market. But the tooling is only part of the story. The harder — and more consequential — part is organisational change.

DevOps Is a Cultural Shift, Not Just a Toolchain

Adopting DevOps requires no less than a fundamental rethink of how teams are structured, how decisions are made, and how work flows through an organisation. Specifically, it demands:

• Cross-functional team formation: Teams should be built around business processes and product outcomes, not around traditional departments such as Dev, QA, and Ops in isolation.
• Autonomous decision-making: Competence teams — composed of developers, UX specialists, and product managers — need real authority over technical and operational choices, including budget and resource decisions within their scope.
• Tolerance for iteration: Leadership must accept that continuous improvement is the goal, not a one-time delivery. This requires both courage and a clear vision of the end state.

Without genuine autonomy, cross-functional teams simply become another layer of bureaucracy. The structural change must be accompanied by a shift in management culture.

The “Always Beta” Mindset

One useful framing for DevOps culture is the mantra “Always Beta”. This does not mean shipping broken software or treating production as a permanent construction site. It means treating development itself as a continuous process — one that allows for incremental improvements while the system is live.

Retail and e-commerce companies illustrate this well. Organisations operating large online shops work with real-time KPIs and understand that even small changes to a page layout can have a measurable impact on conversion. Practices like A/B testing — running two variants simultaneously to determine which performs better — are a natural expression of the Always Beta mindset. They treat the production environment as a learning environment, not a static destination.

Where DevOps Delivers Quickly: Front-End and Cloud-Native Applications

DevOps shows the clearest gains in areas where teams can work in short cycles with fast feedback loops. Examples include:

• Microservices architectures: Small, independently deployable services allow teams to ship and test changes without coordinating a full system release.
• UI and front-end applications: Modern tooling allows interfaces to be assembled and validated rapidly. User interactions can be simulated early, and usability problems identified before they reach production.
• Cloud-native deployments: Platforms such as Cloud Foundry enable automated build, test, and deploy pipelines that reduce manual handoffs between teams.

In these contexts, replacing a traditional waterfall delivery with small DevOps teams and automated test procedures can dramatically compress development timelines.

Where DevOps Gets Hard: Monolithic ERP Core Systems

DevOps encounters real constraints when applied to mature, heavily customised ERP landscapes. The root cause is architectural: ERP systems like SAP are built on a stable, monolithic core that has often been modified over many years to fit specific business requirements.

In this environment, a change that benefits one department can trigger unexpected side effects in another. The system is interconnected in ways that are not always well-documented, and the test scenarios needed to validate a change safely can be expensive and time-consuming to build. This makes the fast feedback cycles that DevOps depends on difficult to achieve at the core level.

The practical implication: when operating within a mature SAP landscape, DevOps teams must invest heavily in test automation and clearly defined acceptance criteria before they can iterate safely. Trust in automated test coverage is non-negotiable.

SAP S/4HANA and the Greenfield Opportunity

SAP’s S/4HANA platform offers a potential path forward. Organisations willing to undertake a greenfield migration — cleaning up years of customisation and aligning more closely with SAP standard processes — can emerge with a system architecture that is more compatible with DevOps practices.

The key shift is that S/4HANA is oriented around business processes rather than organisational departments. This aligns naturally with how DevOps teams are structured. When the underlying system architecture and the team structure share the same logic, agile development at the core becomes achievable rather than theoretical.

The trade-off is significant: a greenfield approach requires substantial upfront investment and organisational discipline. But for companies prepared to make that commitment, it removes one of the principal blockers to applying DevOps practices across the full application landscape.

Key Takeaways

• DevOps is as much a cultural transformation as a technical one. Structural and decision-making changes must accompany tooling adoption.
• Cross-functional teams built around business outcomes — not departments — are the foundation of effective DevOps.
• The Always Beta mindset treats production as a continuous improvement environment, not a static endpoint.
• Front-end, microservices, and cloud-native applications are the easiest places to start and demonstrate value quickly.
• Mature ERP core systems present genuine constraints. Invest in test automation and clearly defined outcomes before iterating at that layer.
• Platform modernisation (such as SAP S/4HANA migration) can unlock DevOps practices in areas where legacy architecture previously made them impractical.

Partnership Overview

Agile Stacks announced its entry into the CloudBees Elevate Partner Program, a channel initiative designed to extend CloudBees’ enterprise DevOps capabilities through complementary technology and services partners. The partnership pairs CloudBees’ enterprise-grade continuous delivery platform — built on Jenkins — with Agile Stacks’ GitOps automation service, which is aimed at dramatically reducing the time and cost required to stand up production-ready DevOps environments in the cloud.

What Each Company Brings to the Table

CloudBees: Enterprise Jenkins and Continuous Delivery

CloudBees is built on top of Jenkins, one of the most widely used open-source automation servers in the DevOps ecosystem. CloudBees adds enterprise-critical capabilities on top of the upstream Jenkins project:

• Security and compliance controls for regulated industries
• Scalability to support large numbers of development teams and concurrent builds
• Centralized management of Jenkins masters across an organization
• Expert support backed by a dedicated customer success team

CloudBees also supports running Jenkins workloads on Kubernetes, enabling dynamic, containerized build agents that scale on demand — a critical requirement for organizations with variable CI/CD workloads.

Agile Stacks: GitOps Automation and DevOps Cloud Stacks

Agile Stacks describes its platform as a “non-opinionated” GitOps automation service — meaning it does not lock customers into a specific toolchain. Instead, it provides a framework for deploying full-function DevOps cloud stacks composed of pre-tested, integrated open-source tools. Key claims from the company include:

• Deployment of complete DevOps stacks within minutes, compared to typical enterprise rollout timelines that can stretch from six months to two years
• Application Templates that automatically create CI/CD pipelines binding development, test, and production environments to stacks
• A DevOps-first architecture delivered out of the box, reducing the custom integration work traditionally required

How the Integration Works

The combined offering targets enterprise teams that want to run CloudBees Jenkins at scale on Kubernetes without spending months on infrastructure setup. The integration works as follows:

1. An organization uses Agile Stacks Application Templates to provision a full DevOps cloud stack — including networking, storage, security, and tooling — on their chosen cloud provider.
2. CloudBees Jenkins is included within that stack, pre-configured with enterprise-grade settings rather than requiring manual setup.
3. Pipeline configuration is automated by Agile Stacks, linking source repositories, build environments, and deployment targets without requiring hand-crafted Jenkinsfiles from scratch.
4. Teams can scale builds horizontally across Kubernetes worker nodes, using the elastic compute capacity of their cloud provider.

Business Goals of the Partnership

Beyond the technical integration, the partnership involves joint go-to-market activities:

• Joint sales motions targeting enterprise accounts looking to accelerate DevOps adoption
• Co-marketing initiatives focused on educating organizations about GitOps practices and cloud resource management
• A shared emphasis on reducing operational expenditure by automating infrastructure provisioning and pipeline management

Industry Context: Why This Partnership Matters

By 2018, Kubernetes had become the dominant container orchestration platform, but running stateful enterprise tools like Jenkins reliably on Kubernetes remained a significant operational challenge for many teams. Separately, GitOps — the practice of using Git as the single source of truth for infrastructure and application state — was gaining traction as a way to apply software engineering discipline to infrastructure management.

Partnerships like this one reflected an emerging pattern: specialist automation vendors wrapping and extending core DevOps platforms (Jenkins, in this case) to lower the barrier of entry for enterprises that lacked the in-house expertise to build and maintain complex CI/CD infrastructure from scratch.

Quotes from Both Companies

“We empower enterprises in today’s global digital economy to innovate and adapt rapidly using our Agile Stacks automated agile software delivery service, which significantly streamlines the deployment and update management of the many tools they rely on. We are excited to partner with CloudBees in delivering enterprise Jenkins, the DevOps and continuous delivery solution of choice for many enterprises, with our service.”

— John Morada, Vice President of Global Business Development, Agile Stacks

“We pride ourselves on providing enterprise-grade continuous delivery functionality to DevOps teams and then complementing that with companies, such as Agile Stacks, to enable the success of non-opinionated enterprise-grade open source technologies. Working together, we are addressing our customers’ growing demand to run our platform on Kubernetes, while cost-effectively providing the security and compliance, flexibility, and scale capabilities they need to drive their businesses.”

— Aaron Morrison, Director of Worldwide Channel Sales, CloudBees

About Agile Stacks

Agile Stacks provides an all-in-one, non-opinionated GitOps automation service aimed at enterprises building cloud-native platforms. Its stacks incorporate pre-tested, integrated open-source tools that are ready to use from the moment they are deployed, targeting the performance, agility, and scalability requirements of modern digital businesses.

The Mainframe Is Not Going Away

For many in the DevOps world, the mainframe feels like a relic — a machine that belongs to a different era of computing. The reality is more complicated. According to data center industry group AFCOM, roughly half of all companies with data centers operate at least one mainframe. IBM remains the dominant hardware vendor, with Unisys as a secondary player. For high-transaction workloads, data analytics, and large-scale scientific processing, mainframes continue to be the platform of choice in major enterprises — particularly in banking, insurance, and government.

That staying power creates a real engineering challenge: how do you apply modern software delivery practices — continuous integration, automated testing, rapid iteration — to a platform built around COBOL, PL/I, and Assembler?

Detroit-based Compuware has staked its business on answering that question. The company is positioning itself as the primary partner for enterprises that want to bring Agile and DevOps discipline to their mainframe operations.

Why Mainframe DevOps Is Hard

The core tension in mainframe development is generational. Many of the engineers who built and maintained these systems are retiring, taking decades of institutional knowledge with them. Incoming developers — often experienced in Java, Python, or C++ — may have little or no exposure to COBOL or IMS (Information Management System). Yet the applications they are inheriting are frequently poorly documented and deeply complex.

In traditional mainframe shops, this combination of complexity and knowledge loss makes rapid iteration risky. Without automated unit tests, a developer making changes to an unfamiliar COBOL program has limited ability to verify correctness quickly. This pushes teams toward long release cycles and heavy manual testing — the opposite of what DevOps requires.

Compuware CEO Chris O’Malley framed the urgency clearly in comments to eWeek:

“The mainframe goes back a long time, and the realities of having to do software development in ways that are continuously improving in terms of efficiency and in quality are now requirements; these are absolute imperatives to CIOs. In order to do that, you have to bring agile and DevOps into your mainframe platform. Compuware is the only mainframe partner that can help them do that.”

— Chris O’Malley, CEO, Compuware

What Compuware Announced in July 2018

On July 18, 2018, Compuware announced three significant moves — an acquisition, platform enhancements, and a new partnership — all aimed at expanding automated unit testing on the mainframe through its Topaz for Total Test platform.

1. Acquisition of XaTester from Xact Consulting A/S

Compuware acquired XaTester, a unit testing tool developed by Danish firm Xact Consulting A/S. XaTester enables developers to create unit tests for programs written in COBOL, PL/I, and Assembler, covering both batch and CICS-based applications.

CICS (Customer Information Control System) is significant here. It is the transaction processing middleware of choice for many of the world’s largest financial institutions and insurance companies, valued for its high integrity, performance, and scalability. Until this acquisition, automated unit testing for CICS-based programs was a meaningful gap in Compuware’s toolchain.

2. Enhanced Topaz for Total Test: IMS Support

Compuware expanded Topaz for Total Test to include automated unit testing for IMS (Information Management System) batch and transactional applications. IMS support is particularly important because contemporary developers frequently have little or no hands-on experience with IMS code, making manual validation slow and error-prone.

These enhancements build on Topaz for Total Test’s existing capabilities, which already included:

• Unit testing for COBOL batch applications
• Integration with Jenkins for CI pipeline automation
• Integration with SonarSource SonarQube for code quality analysis
• Integration with Compuware ISPW for source code management and deployment

3. Partnership with Parasoft

Compuware announced a partnership with Parasoft, a vendor specializing in automated software testing. The initial deliverable is an integration between Parasoft SOAtest and Topaz for Total Test. This integration allows developers to test API calls between mainframe and non-mainframe systems — an increasingly important capability as mainframe applications are expected to participate in modern service-oriented and microservices architectures.

Why Unit Testing Is the Keystone

Unit testing — the practice of validating small, isolated units of code automatically — is a foundational enabler of modern software delivery. On non-mainframe platforms, it is essentially assumed. Developers working in Java or Python have mature frameworks (JUnit, pytest, and many others) that integrate cleanly into CI/CD pipelines.

On the mainframe, this has historically been difficult. Compuware’s strategy with Topaz for Total Test is to give mainframe developers equivalent capabilities — so that unit tests can be created, run automatically on code changes, and surfaced through the same toolchain used for other platforms.

The benefits are standard DevOps outcomes applied to a non-standard platform:

• Earlier defect detection: Problems are found at the unit level, before integration testing or production deployment.
• Faster iteration: Developers can make incremental changes and verify them quickly, reducing the cost and risk of each change.
• Living documentation: Well-written unit tests describe expected behavior, helping new developers understand unfamiliar codebases.
• Safer knowledge transfer: As experienced mainframe engineers retire, automated tests preserve behavioral expectations that would otherwise exist only in human memory.

The Broader Topaz Platform

Topaz for Total Test sits within Compuware’s broader Topaz suite, which is designed to give DevOps practitioners a visually intuitive environment for working with mainframe applications — regardless of the application’s age, complexity, or state of documentation. The suite is intended to be accessible to developers who are not mainframe specialists, lowering the expertise barrier for teams that need to maintain or extend mainframe-based systems.

Compuware’s stated goal is a unified DevOps toolchain that treats the mainframe as a peer participant alongside cloud and distributed systems — not a silo requiring entirely separate processes and skills.

The Competitive Landscape

Compuware’s position in this market is distinctive. On the language side, Micro Focus is the primary commercial custodian of COBOL tooling, providing compilers, runtime environments, and development tools across both mainframe and distributed platforms. IBM itself provides development tooling through its Z Open Development initiative and related products. Compuware’s differentiation is its focus on the DevOps workflow layer — automated testing, source management, and toolchain integration — rather than the language runtime itself.

Whether this positioning is sustainable over the longer term depends on how quickly large enterprises actually adopt Agile and DevOps practices on their mainframe estates, and whether competing vendors or open-source initiatives close the capability gap. As of the 2018 announcement, Compuware was characterizing itself as the only vendor fully committed to this specific problem.

Key Takeaways for DevOps Engineers

• Mainframes remain operational in a large proportion of enterprise data centers, particularly in financial services and insurance.
• The skills gap between retiring mainframe specialists and incoming generalist developers is a real risk that DevOps tooling can partially mitigate.
• Automated unit testing is the critical enabler: without it, rapid iteration on mainframe code is too risky to practice reliably.
• Compuware’s Topaz platform integrates with Jenkins, SonarQube, and other standard DevOps tools, allowing mainframe workloads to participate in cross-platform pipelines.
• The Parasoft SOAtest integration specifically addresses the API boundary between mainframe and non-mainframe systems — a growing concern as enterprises modernize architectures without fully replacing mainframe backends.

DevOps Is Becoming a Core Business Priority

DevOps and Fast IT adoption have been identified as two of the most significant growth areas in enterprise technology. According to a report from Venafi, a privately held cybersecurity company, this growth is being propelled by the widespread adoption of cloud computing and the operational benefits it brings: speed, agility, efficiency, and economies of scale.

While the Venafi report focuses primarily on security vulnerabilities in machine-to-machine communications, it also highlights how dramatically the pace of software delivery has accelerated — and why that acceleration is reshaping how engineering teams operate.

What Is DevOps (and Why Is It Sometimes Called Fast IT)?

DevOps is a broad set of concepts and cultural practices centered on building, evolving, and operating rapidly-changing, resilient systems at scale. As defined by Agile Admin, it encompasses “the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.”

The term Fast IT is sometimes used interchangeably with DevOps, emphasizing the speed at which software can be delivered when development and operations teams work in close collaboration. Both terms point to the same underlying shift: breaking down silos between teams and enabling faster, safer software delivery.

In practice, DevOps brings together popular frameworks and methodologies so they work in concert across the entire systems-development lifecycle. The result is a pipeline that supports:

• Agile development — iterative, incremental delivery of software in smaller, manageable segments
• Continuous integration (CI) — automatically building and testing code changes as they are committed
• Continuous delivery (CD) — ensuring code is always in a deployable state, ready for release at any time

Containers and Microservices: The Engine Behind Fast IT

A key enabler of Fast IT is the use of containers — cloud-based, self-contained runtime environments that package an application and all its dependencies together. Containers allow teams to run applications reliably across different computing environments without worrying about configuration drift or dependency conflicts.

Containers work hand-in-hand with microservices — an architectural approach where an application is broken into small, independently deployable modules. Rather than deploying one large monolithic application, teams deploy and update individual services independently, dramatically reducing the risk and complexity of each release.

Together, containers and microservices allow engineering teams to release software faster, recover from failures more quickly, and scale individual components of an application without scaling the entire system.

Container Adoption: A Signal of DevOps Maturity

As a measure of how rapidly DevOps practices were being adopted, the Venafi report cited a Gartner projection — made around mid-2018 — that 50 percent of enterprises would be running containers in production environments by the end of that year. A significant driver of this growth was the explosion of Internet of Things (IoT)-generated data, which placed new demands on scalable, modular infrastructure.

This rapid proliferation of containerized workloads and connected machines directly increased the complexity of infrastructure — and with it, the number of machines that needed to communicate securely with one another.

The Security Challenge: Machine Identity at Scale

The Venafi report’s central concern is what happens to security when the number of machines — containers, virtual machines, IoT devices, APIs — grows exponentially. Each machine that communicates with another must be authenticated. This is handled through machine identities: cryptographic credentials such as TLS/SSL certificates and SSH keys that verify a machine is who it claims to be.

As DevOps pipelines scale and containers spin up and down dynamically, managing these machine identities becomes genuinely difficult. The risks include:

• Certificates expiring unnoticed, causing outages
• Poorly managed SSH keys granting unauthorized access
• Lack of visibility into which machines hold which credentials
• Insufficient automation for certificate issuance and renewal at container speed

For DevOps teams, this means that security practices must evolve alongside delivery pipelines. Automating certificate lifecycle management and integrating secrets management tools into CI/CD workflows are increasingly important steps for teams running containers in production.

Key Takeaways for DevOps Teams

• DevOps and cloud adoption are deeply linked — the speed and scale that cloud enables are primary drivers of DevOps growth.
• Containers and microservices are central to Fast IT, enabling faster releases and more resilient systems.
• Rapid container proliferation introduces machine identity management as a critical — and often overlooked — security discipline.
• DevOps maturity requires security to be embedded in the pipeline (DevSecOps), not bolted on after deployment.

The growth trajectory for DevOps practices has only continued since this report was published in 2018. For teams still building out their pipelines, the fundamentals remain the same: automate delivery, embrace incremental change, and treat security as a first-class concern from day one.

The State of DevOps Automation

Adoption is uneven. A significant share of DevOps teams report having no structured automation program in place, and even among those that do, only a minority have self-service capabilities for R&D and dev/test workflows. That gap matters: without meaningful automation, DevOps cannot deliver the agility required to support modern application development and deployment cycles.

One concrete symptom is infrastructure wait time. Less mature organizations commonly report waiting up to a month to get access to provisioned infrastructure, while leading teams have reduced that to hours. The difference is almost always automation.

Why Today’s Automation Is Different

Automation is not new in IT operations, but the current wave is qualitatively different from earlier iterations. Legacy automation tends to be siloed — scripts or runbooks that solve narrow problems in isolation. The result is that it rarely has a meaningful impact on overall delivery performance.

What has changed is the rise of virtualization and software-defined infrastructure. When compute, networking, and storage can be expressed and managed as code, automation can operate at a systemic level rather than touching individual components one at a time. Infrastructure as Code (IaC) tools such as Terraform, Ansible, and Pulumi are the practical expression of this shift. Organizations that have adopted integrated automation toolchains report meaningful gains in both deployment frequency and time-to-delivery.

Six Areas Ripe for Automation

When deciding where to direct automation investment, it helps to work from a prioritized list of domains rather than automating opportunistically. The following six areas are widely recognized as high-value targets:

• Networking — Network configuration management has historically been manual and change-averse. Software-defined networking and tools like Ansible network modules now make declarative, repeatable network automation practical.
• Infrastructure provisioning — Modern automation stacks can provision hundreds or thousands of servers (or cloud instances) in a single operation. IaC tools remove the bottleneck of manual server builds.
• Operating environment management — Configuration management tools enable centralized, consistent control over OS-level settings, patching, and compliance baselines across large fleets.
• Security — Security checks, policy enforcement, and vulnerability scanning should be embedded in the pipeline rather than applied after the fact. Automating security gates prevents them from becoming delivery bottlenecks.
• Orchestration — Coordinating the interaction between services, deployments, and environments requires a layer of orchestration automation, from container scheduling (Kubernetes) to workflow engines.
• Testing and quality gates — Automated test suites, linting, and static analysis integrated into CI pipelines catch regressions early and remove the need for manual sign-off on routine changes.

Governance: The Non-Technical Half of Automation

Technology alone does not make automation succeed. Without a governance model, automated systems can drift, accumulate unchecked costs, introduce compliance gaps, or create new single points of failure. A practical DevOps automation governance model should address at least six areas:

• Compliance by Design — Security and data-handling requirements must be encoded into automation templates and pipelines from the start, not bolted on after deployment. This is especially important when workloads span multiple cloud environments.
• Insights and cost visibility — When resource consumption is dynamic and automated, you need instrumentation and dashboards that show exactly what your systems are doing and what it costs in real time.
• Continuous Monitoring — Monitoring in an automated environment should be proactive, not reactive. Alerting and remediation logic should identify risk and compliance drift early enough to address it automatically or with minimal human intervention.
• Automated Provisioning — Applications should be able to move through dev, test, staging, and production environments quickly, with infrastructure spun up and torn down on demand to avoid resource waste.
• Integrated CI/CD — Script- and template-based pipelines reduce deployment time and create an auditable trail of every change. An integrated audit log is essential for regulated industries and post-incident analysis.
• Team Support and Shared Responsibility — Governance is not the job of a single platform team. It requires buy-in and participation across development, operations, security, and management. Automation often creates new roles — platform engineers, SREs, automation architects — that need explicit definition.

Avoiding the Autopilot Trap

A critical misconception about DevOps automation is that its goal is to remove humans from the loop. It is not. Over-reliance on automation — deploying it without adequate monitoring, governance, or human judgment — can be just as damaging as under-reliance.

The real value of automation is to amplify human decision-making: freeing engineers from repetitive, low-value tasks so they can focus on designing resilient systems, responding to novel incidents, and pursuing new product and market opportunities. Automated systems still need to be observed, tuned, and periodically rethought as the business evolves.

Put differently, automation raises the ceiling on what a DevOps team can accomplish — but it does not lower the bar for engineering judgment. The teams that get the most out of automation are those that treat it as a discipline, not a destination.

Getting Started: A Practical Approach

• Audit first. Map your current manual processes before automating anything. Understand where time is actually lost.
• Automate the most painful bottleneck first. Infrastructure provisioning and environment setup are common early wins with high ROI.
• Build governance in from day one. Establish tagging standards, cost alerts, and compliance checks before your automated footprint grows.
• Measure outcomes. Track deployment frequency, lead time for changes, mean time to recovery (MTTR), and change failure rate before and after automation efforts.
• Iterate. No automation program is complete. Treat it as a product with a roadmap, not a project with an end date.

Despite its well-documented benefits, DevOps adoption is rarely smooth. Cultural resistance, architectural debt, budget overruns, and tooling missteps can all stall or derail an initiative. Understanding these challenges in advance—and having concrete strategies to address them—significantly increases the likelihood of a successful rollout.

Major DevOps Implementation Challenges and How to Resolve Them

1. Organizational Alignment

One of the most fundamental prerequisites for DevOps success is alignment between development and operations teams. Development teams typically follow agile, sprint-based cycles optimized for speed, while operations teams often maintain lengthy change management procedures designed to protect production stability. These differing rhythms create friction that slows delivery and erodes trust between groups.

Resolution: Bring stakeholders from both sides together early in the planning process. Establish shared goals—such as deployment frequency, mean time to recovery (MTTR), and change failure rate—that both teams are accountable for. Joint planning sessions, shared on-call rotations, and co-located (or virtually co-located) teams are practical first steps toward genuine alignment.

2. Legacy Infrastructure and Systems

Legacy infrastructure is a persistent obstacle. Older systems frequently lack APIs, automation hooks, or the flexibility needed to support continuous integration and continuous delivery (CI/CD) pipelines. Beyond technical limitations, legacy platforms often carry stability risks and reduced vendor support, making rapid iteration difficult.

Resolution: A full infrastructure overhaul is rarely feasible in a single step. Prioritize incremental modernization: identify the components that most constrain your delivery pipeline and address those first. Containerization and infrastructure-as-code (IaC) tools can help abstract away legacy dependencies while a longer-term migration plan is executed.

3. Silo Mentality Among Teams

Organizational silos—where development, QA, security, and operations each optimize for their own metrics without regard for the others—are fundamentally incompatible with DevOps principles. Siloed teams create hand-off bottlenecks, blame cultures, and slow feedback loops.

Resolution: Form cross-functional product teams that include members from development, operations, QA, and where appropriate, security. These teams should own a service or product end-to-end, from code commit to production monitoring. Shared tooling, shared dashboards, and shared post-mortems help reinforce a collective sense of ownership over outcomes.

4. Resistance to Change

People naturally resist change, and DevOps transformation asks a great deal of individuals: new tools, new workflows, new responsibilities, and new relationships with colleagues they may rarely have interacted with before. In some cases, employees may perceive automation or new roles as a threat to their job security.

Resolution: Address resistance head-on through transparent communication. Explain what DevOps means for each team, how roles will evolve rather than disappear, and what support—training, coaching, new career paths—is available. A phased rollout allows people to build confidence gradually. Celebrate early wins publicly to demonstrate that the approach is working and that individuals within it are valued.

5. Difficulties in Scaling Up

DevOps often shows early promise within a single team or a pilot project, only for momentum to stall when the organization tries to scale the model across multiple teams, products, or business units. Practices that worked at small scale—informal communication, ad-hoc tooling choices—become bottlenecks at larger scale.

Resolution: Establish a DevOps center of excellence or platform engineering team to define and maintain shared standards, toolchains, and best practices before rolling out broadly. A standardized internal developer platform reduces duplication of effort and ensures that teams scaling into DevOps benefit from what earlier adopters learned. Document your reference architectures and onboarding processes so adoption is repeatable.

6. Security and Governance Issues

The emphasis on speed in DevOps can create pressure to skip or defer security and compliance controls. When security is treated as a gate at the end of the pipeline rather than a property built into it, the result is either delayed releases or, worse, vulnerabilities shipped to production.

Resolution: Adopt a DevSecOps approach by integrating security scanning, policy enforcement, and compliance checks directly into the CI/CD pipeline. Static analysis (SAST), dependency vulnerability scanning, and infrastructure policy validation tools can run automatically on every commit. Application Release Automation (ARA) platforms can also help enforce governance policies without introducing manual bottlenecks. Security becomes a shared responsibility rather than a final checkpoint.

7. Excessive Focus on Tools

The DevOps tooling ecosystem is vast, and it is tempting to assume that adopting the right set of tools will, by itself, produce a DevOps culture. It will not. Tools that are poorly integrated with existing infrastructure, inadequately maintained, or adopted without team buy-in create complexity rather than reducing it.

Resolution: Choose tools that solve specific, identified problems and that integrate well with your existing stack. Evaluate tool choices against your security requirements before adoption. Most importantly, remember that people and processes drive outcomes—tools only amplify what teams are already doing. Invest in training so that teams understand not just how to use a tool but why it was chosen and what problem it is solving.

8. Budget Overruns

DevOps transformations frequently cost more than initially estimated. Training, tooling licenses, infrastructure upgrades, consulting engagements, and the productivity dip that accompanies any significant workflow change all contribute to costs that are easy to underestimate during early planning.

Resolution: Build a realistic budget that includes a contingency buffer for unexpected costs. Break the transformation into phases with defined outcomes so that each phase can be funded, evaluated, and course-corrected independently. Track return on investment metrics—deployment frequency, lead time for changes, incident rates—to build a business case for continued investment.

A People-Process-Technology Roadmap

Successful enterprise DevOps adoption requires a roadmap organized around three pillars:

• People: Build cross-functional teams, invest in training, address cultural resistance, and establish shared accountability metrics.
• Process: Define standardized, automated pipelines for integration, testing, security, and deployment. Document and iterate on these processes continuously.
• Technology: Modernize infrastructure incrementally, choose tools that integrate well and meet security requirements, and build a shared internal platform that enables teams to self-serve.

Organizations that address the eight challenges outlined above—particularly at the start of their DevOps journey rather than after problems have already emerged—are far better positioned to achieve continuous delivery, robust monitoring, and the cross-team visibility that makes DevOps genuinely transformative.

DevOps is not a destination but an ongoing practice. The teams that treat each challenge as a learning opportunity rather than a failure build the adaptive culture that sustains DevOps long after the initial transformation is complete.

The Problem with Monolithic Architecture

For decades, Federal agencies and large enterprises built their software the same way they built their headquarters: as massive, rigid, monolithic structures. A single large application — a benefits payment system, a tax calculation engine, a warfighter logistics platform — would grow over time, accumulating complexity, technical debt, and fragility. When one component failed, the entire system could collapse. When demand spiked unexpectedly, the only answer was to buy bigger hardware.

Before the DevOps movement and agile development practices took hold, software teams had little choice. Large applications could not easily be decomposed and distributed across teams or infrastructure. Resource bottlenecks — whether from the database, memory, or network — would choke the entire stack with no clean way to scale only the affected part.

What Microservices Are

A microservices architecture structures an application as a collection of small, loosely coupled services. Each service implements a specific, well-defined business capability — a tax calculation, an authentication check, an inventory lookup — and communicates with other services through simple, well-defined interfaces, typically lightweight APIs.

• Independent deployability: Each microservice can be built, tested, deployed, and scaled independently of every other service.
• Technology flexibility: Different services can use different languages or data stores if the team chooses.
• Fault isolation: A failure in one service does not automatically cascade and bring down the entire application.
• Team autonomy: Small, focused teams own individual services end-to-end, reducing coordination overhead and enabling faster iteration.

As David Egts, chief technologist at Red Hat, put it: instead of one large team working on one large application, you can have small teams working on different microservices — and all they have to worry about is their own service.

What the Data Shows

In the fall of 2017, Red Hat conducted a survey of Red Hat JBoss Middleware and Red Hat OpenShift customers to understand how organisations were adopting microservices. The survey explored adoption patterns, perceived benefits, key challenges, and competitive advantages.

According to that survey, 69 percent of respondents reported using microservices for both new greenfield applications and for re-architecting existing legacy systems. This dual use — not just for new projects — suggests that microservices deliver value at multiple stages of an organisation’s IT transformation, not only when starting from scratch.

Why Microservices and the Cloud Are a Natural Fit

Cloud infrastructure is built around elasticity — the ability to provision and release compute resources rapidly in response to demand. Microservices are architecturally aligned with that model in a way that monolithic applications are not.

Consider a practical example: an online retailer runs a sales tax calculation service. On a high-traffic day like Black Friday, that single service receives an enormous surge of requests. With a microservices approach running on a cloud platform, the organisation can spin up additional containers running that specific service to absorb the load — and then scale them back down once traffic normalises. No capacity is wasted during quiet periods, and no users are turned away during peaks.

In the era of mainframes and Unix servers, the only path to more capacity was purchasing larger hardware. Cloud-based microservices replace that model with dynamic, on-demand scaling — and the same principle applies to government agencies managing variable workloads.

Cloud platforms can also automatically shift microservice containers to healthy servers when underlying software or hardware fails, providing continuous availability without manual intervention.

The DevOps Angle: Continuous Delivery at Scale

Microservices architecture is not just an infrastructure concern — it has direct implications for how DevOps teams work. The goal is simple to state and difficult to achieve: deliver quality software faster. Microservices support that goal in several concrete ways:

• Faster release cycles: Because services are independent, a team can release an update to the payment service without waiting for a full application deployment cycle.
• Reduced blast radius: A bug in one service does not necessarily require rolling back the entire application.
• Parallel development: Multiple small teams can work simultaneously on different services without blocking each other.
• Continuous integration and delivery: Each microservice can have its own CI/CD pipeline, enabling truly continuous deployment across a large, complex system.

Challenges DevOps Teams Must Address

Breaking an application into microservices is not the end of the work — in many ways, it is the beginning. DevOps teams adopting microservices need to address a range of new operational concerns:

• Service orchestration: Managing dozens or hundreds of services requires a container orchestration platform such as Kubernetes or Red Hat OpenShift to handle scheduling, scaling, and health management.
• Observability: Distributed systems are harder to debug. Teams need robust logging, distributed tracing, and monitoring across all services.
• Data management: Each service may own its own data store. Keeping data consistent across services adds architectural complexity.
• Network complexity: Service-to-service communication introduces latency and failure modes that do not exist in a monolith. Service meshes can help manage this.
• Security: Each service endpoint is a potential attack surface. API authentication, encryption in transit, and least-privilege access controls must be applied consistently.

Once these operational challenges are addressed, the result is a highly agile network of services that teams can deploy, scale, or retire independently — and that agencies can adapt quickly to changing requirements or usage patterns.

On-Premises vs. Cloud-Based Microservices

Microservices can be deployed both on-premises and in the cloud. For organisations with strict data sovereignty or compliance requirements — common in government environments — on-premises container platforms provide many of the same architectural benefits. However, cloud-based deployments offer additional advantages: elasticity is easier to realise, managed services reduce operational burden, and multi-region deployments for high availability are more accessible.

Hybrid approaches — running some services on-premises and others in the cloud — are also viable and increasingly common for agencies navigating complex regulatory environments.

Key Takeaways for Agency DevOps Teams

• Microservices decompose large, fragile monoliths into independently deployable, scalable units aligned with specific business capabilities.
• The architectural approach supports continuous delivery pipelines and autonomous team structures — core DevOps principles.
• Cloud elasticity and microservices are complementary: scaling individual services on demand reduces waste and cost while maintaining availability.
• Adopting microservices requires investment in orchestration, observability, and operational tooling — breaking up the monolith is only the first step.
• Both new projects and legacy modernisation efforts can benefit from a microservices approach.

DevOps at the End of 2017: A Foundation Being Tested

By the close of 2017, DevOps had spent roughly half a decade moving from buzzword to mainstream methodology. The core premise — that software development and IT operations teams should collaborate closely, share responsibility, and automate wherever possible — had proven itself in measurable ways. High-performing DevOps organizations were deploying software significantly more frequently, recovering from failures faster, and reducing the number of change-related incidents compared to organizations still operating in siloed models.

But as adoption grew, so did the complexity of the problems DevOps teams faced. Security, database management, AI integration, and organizational culture were all emerging as friction points. Going into 2018, a number of industry practitioners and executives shared their predictions on where the discipline was headed.

DevSecOps: Security Moves Left — and Into Everything

One of the clearest themes across 2018 predictions was the convergence of development, operations, and security into a unified practice increasingly called DevSecOps.

“Breaches and attacks are only going to increase and viruses are getting more sophisticated. Application development is going to need to natively include specific safeguards and protections. Security tools that are layered into the infrastructure are still needed — they just aren’t going to be enough as more and more apps adopt the cloud model.”

Manoj Nair, Product Director, HyperGrid

The argument here is structural: bolting security on at the end of the delivery pipeline — or relying solely on perimeter-level controls — was no longer viable as cloud-native architectures, microservices, and container-based workloads became the norm. Security needed to be embedded at every stage of the CI/CD pipeline, from code commit through deployment.

“We will continue to see end-users make a tighter connection between a company’s brand and the quality of its code, based on their experiences across a company’s applications. As a result, more organizations will look to integrate security across all phases of development and intensify their automated continuous testing efforts as they work to release higher quality code, faster.”

Aruna Ravichandran, VP of DevOps Product and Solutions Marketing, CA Technologies

This prediction aligned with what was already being observed in the market: security scanning tools being integrated directly into CI pipelines, infrastructure-as-code being linted for misconfigurations before deployment, and compliance checks being automated rather than performed manually at release gates.

Cloud Security and the Shared Responsibility Model

A specific prediction tied to cloud adoption was the growing acceptance — and operationalization — of AWS’s shared responsibility model. The model divides security obligations between the cloud provider (responsible for the security of the cloud) and the customer (responsible for security in the cloud). In practice, many organizations had struggled to act on their side of that boundary.

The 2018 prediction was that DevOps practices — particularly the ability to script, version, and automate infrastructure configuration — would become the primary mechanism through which organizations fulfilled their cloud security responsibilities. Automated policy enforcement, configuration drift detection, and secrets management were all expected to mature as core DevOps capabilities rather than afterthoughts.

The Database: DevOps’ Forgotten Component

Several contributors independently identified the same blind spot: while DevOps had made significant progress automating application builds, tests, and deployments, the database remained largely manual — and was becoming a serious bottleneck.

“The database is the hardest part in the application stack to manage, so it just doesn’t make sense that it’s always the forgotten piece of the puzzle. IT teams have been so focused on time-to-market and getting development to push out applications at the speed of light, but still manually manage the change process of databases that contain massive amounts of information.”

Robert Reeves, Co-founder and CTO, Datical

Reeves went further, framing DevOps not as a toolchain but as a continuous process of identifying and eliminating friction — and that the database was an obvious source of that friction that the industry had been reluctant to address:

“It’s time to stop having DBAs perform manual SQL script review prior to a release and start automating the review so that they can continue to innovate and bring strategic value to their organization.”

Robert Reeves, Datical

Patrick O’Keeffe of Quest Software reinforced this, predicting that 2018 would see a shift in DevOps focus from upstream concerns (source control, CI) to downstream delivery — and that DBAs would need to become active participants in that pipeline rather than gatekeepers outside it.

“Due to their complex nature and common data movement challenges, databases tend to pose major bottlenecks to DevOps teams and processes. The DBA will play an integral role in alleviating these challenges and play a critical role in how businesses move to enabling DevOps digital transformation.”

Patrick O’Keeffe, Executive Director of Software Engineering, Quest Software

The Emergence of DataOps

Alongside the database conversation, O’Keeffe introduced a term that was just beginning to gain traction in late 2017: DataOps.

“Where DevOps aligns developers and IT teams to accelerate software delivery and infrastructure changes, DataOps is all about streamlining the preparation of data so developers can leverage it during the application building process.”

Patrick O’Keeffe, Quest Software

DataOps applies DevOps-style automation, version control, and collaboration principles to the data pipelines that feed analytics and machine learning workloads. In 2017, this was a nascent concept; by the early 2020s, it had developed into a recognized discipline with dedicated tooling and teams at larger data-driven organizations.

AI, Open Data, and What It Meant for DevOps Teams

Dr. John Bates, CEO of Testplant, offered a broader view that extended beyond DevOps mechanics into the emerging role of AI and data sharing in software engineering:

“AI and deep-learning will now drive the creation of ‘open data’ communities where people share data repositories to help train algorithms — much as open-source communities have shared code for decades.”

Dr. John Bates, CEO, Testplant

This prediction foreshadowed the growth of model hubs, shared datasets, and collaborative ML infrastructure that would accelerate in subsequent years. For DevOps practitioners, the practical implication was that managing data pipelines and model artifacts would become as important as managing application artifacts — a further argument for the DataOps discipline.

Bates also predicted that monitoring and testing would converge in DevOps pipelines — a trend consistent with what later became observable reliability engineering (SRE) practices, where production telemetry feeds back into test design and release decisions.

Product Owners Pulled Into the Pipeline

One organizational prediction from Bates deserves particular attention for how it reflected evolving thinking about continuous delivery:

“DevOps has been about ‘Dev’ and ‘Ops’, but in 2018 the business will be pulled directly into DevOps, especially product owners. This will enable the true continuous delivery envisaged.”

Dr. John Bates, Testplant

The argument is that genuine continuous delivery requires fast feedback not just from automated tests, but from business stakeholders who can evaluate whether features are achieving intended outcomes. Without that loop closed, teams can deliver code continuously while still building the wrong things — quickly.

The CIO Reclaims the Information Mandate

Bates made one further prediction that speaks to a governance challenge that grew more acute over subsequent years:

“As companies realize that their data is their most valuable asset, but that it’s now fragmented across functions who have all deployed independent SaaS solutions, they will look to the CIO to bring their data together. So the CIO will once again be the center of information.”

Dr. John Bates, Testplant

This framing — data fragmentation as an organizational risk, not just a technical one — anticipated the data governance conversations that would dominate enterprise IT strategy through the early 2020s, particularly around data mesh, data catalogs, and the difficulty of deriving consistent analytics from dozens of independent SaaS tools.

Key Themes in Retrospect

Looking at these 2018 predictions as a set, several themes stand out:

• Security shifts left: DevSecOps was no longer optional; embedding security into the pipeline was treated as a prerequisite for sustainable delivery velocity.
• The database bottleneck: DevOps automation had largely bypassed database change management, and this was increasingly recognized as a critical gap.
• DataOps as a discipline: Applying DevOps principles to data pipelines was an emerging need, particularly as AI and analytics workloads grew.
• Broader stakeholder inclusion: True continuous delivery required pulling product owners — and eventually business stakeholders — into the feedback loop.
• AI and open data: Machine learning was beginning to create new collaboration and tooling requirements that paralleled the open-source software model.

Many of these predictions proved accurate in direction if not always in timing. DevSecOps became a standard term and practice. Database DevOps tooling matured. DataOps became a recognized field. The CIO data governance question remains unresolved for many enterprises to this day.

This article is based on industry commentary originally published by eWeek in December 2017, covering predictions from practitioners at HyperGrid, Quest Software, Testplant, CA Technologies, and Datical.

Agile and DevOps: Complementary, Not Competing

A common misconception is that adopting DevOps means abandoning Agile. In practice, many high-performing teams use both together. Agile governs how software is developed—emphasising iterative delivery, responsiveness to change, and close collaboration within development teams. DevOps extends that philosophy to include the operations side of the house, closing the gap between writing code and running it reliably in production.

“Agile practices and DevOps are not mutually exclusive practices, and they are frequently used together to achieve continuous deployment of features to users. The key is using feature branching that is only merged in after a successful completion of comprehensive automated tests.”

David Lee, COO, Kastling Group

Marty Puranik, CEO of Atlantic.Net, frames the distinction clearly: Agile is primarily about producing software efficiently, often software that was historically packaged and sold. DevOps, by contrast, is about keeping that software running continuously in production—the SaaS operating model—which introduces its own set of ongoing responsibilities.

The Cultural Shift Is the Hard Part

Most practitioners who have lived through this transition will tell you that technology is the easy part. Culture is harder. Historically, development teams owned code up to the point of deployment, and operations teams owned everything after. DevOps asks both sides to share responsibility across the entire application lifecycle.

“Organizations that used to separate development and operations will now ask that the teams share the responsibility across the entire application process. Developers will be responsible not only for providing the code but also for following that code to completion. The same is true for operations.”

Lindsey Havens, Senior Marketing Manager, PhishLabs

Common values around ownership and accountability are central to a DevOps culture. Without them, tooling investments will deliver only marginal gains. Leadership needs to make the expectations explicit and support teams through the discomfort of changing deeply ingrained working habits.

Practical Steps for MSPs Moving from Agile to DevOps

1. Start Small and Iterate

The breadth of change involved in a DevOps adoption makes it tempting to plan a large, organisation-wide rollout. Resist that temptation. A pilot program with a small, dedicated team is far more likely to succeed and will generate the proof points needed to scale the approach across the wider organisation.

“For MSPs, a progressive IT motto should be: start small and iterate. DevOps comes with a fundamental shift in culture, process, skill sets and technologies. Using a small, core team to introduce the new working paradigm is essential. By focusing on delivering something of business value—small in scope and within the team’s control—it will empower the team to make decisions in real time.”

Jason McDonald, President, Contino

A solid, scalable IT architecture is foundational here. Starting small forces the team to focus on that foundation rather than trying to solve every problem at once. Once the foundation is in place, growth is organic and controlled.

2. Choose Tools That Support the Process—Not the Other Way Around

Both Agile and DevOps have rich tooling ecosystems, but experienced practitioners caution against letting tool selection drive process decisions. The tool should serve the process, not define it.

“A favourite tool of agile is the Kanban board, which is literally made of notecards and tape. But it accomplishes the goal of visualising work-in-progress—a key component of an efficient agile process.”

Ned Bellavance, Director of Cloud Solutions, Anexinet

For teams that do want a reference stack, tools commonly mentioned in DevOps contexts include:

• Issue and project tracking: Jira, Azure DevOps (formerly Visual Studio Team Services)
• Continuous integration: TeamCity, Jenkins, GitHub Actions
• Continuous delivery: Octopus Deploy, Spinnaker
• Infrastructure as code: Terraform, Ansible, PowerShell DSC

The right answer for any given organisation will depend on its existing investments, team skills, and the cloud platforms it targets. Adopting an entirely new toolchain simultaneously with a cultural change is a recipe for overload—where possible, extend existing tools before replacing them.

Security Cannot Be an Afterthought

One of the most significant risks in moving from Agile to DevOps is carrying forward legacy security habits that were designed for slower release cycles. In an Agile context, security is often treated as another user story to be prioritised in the backlog. In a continuous deployment pipeline, that model breaks down—there is no natural gate at which security reviews can occur if releases are happening multiple times per day.

“Hackers also use DevOps methods in their attacks, so organisations must develop and implement approaches that act with the same rapidity of DevOps, while also integrating security throughout each stage of the development and deployment lifecycle.”

John Martinez, VP of Customer Solutions, Evident.io

The answer is DevSecOps: embedding security controls directly into the pipeline so they run automatically at every stage. Practical examples include:

• Static application security testing (SAST) integrated into the CI build step to catch vulnerabilities in code before it is merged.
• Infrastructure-as-code scanning to verify that cloud resource configurations are secure before deployment.
• Continuous compliance monitoring of running cloud environments to detect configuration drift against defined security baselines.
• Automated secrets management to prevent credentials from being baked into container images or configuration files.

“Choose effective tools and automate their use. Include security to earn the moniker DevSecOps. For example, include a security check to make sure that your infrastructure as code is secure.”

Josh Rosenthal, Co-founder, CloudSploit

Security must be part of the fabric of the product, present and accounted for at every stage of the DevOps lifecycle, not bolted on at the end.

Compliance in Cloud Environments

For cloud companies and MSPs serving regulated industries, compliance adds another dimension to the Agile-to-DevOps transition. Security and compliance requirements need to be encoded into the pipeline itself—automated checks that verify infrastructure and application configurations meet the required standards before anything reaches production. Clear metrics on transition timelines and integration guidelines help teams move with confidence rather than hesitation.

Establishing compliance as code—expressing regulatory controls as testable, version-controlled policies—aligns well with a DevOps operating model and removes the manual verification bottlenecks that slow down continuous delivery.

Key Takeaways

• Agile and DevOps are complementary. DevOps extends Agile principles from development into operations and production.
• Culture change is the primary challenge. Shared ownership and accountability across dev and ops teams must be established deliberately.
• Start with a pilot. A small, focused team reduces risk and creates the evidence base needed to scale adoption.
• Let process drive tool selection. Tools should serve the workflow, not define it.
• Shift security left. Integrate automated security and compliance checks into every stage of the pipeline from day one.
• Onboard developers properly. New working practices—feature branching, automated testing gates, shared on-call responsibility—need to be part of the standard onboarding process, not discovered after the fact.

Data centre strategies for cloud-scale software are fundamentally different from those designed to support traditional, on-premises enterprise applications. Whether workloads are hosted on public IaaS, in a private cloud, or in a managed private cloud, infrastructure planners need to account for characteristics that make modern cloud software architecturally distinct. This article breaks down those characteristics and their practical implications for infrastructure design.

Why Cloud-Scale Software Demands a Different Infrastructure Approach

Traditional enterprise software was typically deployed for a known, bounded user population on dedicated hardware. Cloud-scale software inverts nearly every one of those assumptions: user populations are large and variable, deployments are shared, release cycles are compressed, data volumes are enormous, and analytical workloads are central rather than peripheral. Each of these factors has direct consequences for how data centres must be designed and operated.

1. Application Multi-Tenancy

Cloud software is inherently multi-tenant: many users from different organisations share the same underlying infrastructure while maintaining strict logical separation of their private data. This architectural pattern places specific demands on data centre infrastructure:

• Consistent performance isolation. A surge in activity from one tenant must not degrade the experience of others. Infrastructure must provide predictable, isolated performance under variable aggregate load.
• Shared-resource efficiency. Multi-tenancy only delivers its cost and operational benefits when the infrastructure can efficiently pool and reallocate compute, memory, network, and storage resources across tenants in real time.
• Security boundaries. Logical separation must be enforced at every layer — networking, storage, and application runtime — to prevent data leakage between tenants.

Hyperconverged infrastructure (HCI) and software-defined storage platforms have become popular choices for multi-tenant deployments precisely because they allow fine-grained quality-of-service policies to be applied per tenant or per workload.

2. Agile Development and DevOps Velocity

One of the core advantages of centralised, cloud-scale infrastructure is the ability to ship new features frequently — weekly or even daily release cycles are common in mature DevOps organisations. This cadence creates infrastructure requirements that are easy to overlook during initial capacity planning:

• Programmable, API-driven provisioning. Teams cannot afford to wait days for manually provisioned test or staging environments. Infrastructure must expose APIs that CI/CD pipelines can consume to spin environments up and tear them down automatically.
• Immutable deployments and rapid rollback. Container-based and immutable-infrastructure patterns require storage and networking layers that support fast image distribution and stateless compute scaling.
• Observability from day one. High-frequency releases increase the probability of regressions reaching production. Infrastructure must expose rich telemetry — metrics, logs, and traces — that development teams can consume alongside application-level signals.

Infrastructure that requires lengthy change-management cycles or manual intervention becomes a bottleneck that negates the value of the DevOps practices built on top of it. The data centre strategy must treat infrastructure-as-code as a first-class requirement, not an afterthought.

3. Elastic and Cost-Efficient Scale

Cloud-scale businesses are built on the assumption of growth: more users, more connected devices, more data. Data centre infrastructure must scale seamlessly and predictably to match that growth. This has two interrelated dimensions.

Technical Scalability

Infrastructure must scale horizontally without architectural re-engineering. Storage, compute, and networking capacity should all be addable in incremental units without requiring application downtime or data migration. Scale-out architectures — distributed storage clusters, disaggregated compute pools, software-defined networking — are better suited to this requirement than traditional scale-up, monolithic designs.

Cost Scalability

One well-known challenge with public IaaS is that storage and compute costs tend to grow nearly linearly with workload volume, making it difficult to realise economies of scale at the infrastructure layer as the business grows. A well-designed data centre strategy — whether built on private cloud, colocation, or a hybrid model — should aim to improve cost efficiency per unit of workload as scale increases, rather than simply tracking it linearly. This typically means evaluating:

• Total cost of ownership (TCO) of owned or leased infrastructure versus on-demand public cloud spend at projected scale.
• Reserved or committed-use pricing models on public IaaS for predictable baseline workloads.
• Hybrid architectures that run steady-state workloads on owned infrastructure while bursting to public cloud for unpredictable peaks.

4. High-Performance Analytics

Analytics is no longer a secondary, batch-oriented workload running overnight in a data warehouse. For modern cloud-scale applications, real-time analytics is a core part of the product experience and the operational backbone of the business:

• Personalisation engines in eCommerce and streaming platforms make real-time decisions based on user behaviour data at massive scale.
• Business intelligence in enterprise SaaS applications surfaces insights that must reflect near-current state of operational data.
• Machine learning inference for AI-enabled features — recommendations, fraud detection, image recognition — runs continuously against live data streams.
• Conversational interfaces and chatbots rely on low-latency model inference to maintain acceptable response times.

These workloads share a common infrastructure requirement: very high I/O throughput combined with low, consistent latency. All-flash storage arrays and NVMe-based storage architectures have become key enablers for delivering analytics performance at the scale cloud software demands. Emerging NVMe-over-Fabrics (NVMe-oF) networking protocols promise to extend the low-latency characteristics of local NVMe storage across a shared network fabric, enabling storage disaggregation without the latency penalty of traditional SAN or NAS.

Navigating a Complex Ecosystem

Product, DevOps, IT, and finance teams charting data centre strategies for cloud-scale applications face an increasingly complex decision landscape. The technology choices — hyperconverged infrastructure, software-defined storage, bare-metal cloud, colocation, public IaaS, hybrid architectures — are not mutually exclusive, and the right answer is rarely a single vendor or a single deployment model.

A practical evaluation framework should test candidate strategies against each of the four dimensions above:

1. Multi-tenancy: Does the infrastructure enforce performance isolation and data security between tenants without requiring over-provisioning?
2. DevOps agility: Can the infrastructure be provisioned, configured, and decommissioned entirely through APIs and infrastructure-as-code tooling?
3. Scale economics: Does the cost model improve — or at least hold flat — as workload volume grows, rather than scaling linearly?
4. Analytics performance: Can the storage and compute layers deliver the throughput and latency required by real-time analytical workloads at production scale?

Infrastructure vendors and cloud service providers that can credibly address all four dimensions will be well positioned to support the next generation of cloud-scale software. Those that excel on compute or networking alone, but create bottlenecks at the storage or orchestration layer, will increasingly be disqualified from shortlists as engineering teams become more sophisticated in their evaluations.

Key Takeaways

• Cloud-scale software requires infrastructure designed for multi-tenancy, rapid deployment cycles, elastic scale, and high-performance analytics — not simply more of what supported traditional enterprise software.
• DevOps velocity is only achievable when the underlying infrastructure is fully API-driven and can be managed as code.
• Cost efficiency at scale requires deliberate architectural choices; public IaaS alone does not automatically deliver economies of scale.
• Real-time analytics workloads are increasingly central to cloud software products and demand low-latency, high-throughput storage infrastructure such as all-flash and NVMe-based systems.
• Most production environments will adopt hybrid strategies, mixing public IaaS, private cloud, and colocation to balance agility, performance, and cost.

The Security Gap Hidden Inside Your CI/CD Pipeline

Modern Agile and DevOps practices have fundamentally changed how software is shipped. Instead of quarterly or annual releases, teams push small, frequent updates daily or weekly, supported by continuous integration (CI) pipelines that automate building, testing, and packaging code. This velocity is a competitive advantage — but it comes with a compounding risk: every new build is also a potential new attack surface.

APIs are at the centre of this risk. They power the backends of mobile apps, web applications, microservices, IoT devices, and more. As API adoption continues to grow, so does the opportunity for security vulnerabilities to slip into production — especially when security testing is bolted on at the end of a development cycle rather than integrated from the start.

The answer is DevSecOps: treating security as a first-class concern that is automated, continuous, and owned jointly by development, operations, and security teams — not handed off to a separate team at the end of a sprint.

Why APIs Are a Priority Target

APIs underpin virtually every category of modern software. A vulnerability in a shared API can affect multiple services simultaneously, making API security testing a high-leverage activity. Critical service categories that depend on APIs include:

• Mobile applications
• Web applications
• Desktop applications
• Browser-based applications
• Microservices architectures
• IoT platforms
• Embedded devices

Despite this breadth of exposure, the security tooling available to test APIs has historically lagged behind the pace of modern development. Several structural gaps make the status quo inadequate.

The Problems With Legacy Security Testing Approaches

1. Tools Are Not API-Specific

Many security tools in common use are generalist web scanners, not purpose-built API testing tools. They work by pointing at a single endpoint, crawling available routes, and capturing traffic between the API and a client. The tool then returns a pass/fail result — but identified vulnerabilities still need to be manually located, verified, and triaged by a penetration tester before they can be acted on by a developer.

Older point-and-shoot scanners also require manual configuration for each endpoint before testing can begin. This is time-consuming, error-prone, and typically only allows one test to run at a time. Many of these tools also lack support for modern authentication schemes (OAuth, JWT, API keys), resulting in poor code coverage and missed vulnerabilities.

2. Limited Integration With Developer Toolchains

Security tools that cannot integrate with the tools developers already use create friction and are quickly sidelined. Most legacy security tools lack integrations with CI build systems, log management platforms, and bug-tracking software (such as Jira or GitHub Issues). Because they are designed for security specialists rather than engineering teams, they require manual effort to test, verify, and document vulnerabilities — making them both slow and expensive to operate at scale.

3. Testing Happens Too Late in the Cycle

Because integrating existing security tools into developer workflows is difficult, security testing is frequently deferred to the end of the development process, performed by dedicated security professionals just before a major release. This is the most expensive time to find bugs. The longer a vulnerability lives in the codebase, the more code has been written around it and the more costly it is to fix.

The time and cost pressures of manual testing also encourage teams to test only once per major development cycle rather than with every build. This means intermediate builds — which may contain newly introduced vulnerabilities — are regularly promoted to production without being tested.

4. Vulnerable Builds Cannot Be Automatically Blocked

Most legacy security tools have no mechanism to automatically halt a build pipeline when a known vulnerability is detected. Without this gate, discovered vulnerabilities must be remediated manually and out-of-band, often causing significant delays to ship dates — or being silently deferred and shipped anyway.

The DevSecOps Model: Security Baked Into Every Build

DevSecOps addresses these gaps by automating security testing and integrating it directly into the CI/CD pipeline. Rather than treating security as a separate phase, DevSecOps makes it a continuous, automated step in the same workflow engineers already use to build and ship software. Security findings become build artifacts, not separate reports.

The DevSecOps approach is built around seven key practices for API security in particular:

1. API-Specific Security Testing

Use tools that are designed to test modern APIs, not generic web scanners. Testing should be driven by up-to-date vulnerability lists that reflect how APIs are actually attacked — including the OWASP Top 10 and API-specific variants. Many of the most serious vulnerabilities emerge from complex interactions between multiple APIs and services, which point-and-shoot scanners are poorly equipped to surface.

2. Automated Test Case Generation

Engineering teams already write unit and integration tests to verify that their APIs behave correctly. These tests are designed with good code coverage and include proper authentication flows. Rather than building a separate suite of security tests from scratch, convert existing unit tests into security tests automatically. This approach shifts security testing “left” — earlier in the development lifecycle — where defects are cheaper and faster to fix.

3. Do Not Break the Developer Workflow

Security checks should run as a step inside the existing CI pipeline — not as a separate out-of-band process that requires engineers to switch tools. When a vulnerable build is detected, the pipeline should automatically prevent that build from being promoted to production. All results — faults, logs, ticket creation — should surface within the tooling engineers already use. Asking teams to learn and operate a separate security platform adds friction and reduces adoption.

4. Fuzz Testing

Fuzzing involves mutating valid API requests — changing data types, boundary values, encoding, and structure — to discover vulnerabilities that are not covered by standard OWASP-based tests. Fuzzing is particularly effective at uncovering logic errors and memory-handling bugs that structured tests will miss. Including a fuzzing step in the pipeline significantly increases the chances of catching zero-day class vulnerabilities before they reach production.

5. Configurable Testing Profiles

Teams that ship multiple times per day cannot run a full, hours-long security scan on every commit. Testing profiles allow teams to balance coverage against speed: a lightweight profile can run on every push, while a comprehensive scan runs nightly or before a major release. Each profile should be configurable so teams can include or exclude specific checks, test modes, or endpoint groups depending on the context and deadline.

6. Actionable Fault Results

A security finding is only useful if a developer can act on it quickly. Fault reports should include enough context for a developer to understand, reproduce, and fix the issue without requiring a security specialist to translate the finding. Ideally, fault results are delivered directly into the bug-tracking or CI system the team already uses — not into a separate security dashboard that engineers rarely visit.

7. False Positive Management

False positives are one of the most significant productivity drains in automated security testing. If engineers must manually triage every reported fault to determine whether it is a real vulnerability or a false alarm, the overhead quickly becomes unsustainable. A well-designed DevSecOps tool allows teams to mark confirmed false positives so that they are suppressed in future runs. Future builds still test for the condition, but do not block pipelines or create noise in the ticket system — preserving both coverage and developer trust in the tooling.

Conclusion: Security Is an Engineering Problem Now

The shift to DevSecOps is not primarily about adopting new tools — it is about accepting that security is now part of the engineering team’s job, not a downstream audit performed by a separate function. As API-driven architectures become the norm and CI/CD pipelines shorten release cycles to hours rather than months, the window for catching security issues before they reach production shrinks accordingly.

Organizations that embed automated, API-specific security testing into their existing build pipelines — shifting testing left, automating test generation, configuring build gates, and managing false positives at scale — will be far better positioned to ship fast without accumulating the kind of security debt that ends up on the front page of the news.

The goal is not to slow development down with security. The goal is to make security invisible to developers by making it automatic.

Why DevOps Looks Like a Farce (at First)

Consider for a moment the core practices DevOps advocates for, and then imagine arguing against each of them in turn:

• Developers collaborating directly with users — Sounds risky. Surely customers should only interact with a formal interface?
• Developers and operations staff working together — Why not keep all communication through a ticketing system?
• Operations included in release planning — Won’t they just slow things down?
• Customer service briefed on new features before release — What value does that add?
• Test-first development practices — Seems like overkill for most teams.

Every one of those objections reflects how many organizations actually operated before the agile movement — and how some still operate today. The irony is that IT staff often have the clearest view of what keeps systems stable, and customer service representatives frequently surface the most actionable product feedback, simply because they interact with both the software and its users every day. Locking those people out of the development process is a real organizational cost, not a risk mitigation strategy.

Bimodal IT in a Nutshell

Gartner’s Bimodal IT model proposes that organizations run two distinct modes of IT delivery in parallel:

• Mode 1 — Predictable, stable, and deliberate. Suited to legacy systems and core business applications where reliability is paramount. Change is gradual and well-understood.
• Mode 2 — Exploratory and fast-moving. Suited to new product development, emerging technology adoption, and innovation initiatives where speed and experimentation matter more than certainty.

The underlying assumptions of Bimodal are worth naming explicitly:

• Agile and DevOps are best suited to new projects and nimble organizations.
• Applying DevOps to legacy systems introduces unacceptable risk.
• Organizations must choose between speed (DevOps) and quality (traditional approaches).
• Different applications require fundamentally different operating modes.

The Upsides of Bimodal IT

Bimodal IT is not without merit. Its strongest contribution is the explicit encouragement of pilot projects around emerging technology areas. Organizations have historically benefited from treating early-stage technology exploration — whether cloud adoption, NoSQL databases, mobile application development, or earlier waves like the move to the web — as a distinct track that operates under different rules and risk tolerances than core systems. For many enterprises, the instinct to ring-fence experimentation is pragmatic and organizationally sensible.

The Downsides of Bimodal IT

The problems with Bimodal IT become apparent when you try to apply it consistently across a real organization:

Who Decides Which Mode an Application Belongs To?

Categorizing applications as Mode 1 or Mode 2 forces difficult and often arbitrary value judgments. Which applications — and therefore which users — deserve faster delivery? More innovation? Higher quality? Better support? Bimodal does not provide a clean answer to those questions, and in practice, the answers tend to reflect organizational politics rather than user needs.

It Recreates the Silos DevOps Was Designed to Eliminate

Running two separate development processes, toolchains, and team cultures creates exactly the kind of organizational fragmentation that DevOps addresses. Mode 1 teams risk becoming a professional dead-end. Mode 2 teams risk becoming disconnected from the operational realities that keep production systems stable.

Most Applications Cannot Be Cleanly Categorized

Modern systems are rarely self-contained. Consider airline reservation and ticketing applications: the mobile-facing booking interface may follow a fast agile release cycle, but it is almost certainly tightly coupled to legacy transactional back-end systems. A change on either side typically requires coordination with the other. Labeling one component Mode 1 and the other Mode 2 does not eliminate that interdependency — it just obscures it.

The Speed-Quality Tradeoff Is a False Choice

Research from Puppet Labs and others (including the annual State of DevOps reports) has consistently found that high-performing technology organizations deliver both faster deployment frequency and higher reliability than low-performing organizations — not one at the expense of the other. The tradeoff Bimodal assumes is empirically weak. Additionally, practitioners including Gene Kim have cited data suggesting that a significant proportion of successful DevOps transformations — reportedly around 60% — involve legacy rather than greenfield applications, which challenges the assumption that DevOps is unsuitable for established systems.

How to Reconcile Bimodal IT with DevOps

The underlying intent of Bimodal IT — recognizing that different systems carry different risk profiles, and that organizational change requires structure — is reasonable. The implementation, however, tends to produce the wrong outcomes. The following approaches preserve that intent while avoiding the pitfalls.

Focus on People, Not Applications

Every organization has individuals who are more receptive to new methods, regardless of which system they support. Instead of assigning an application to Mode 1 or Mode 2 and locking its team into that mode, identify the people most likely to succeed with a DevOps approach and build early wins around them. Let demonstrated success drive broader adoption rather than top-down categorization.

Invest in Collaboration and Feedback Loops

Most organizations — regardless of application type — have room to improve communication between development, operations, and end users. Shortening feedback loops is beneficial across the board. It is not a Mode 2-only concern.

Break Down Silos Rather Than Formalizing Them

Bimodal can inadvertently legitimize the organizational silos that DevOps is designed to dismantle. People generally do not want to be confined to a narrow, unchanging role. Offering expanded responsibilities and cross-functional opportunities tends to be more motivating and more productive than enforcing strict operational boundaries.

Treat Shadow IT as a Signal, Not a Problem

Unsanctioned technology adoption — shadow IT — has a long history of producing genuine innovation, from early PC adoption to the emergence of cloud services. Rather than suppressing it, investigate why teams are working around official channels and address the underlying friction. The goal is to make the official path less painful, not to eliminate the pressure that drives people to find workarounds.

Reframe Speed as Speed to Value

The concern that DevOps moves too fast for legacy applications usually conflates release velocity with release risk. A more productive framing is speed to value: getting the most important changes to users sooner, in smaller increments, with fewer changes per release. Smaller releases are inherently lower risk than large batch deployments, regardless of the age of the underlying system.

Expand the DevOps Definition Beyond Dev and Ops

DevOps is a useful shorthand, but the label can create artificial boundaries. The principles apply equally to quality assurance, customer service, product management, and executive leadership. A broader definition of who participates in DevOps practices — and who benefits from them — will produce better outcomes than one confined to engineering teams.

Conclusion

Bimodal IT and DevOps are not inherently incompatible. The stability and risk-awareness that Bimodal advocates for are legitimate organizational concerns. The rapid delivery, continuous feedback, and cross-functional collaboration that DevOps advocates for are not threats to stability — they are, when properly implemented, mechanisms for achieving it.

The key shift is to stop treating application age as a proxy for the appropriate development approach, and to stop treating speed and quality as a zero-sum tradeoff. Innovation can and should happen across your entire portfolio — in legacy systems as much as in new ones. A DevOps practice that is inclusive of the whole organization, measured honestly, and built on genuine cross-functional collaboration will reduce risk, increase agility, and create competitive advantage from systems that might otherwise be treated as liabilities.

But agile is not a silver bullet. Many teams adopt the methodology only to find that their projects stall, quality suffers, or stakeholder frustration grows. Understanding why agile fails in practice is the first step toward preventing those failures.

The following six patterns are among the most common reasons agile projects lose momentum—or grind to a halt entirely.

1. Unsustainable System Architecture

The Agile Manifesto emphasizes that “good design and technical excellence enhance agility,” and the community broadly advocates for adaptive, emergent architecture. The challenge is knowing how much up-front design is enough. Too little design leads teams to layer new code on top of old code, sprint after sprint, until the codebase becomes rigid and costly to change—the very opposite of agile.

Unlike traditional waterfall projects, where a formal architectural blueprint is produced before a single line of code is written, agile architecture is an ongoing activity. A design that looks coherent at sprint planning can quietly drift into something unmaintainable over several iterations if there is no periodic architectural review. Technical debt accumulates silently until it becomes a blocker.

Mitigation: Reserve time in each sprint for architecture review and refactoring. Lightweight techniques such as Architecture Decision Records (ADRs) give teams a way to document key design choices without heavyweight upfront documentation.

2. Limitations of Existing Tooling

Many enterprise environments still rely on Application Lifecycle Management (ALM) tools built around waterfall assumptions—monolithic, on-premises platforms that were not designed for iterative delivery. When organizations attempt to graft agile workflows onto these tools, they often find the tooling working against the process.

Modern agile toolchains—such as Jenkins for CI/CD, Jira or similar issue trackers, and purpose-built test management platforms—were designed for software engineers. Business stakeholders and non-technical users who need to provide acceptance feedback are often unable or unwilling to work within these systems. The result is a communication gap between the delivery team and the people whose feedback is essential to agile success.

Beyond usability, enterprise-grade tooling can be expensive to license, maintain, and integrate, and it frequently carries hardware requirements that are disproportionate to its actual utility on smaller projects.

Mitigation: Audit your toolchain from the perspective of every role that needs to participate—including business users. Choose tools with lightweight interfaces for non-technical stakeholders, and prefer cloud-hosted options to reduce infrastructure overhead.

3. The Culture Gap

Agile demands a fundamentally collaborative culture. Daily standups, retrospectives, pair programming, and cross-functional teams all require individuals to work closely together, share work-in-progress openly, and accept continuous feedback. For developers who are accustomed to working autonomously—and who may genuinely produce excellent work in that mode—this can feel like an impediment rather than an enabler.

Resistance to agile practices is rarely a matter of bad intent. Highly skilled engineers are often attracted to deep, focused work, and the constant context-switching of collaborative agile ceremonies can feel disruptive. The culture change required to succeed with agile is frequently underestimated at the organizational level.

The business case for collaboration is strong: diverse, cross-functional teams generally produce better outcomes than isolated individuals, and blended skill sets—combining technical, testing, and domain knowledge—are increasingly valuable. But mandating collaboration without addressing the underlying culture will produce compliance without engagement.

Mitigation: Invest in agile coaching, not just agile tooling. Create psychological safety so that team members can raise concerns without fear. Recognize and reward collaborative behavior at the team level, not just individual output.

4. Difficulty Scaling Agile to Large Projects

Agile methods originated with small, co-located teams working on well-scoped problems. Scaling those methods to large, geographically distributed programs introduces coordination challenges that are qualitatively different—not just quantitatively larger.

As project size grows, the number of inter-team communication channels grows exponentially. Dependencies between teams create bottlenecks, especially when testing cycles are distributed across time zones or organizational units. Integration with non-delivery functions—HR, legal, procurement, marketing—adds further complexity that agile ceremonies were not designed to handle out of the box.

Frameworks such as SAFe (Scaled Agile Framework), LeSS (Large-Scale Scrum), and Nexus exist specifically to address enterprise scaling, but they add overhead and require significant investment to implement effectively.

Mitigation: Choose a scaling framework that matches your organization’s actual complexity, and implement it incrementally. Establish clear dependency management practices between teams and invest in shared visibility tooling so that delays in one team’s work surface early rather than late.

5. Choosing Agile for the Wrong Type of Project

Agile delivers its greatest benefits when requirements are expected to evolve and when frequent user feedback is both possible and valuable. It is a poor fit for projects where neither condition holds—for example, projects with highly regulated, fixed requirements that cannot change during development, or purely exploratory research efforts where there is no user base available to provide meaningful feedback.

In complex matrix management environments, where multiple teams with partial project assignments work together, the coordination cost of agile ceremonies can outweigh their benefits. The problem intensifies when team members are only partially committed to a project, reducing accountability and slowing feedback loops.

The concept of VUCA—volatility, uncertainty, complexity, and ambiguity—describes the reality of most modern business environments and is frequently cited to argue that fixed requirements are rare in practice. While this is often true, it is not universally true, and project managers should evaluate each project on its own merits rather than defaulting to agile as the universal answer.

Mitigation: Use a structured project classification exercise before selecting a delivery methodology. Factors to consider include requirements stability, availability of end users for feedback, team size and distribution, and regulatory constraints. Hybrid approaches that blend agile and traditional methods are legitimate and sometimes optimal.

6. Getting Bogged Down by Collaboration Overhead

Agile depends on fast, accurate communication—between developers, testers, business analysts, and end users. When team members are distributed across locations or time zones, the friction involved in maintaining that communication can slow the entire delivery process to a crawl.

Many enterprise organizations still manage significant portions of their delivery process—including test case management and defect tracking—using spreadsheets, Word documents, or email threads. This creates fragility: a single tester going on leave without handing off their work, or an email chain that loses a key decision, can stall an entire sprint.

End-to-end visibility is especially critical in user acceptance testing (UAT), where business users must be actively engaged at specific points. Without automated notifications, clear ownership, and real-time status tracking, UAT becomes a bottleneck that is difficult to predict or manage.

Mitigation: Replace ad-hoc email-based coordination with structured collaboration tooling that provides automated notifications, audit trails, and real-time status visibility. Define clear handoff protocols between team members and ensure that every activity has a named owner, regardless of who is physically present on a given day.

Making Agile Work: Practical Takeaways

Agile is not inherently superior to other delivery methodologies—it is a tool, and like any tool it works best when applied to the right problem in the right context. Here are several principles that help teams get sustainable value from agile:

• Break work down deliberately. User story mapping, mind mapping, and story-splitting techniques help teams decompose large, complex requirements into smaller, independently shippable increments. Smaller increments mean faster feedback and lower risk.
• Maintain architectural discipline. Agile does not mean no design. Build in time for refactoring and architectural review, and document significant design decisions so that future team members understand the reasoning behind them.
• Automate feedback loops. Automate the communication of test results, build status, and change notifications. Reduce the reliance on manual email for anything that can be triggered and tracked by tooling.
• Be honest about project fit. If your project has genuinely fixed requirements, a tiny team, and no available end users, agile ceremonies will add overhead without benefit. It is not a failure to choose a different approach.
• Invest in culture, not just process. Agile transformations that focus exclusively on ceremonies and tooling while ignoring team culture rarely succeed. Psychological safety, shared ownership, and genuine transparency are prerequisites, not byproducts.

The Bottom Line

When the conditions are right—engaged stakeholders, evolving requirements, collaborative culture, and appropriate tooling—agile development genuinely accelerates delivery and produces software that better serves its users. The iterative feedback loop creates a working partnership between delivery teams and customers that is difficult to replicate with any other methodology.

But agile is not a universal solution. Teams that recognize its failure patterns early, and either address them proactively or choose a more suitable methodology, are far more likely to ship software that makes customers happy—regardless of which methodology they use to get there.

Originally sourced from infoq.com. Published September 2017.

There has been a fundamental shift in how organizations apply business process management (BPM) systems over the last several years. Platforms once deployed primarily to improve operational efficiency and lower costs are now increasingly viewed as full application development environments — places where developers and so-called “citizen developers” alike can build applications that automate business processes and decisions.

The Intersection of BPM and DevOps

When BPM systems are treated as application development platforms, their requirements become tightly coupled to modern software development methodologies — particularly DevOps. According to Phil Simpson, senior principal product marketing manager for JBoss Middleware at Red Hat, modern BPM systems must be compatible with the DevOps approach just as much as any other software development platform.

Charles King, principal analyst at Pund-IT, identified natural affinities between the two disciplines. DevOps aims to integrate the traditionally separate efforts of developers and operations professionals. BPM focuses on enhancing business processes and workflows to address specific customer needs or deliver broader organizational benefits. While these orientations differ — DevOps is often inward-focused, while BPM typically looks outward — both can create substantial customer value.

“Many of the core goals of DevOps, including dynamic planning, Lean and Agile techniques, continuous integration and automated testing, can all impact the timing and quality of processes that impact customers. Strong business rules are the tracks that projects run on and are essential to reaching the desired destination.”

— Charles King, Principal Analyst, Pund-IT

Business rules have always been central to BPM because they underpin the automation of business processes. Embedding those rules into DevOps workflows allows teams to deliver process improvements continuously rather than through large, infrequent releases.

Agile and DevOps as Accelerators for BPM

Jim Sinur, vice president and research fellow at Aragon Research, emphasised that combining business professionals with developers in iteration-based approaches has proven highly effective — provided team communication is strong. This mirrors the cross-functional collaboration that sits at the heart of both Agile and DevOps.

Ed Fox, vice president of network services at telecommunications provider MetTel in New York, described how his organisation puts these ideas into practice:

“We have Agile development teams in our labs that help build segments of code in sprints and put them into actual use with customers to generate a real market feedback loop that validates our product-service-solution direction and delivers value to ourselves and customers during the innovation process.”

— Ed Fox, VP of Network Services, MetTel

Fox argued that DevOps intersects with BPM at every point in his organisation. He described BPM as a “living process” that must dynamically evolve to support new capabilities on the fly — a characteristic he said is best delivered through a SaaS model.

BPM in Practice: Digital Transformation at ABN AMRO

Norbert Siegers, head of IT channel solutions at Netherlands-based ABN AMRO, offered a concrete enterprise example. His group deployed BPM as part of a multi-year digital transformation, moving from a monolithic architecture with waterfall-based development to a set of smaller, independently deployable services built on Agile and DevOps processes.

“We are pursuing a more agile model and using the latest techniques for continuous integration and continuous delivery.”

— Norbert Siegers, Head of IT Channel Solutions, ABN AMRO

This shift from monolith to services is a pattern that frequently accompanies BPM modernisation efforts, as organisations look to decouple process logic from legacy systems and deliver changes more frequently.

From BPM to Digital Process Automation

The evolution of BPM platforms is not just technical — it is also terminological. Rob Koplowitz, a Forrester Research analyst specialising in BPM, noted that the space has invested heavily in a continuous delivery mindset. BPM vendors, he argued, are particularly well-positioned to manage tasks across systems, people and data sources.

Significantly, Forrester has moved away from the term “BPM” in favour of digital process automation (DPA). The reasoning: as organisations adopt new best practices and pursue digital transformation, the category has shifted to directly support that broader mission rather than focus narrowly on process efficiency.

Key BPM Best Practices DevOps Teams Should Adopt

Drawing on the themes raised by practitioners and analysts in this space, the following practices represent the strongest points of convergence between BPM and DevOps:

• Codify business rules explicitly. Business rules should be treated as first-class artefacts, version-controlled and tested alongside application code.
• Apply continuous integration to process changes. Process definitions and workflow configurations should move through the same CI/CD pipelines as application code.
• Use cross-functional teams. Combining business analysts, process owners and developers in the same sprint reduces handoff delays and produces more accurate process models.
• Treat processes as living systems. Design BPM implementations to be modifiable without large-scale redeployments, enabling rapid iteration based on real user feedback.
• Decompose monolithic processes. Break large, end-to-end process definitions into smaller, independently deployable units — mirroring the microservices architecture that often accompanies DevOps adoption.
• Instrument and measure. Apply the same observability practices used in production software (metrics, tracing, alerting) to running business processes.

Conclusion

The convergence of BPM and DevOps is not a passing trend. Organisations that treat process automation as a software engineering discipline — applying Agile iterations, continuous delivery, and cross-functional collaboration — are finding they can respond to business change faster and with higher quality than those that keep process management separate from software delivery. Whether the label used is BPM or digital process automation, the underlying message is the same: process and code now travel the same pipeline.

From a leadership perspective, it is hard to argue against the goals of the current DevOps trend. DevOps seeks to integrate the work performed by a company’s application development team with the work performed by its systems operations team. The aim is to improve the quality and flow of that work through better communication, standardization, simplification, collaboration and automation.

DevOps builds on the Lean and Agile development practices that promote systemic thinking and rapid value creation. In many ways, the DevOps trend teaches IT professionals that they are critical links in a business supply chain, rather than a collection of technical experts narrowly focused on specific IT capabilities. These are laudable aims and goals.

However, it’s important to note that these aims and goals are not new — they were not created by the DevOps movement. IT organizations have long had the means, frameworks, tools and prescriptive guidance to achieve them. Our industry is blessed with a library of best practices and guidance on how to improve communication and facilitate standardization, simplification and collaboration. IT tools for decades have been capable of much higher levels of integration and automation.

Despite the fact that the means have long been there to solve our internal IT delivery problems, the upshot is that our actual experience has been one of silo-based cultures, tribal leadership, intentional process variation and tool strategies that place more value on “best of breed” software than on integrated IT management suites.

So, is there anything different now that has not been the case for the last 20 years? Why do we think that the DevOps trend will result in success where many other frameworks have come and gone without invoking meaningful change?

The answers to these questions do not lie in what is new or unique about the DevOps trend, but in the external pressures that are driving IT to change.

Like most self-help or personal improvement programs, we typically do not pay attention or get serious about real change until there is some external force that intrudes on old habits in a way that makes it difficult if not impossible to ignore. Until the doctor gives you a shock by predicting your early demise, regular visits to the gym remain an elusive goal. Today, a multitude of urgent external drivers are forcing IT shops to change old ways– and fast.

The cloud-enabled competitor at the door

Chief among those external drivers is cloud computing — IT’s big bad wolf at the door. The availability of cloud-based services has flattened the competitive IT landscape across all industries.

The ubiquitous availability of cloud-based technical services, coupled with inexpensive offshoring options, has rendered traditional delivery models obsolete. The lease vs. buy cloud services model has leveled the playing field and enabled business competition to rapidly come to market and disrupt traditional commercial models without having made years of investment in data centers, software factories or network infrastructure.

Thus, newly formed, built-on-the-web, app-enabled brands are now able to use cloud-based services and software-as-a-service business automation for core business processes in a matter of weeks, if not days. So enabled, they come to market with new and disruptive products in a fraction of the time it would take a traditional brick-and-mortar organization to respond.

Digitally disruptive competition does not apply to just external business competition. Another, observable trend is an increase in the use of so-called shadow IT investments, i.e., line of business purchases of IT services directly from the external market or in support of their own IT capability. Often, shadow IT is due to the perception by the business that internal IT’s services are of poor quality, slow speed or cost more than what they can buy or contract for themselves. Rather than shadow IT, I prefer to call this phenomenon what it truly is: loss of market share for internal IT. If it goes on long enough and an IT organization loses the funding it requires based on expected revenue, there is only one financial outcome possible.

Disruptive competition and the emergence of DevOps

Digitally disruptive competition is at the CIO’s door. And with this new degree of external pressure, organizations are forced to finally address long-standing cultural issues, political barriers and leadership gaps that have traditionally brought most IT improvement programs to their knees. So, now is the time to change — and change rapidly — in order to survive.

The banner many organizations are flying to combat digitally disruptive competition is DevOps.

To test this hypothesis, Pink Elephant recently conducted research into the motivations and critical success factors behind DevOps adoption. This research demonstrated that out of the 160 companies surveyed, more than two-thirds are focused on increasing “speed to value” by using the various principles and practices of the DevOps movement.

A summary of some of the key findings demonstrate the reality of the DevOps movement in our industry and why we believe that the IT push to improve the quality and flow of that work through better communication, standardization, simplification, collaboration and automation is different this time around.

The Speed vs. Quality Tension in Modern Software Delivery

Does your software testing process slow down your ability to release new applications quickly enough to meet market demand? For a significant proportion of organisations, the honest answer is yes. Despite widespread investment in agile methodologies and rapid development practices, traditional quality assurance (QA) approaches continue to act as a bottleneck — one that can quietly undo the speed gains teams work hard to achieve.

According to Jonathon Wright, who spoke at an ITWC webinar, 94% of senior managers report increased pressure to release applications more quickly. Drivers include rising customer expectations, intensifying competitive threats, and the relentless pace of technology change. The implication is clear: the way organisations have historically approached QA is no longer fit for purpose in a world that demands continuous delivery.

Why Traditional QA Creates Bottlenecks

In a conventional software development lifecycle, testing is treated as a phase that happens after development is complete. Teams write code, hand it over to QA, wait for results, fix defects, and then release. This sequential model made reasonable sense when release cycles were measured in quarters or years. It becomes a serious liability when business strategy demands weekly or even daily deployments.

Wright noted that most large organisations still take around 18 months to respond meaningfully to marketplace changes. In an era of what he described as “massive digital disruption,” that lag is no longer acceptable. The cost is not just speed — it is relevance.

The problem is compounded by user behaviour. Research cited in the webinar suggests that 79% of users will abandon a website or application if they cannot quickly find what they need. This means poor quality is not just a technical embarrassment — it has a direct and measurable impact on customer retention and revenue.

The Real Obstacle Is Culture, Not Technology

One of the more counterintuitive insights from Wright’s presentation is that the primary barrier to better QA is not tooling or infrastructure — it is organisational culture. “It’s about getting ready for the change from a people, a mindset and a process perspective,” he said. “The practices and behaviours have to change.”

This observation aligns with broader findings in the DevOps community. Technical improvements in automation, test frameworks, and CI/CD pipelines deliver far less value when teams still think of quality as someone else’s responsibility, or when QA remains siloed from development. Changing the tools without changing the culture tends to produce only incremental gains.

Digital Assurance: A Framework for Faster, Higher-Quality Delivery

Wright proposed a modern approach to QA that he labelled digital assurance. Rather than treating testing as a discrete phase, digital assurance embeds quality thinking throughout the entire software development and delivery lifecycle. It rests on two complementary concepts: shifting left and shifting right.

Shifting Left: Test Earlier

Shifting left means introducing testing activities much earlier in the development lifecycle — ideally at the requirements stage. Rather than waiting until code is written before thinking about test cases, teams write test cases at the same time requirements are defined. This approach offers several concrete benefits:

• Defects are caught earlier, when they are cheaper and faster to fix.
• Requirements ambiguities surface before they are baked into code.
• Developers gain a clearer, testable definition of “done” from the outset.
• QA engineers become active participants in design and planning, not just validators at the end of the pipeline.

Shifting left also supports practices like test-driven development (TDD) and behaviour-driven development (BDD), where automated tests guide the writing of production code rather than following it.

Shifting Right: Incorporate Real-World Feedback

Shifting right focuses on gathering feedback from actual users in production or near-production environments. Rather than relying solely on pre-release testing conducted in controlled conditions, organisations monitor real user behaviour, collect telemetry, and run experiments such as A/B tests or canary releases to validate assumptions about features.

Understanding why a customer wants a particular feature — not just what they asked for — can make a significant difference to the quality and relevance of the result. Shifting right closes the feedback loop between delivery teams and the end users they serve.

Continuous Testing: The Engine of Digital Assurance

Underpinning both shifts is the practice of continuous testing — automated test execution integrated directly into the CI/CD pipeline. Wright was direct on this point: “If you can’t do continuous testing, you’re going to get stuck.”

Continuous testing means that every code commit triggers a suite of automated tests — unit, integration, performance, security — and results are available within minutes rather than days. Teams get fast feedback, failed builds are caught before they propagate downstream, and the overall quality of the codebase is maintained without manual gating.

Implementing continuous testing effectively requires investment in:

• Test automation frameworks that cover a meaningful proportion of the codebase.
• Service virtualisation to simulate dependencies that are unavailable or expensive to use in testing environments.
• Environment management to ensure tests run in consistent, reproducible conditions.
• CI/CD tooling that integrates test execution into every stage of the pipeline.

Real-World Results

Wright cited ING as an example of a large-scale organisation that adopted continuous release and continuous quality practices. According to the figures presented at the webinar, the bank achieved a 97% improvement in time to market for new applications and software updates — a dramatic illustration of what is possible when QA is integrated into, rather than appended to, the delivery process.

Wright also pointed to CA Technologies itself as an example of the model in practice. The company moved from one major product release per year to shipping a new release every month after applying digital assurance principles internally.

“When you build this into your process with DevOps and you change the culture, the processes and the tools, you can get incredible results,” said Julie Gardiner, Director of Product Management at CA Technologies. “You’ll wonder why you weren’t doing it ages ago.”

Key Takeaways for Engineering Teams

• QA is a shared responsibility. Quality cannot be delegated to a separate team that operates at the end of the pipeline. Every member of a delivery team owns quality.
• Shift testing left. Write test cases when requirements are defined, not after code is complete.
• Shift feedback right. Use production monitoring and real user data to validate assumptions and guide improvements.
• Automate continuously. Integrate automated tests into every CI/CD pipeline stage. Manual-only testing cannot scale to the cadence modern delivery demands.
• Address culture first. Technology investments will underdeliver unless teams adopt a quality-first mindset and break down silos between development and QA.

The gap between the speed that agile promises and the speed that traditional QA allows is a solvable problem — but solving it requires more than purchasing new tools. It requires a deliberate change in how teams think about quality and where in the lifecycle they choose to invest in it.

Testing Is the #1 DevOps Bottleneck

Computing Magazine’s DevOps Review 2017 survey asked practitioners a straightforward question: “Where are the main holdups in the software production process?” The answer was unambiguous. According to the survey, 63 percent of self-identified DevOps practitioners named the QA and test phase as their biggest source of delay—a figure more than twice the next most-cited bottleneck, planning, at 29 percent.

The same survey revealed a related tension around quality. When asked what tends to lose out in DevOps environments, nearly 30 percent said quality was being sacrificed—even as over 40 percent listed quality improvement as their top focus for the coming year.

Why Quality Takes a Hit in DevOps Pipelines

The survey identified two root causes for quality being squeezed out of fast-moving delivery cycles:

1. Testing and QA are siloed from the rest of the organisation, operating as a gate at the end of the pipeline rather than as a continuous, shared responsibility.
2. Testing tooling has not kept pace with agile and DevOps practices—many tools were designed for release cadences of two or three times a year, not the continuous delivery model most teams now target.

The core problem is structural. Traditional QA sits at the tail end of the software development lifecycle (SDLC). Agile teams measure development in days and hours, not months, but testing processes have not been redesigned to match. The result: testing becomes the bottleneck that prevents truly continuous delivery.

What Continuous Testing Actually Means

Continuous testing means integrating automated testing throughout every stage of the SDLC—from requirements and design through coding, integration, staging, and deployment—rather than treating it as a discrete phase at the end. Every team owns quality: developers, QA engineers, operations, and product owners all participate.

The practical outcome is an integrated delivery pipeline that can carry a feature from design to production at speed without trading off quality. Three capability areas make this possible in practice.

1. In-Sprint API Testing

Modern applications are collections of microservices communicating over APIs. Testing only at the UI layer is fragile—UI tests break whenever the interface changes, and they provide limited coverage of the business logic living in the services underneath.

Shifting API testing left—into the sprint itself—addresses this, but it demands that test creation be fast and largely automated. Teams need tooling that can:

• Generate comprehensive test cases directly from an API definition (positive paths, negative paths, edge cases, and security checks such as SQL injection).
• Execute those tests automatically as part of a CI build.
• Support popular open-source frameworks so that functional API tests can be re-used as performance tests—and vice versa—without proprietary conversion overhead.

The goal is to make exhaustive API testing a routine part of every sprint rather than a special effort reserved for pre-release hardening.

2. Pipeline Orchestration for Continuous Delivery

Automation alone is not enough if each team still manages handoffs manually via spreadsheets, email, or informal coordination. As the number of applications and release streams grows, disorganised handoffs create their own bottlenecks—rework, failed releases, and slow feedback.

A Continuous Delivery pipeline manager provides toolchain-agnostic orchestration: it plans, executes, and optimises the sequence of stages from build through production promotion. Key capabilities include:

• Automating gate decisions—promoting a build to the next environment only when all required tests pass.
• Eliminating manual tracking with real-time pipeline visibility.
• Feeding test results back to development teams immediately, shortening the feedback loop.
• Supporting multiple application streams concurrently without added management overhead.

SaaS-based CD orchestration tools have lowered the barrier to adopting this approach, making enterprise-grade pipeline management accessible to smaller teams.

3. Open-Source Performance and Functional Testing Frameworks

Legacy testing platforms were architected for infrequent, large releases. They often carry high licensing costs, steep learning curves, and limited integration with modern CI/CD toolchains. As developers take on more testing responsibility within the coding phase, they need lightweight, open-source alternatives.

Widely adopted open-source testing tools relevant to DevOps pipelines include:

• JMeter – load and performance testing for APIs and web applications.
• Selenium – browser-based UI functional testing.
• Gatling – high-performance load testing with a code-first DSL.
• Locust – Python-based distributed load testing.
• Appium – mobile application testing.
• Taurus – an abstraction layer that simplifies running JMeter, Gatling, and other tools through a unified configuration format.
• Jenkins – CI/CD automation server with broad plugin support for triggering and reporting tests.

Migration tooling also exists to help teams convert existing functional tests built on legacy platforms into load tests, and to move test assets from proprietary formats to open-source equivalents.

What Continuous Testing Enables in Practice

Taken together, these three capability areas allow organisations to implement continuous testing end-to-end. In concrete terms, teams can:

• Automatically derive the minimum set of test cases needed for full code coverage from user stories in tools such as Atlassian JIRA, and adapt tests automatically as requirements change.
• Request and provision test data from within the requirements phase, reducing test cycle preparation time.
• Virtualise APIs and dependent services so that testing can proceed before all components are available, and so that API access is managed consistently across the development team.
• Institute automated API testing that generates positive, negative, edge-case, and security test scenarios without manual test scripting for each case.
• Feed test data directly and securely to virtual services, decoupling test execution from environment availability.
• Democratise testing with SaaS-based tools so developers can test early and test often without requiring deep QA expertise.
• Orchestrate the delivery pipeline to promote builds automatically when tests pass, enabling faster, lower-risk releases.

Conclusion

The data is clear: testing is where DevOps pipelines most commonly stall, and quality is what most often gets sacrificed when teams try to move faster without addressing that root cause. Continuous testing—shifting quality left into every sprint, orchestrating handoffs with intelligent pipeline tooling, and adopting open-source frameworks built for agile cadences—is the engineering answer to that problem.

It is not a silver bullet, and it requires genuine cultural change alongside tooling investment. But teams that integrate testing into the fabric of the SDLC, rather than bolting it on at the end, consistently close the gap between speed and quality that defines mature DevOps delivery.

Background: Why a Separate Technical Conference?

The Agile movement has firmly taken hold in software development. Standups, user stories, and sprints are now standard vocabulary across most engineering teams. Yet over roughly four years, the flagship Agile Alliance conference — which grew from around 900 to 1,500 attendees during that period — steadily reduced its technical content. The DevOps track continued to grow, but sessions on programming and testing drew smaller audiences. Fewer technical practitioners attended, fewer attended technical sessions, and organizers responded by scheduling fewer technical talks the following year — a self-reinforcing cycle.

Something had to change. In 2016, the Agile Alliance launched the Agile Alliance Technical Conference (AATC) as a dedicated venue for practitioners who wanted to go deep on engineering craft rather than process or management frameworks.

What Happened at AATC 2017?

The 2017 edition was held in Boston and drew approximately 300 attendees. The opening keynote was delivered by Ron Jeffries and Chet Hendrickson, two of the original signatories of the Agile Manifesto and long-time advocates of Extreme Programming (XP). They performed the bowling game kata — a form of deliberate practice in which a programmer (or pair) solves a well-defined problem repeatedly to build muscle memory for a specific skill.

In this case, the skill was test-driven development (TDD). Along the way, the two discussed code quality, sustainable pace, pair programming, and what “modern” Agile engineering looks like in practice. Because Jeffries and Hendrickson bring fresh nuance to the kata each time they perform it — despite having done it for over a decade — it still resonated as genuinely new material with the audience.

Other sessions at AATC 2017 covered:

• A keynote on Extreme Programming, celebrating its twentieth year as a methodology
• Fundamentals of test-driven development
• Practical refactoring techniques
• Writing and maintaining clean code

Notably absent from the three-day agenda: stories, standups, sprints, or discussions of trendy infrastructure tooling. The conference was deliberately focused on coding craft.

The Technical Renaissance: Are Agile Engineering Practices Going Mainstream?

At first glance, the fact that a technical conference needs to re-teach TDD and refactoring — practices that have existed for twenty-plus years — can look like a failure. It could be read as evidence that the industry is “post-Agile” in the worst sense: teams adopted the ceremonies while abandoning the hard technical disciplines.

But there is a more optimistic interpretation. Linda Cook, an Agile Alliance board member, explained the conference’s intent the evening before it opened: the AATC was designed to reach a different audience — practitioners who were not plugged into the conference circuit, who had not been exposed to XP or TDD through the usual channels, but who were hungry to learn and improve their craft.

Filling a hotel ballroom in Boston with developers eager to learn TDD and refactoring suggests something important: Agile technical practices may finally be reaching the mainstream, rather than remaining the province of a small community of early adopters.

The Diffusion of Innovation Lens

Sociologist Everett Rogers’ diffusion of innovation model describes how new ideas spread through a population across five groups:

• Innovators — create and experiment with new ideas
• Early Adopters — willing to try things with minimal evidence
• Early Majority — adopt after seeing others succeed
• Late Majority — adopt once a practice is well established, often under peer pressure
• Laggards — adopt only when the old approach is no longer viable, or not at all

Viewed through this model, the AATC audience looked like the early and late majority arriving for Agile engineering practices. These were not the innovators who invented TDD in the 1990s, nor the early adopters who championed XP in the 2000s. They were developers at mainstream organizations who had heard about these practices, sensed they were falling behind, and wanted structured help getting started.

Seeing 300 peers — all at different stages of the same journey — in one room is a powerful signal. It validates that the struggle is common, and that the practices are learnable. That is precisely the environment in which a late majority adopts a new approach.

Proto-Agile, Agile, and Post-Agile: Where Does the Industry Stand?

Each morning at AATC, a small group gathered for Lean Coffee — an open-agenda conversational format where participants propose topics, vote on priority, and then discuss in time-boxed rounds. One recurring theme was the state of technical practices: specifically, “Is Extreme Programming dead?”

Woody Zuill, a well-known practitioner and pioneer of Mob Programming, offered a useful three-stage framework at the table:

• Proto-Agile: “We know what we had before isn’t working, and we are discovering new things.”
• Agile: “We actually understand these modern methods and do them well.”
• Post-Agile: Continuing to innovate beyond established Agile practice — not abandoning it.

Zuill’s position was direct: XP cannot be dead, because he uses it every day. “Post-Agile” in the healthy sense means teams have internalized the core practices so deeply that they no longer need to label them — not that they have abandoned them in favor of stand-up theater.

What This Means for Engineering Teams

The signal from AATC 2017 is nuanced but important for anyone leading or working on a software engineering team:

• TDD, refactoring, and clean code are not niche or outdated — they are skills that a large segment of the industry is actively trying to acquire right now.
• The gap between process adoption and technical discipline is real. Many teams run sprints but do not practice continuous integration, TDD, or refactoring. Closing that gap is one of the highest-leverage investments an engineering organization can make.
• Deliberate practice works. Katas, pair programming exercises, and structured code review are how these skills transfer — not just by reading about them.
• “Post-Agile” is not an excuse to stop improving. If your team has abandoned technical practices in the name of pragmatism, the AATC data suggests you are likely in the early or late majority who have adopted Agile ceremonies without the engineering foundation that makes them effective.

Conclusion

The Agile Alliance Technical Conference 2017 offered a clear-eyed snapshot of where the industry stands: technical practices like TDD, refactoring, and clean code are no longer strange or radical ideas, but they are also not yet universal. They are in the process of diffusing from their innovative origins into the mainstream — and a growing number of developers are actively seeking to learn them.

For DevOps practitioners, this matters directly. The feedback loops that make continuous delivery reliable — fast automated tests, small refactored functions, clean interfaces — depend on exactly these skills. The infrastructure may be ready; the question is whether the engineering practices are keeping pace.

1. Align Culture with Organizational Structure

DevOps fundamentally redefines how teams collaborate and how they are structured. The sequential, handoff-driven waterfall model conflicts with the iterative, feedback-oriented mindset that agile and DevOps demand. People are not interchangeable parts in a pipeline—they need a culture that supports shared ownership and open communication.

To address this, organizations should proactively communicate the benefits of a DevOps model, reshape team structures to reduce handoffs, and create an environment where collaboration is the default—not the exception. When the organizational structure mirrors the culture, DevOps becomes a natural extension of Agile across the entire software delivery value stream.

2. Realign Policies and Processes for Collaboration

Existing organizational policies are often designed around traditional, siloed operating models. To achieve true DevOps agility, workflows, roles, responsibilities, and approval processes must be revisited and updated to support cross-functional collaboration between development, testing, infrastructure, and operations teams.

This goes beyond adding a weekly standup or a sprint board on top of existing waterfall processes. Genuine agility means delivering small, production-ready increments frequently, managing change continuously, and integrating all processes—from code commit to deployment—into a single, coherent flow. Development and operations must work together, supported by automation that removes manual bottlenecks.

3. Break Down Silos Between Teams

Silos between development, testing, and operations are one of the biggest barriers to DevOps agility. When teams operate in isolation, slow handoffs, miscommunication, and blame cycles undermine delivery speed and quality.

Eliminating silos requires weaving together tools, teams, and processes so that continuous development, testing, and deployment can happen fluidly. Effective cross-functional collaboration—supported by the right communication and collaboration platforms—ensures there are no walls between agile teams. When teams share context and accountability, cycle times shrink and quality improves.

4. Build an Integrated, End-to-End Toolchain

A well-integrated toolchain is the technical backbone of a DevOps model. Rather than a collection of disconnected tools, each team or function should use, the toolchain must cover the entire software delivery lifecycle and be connected so data and events flow automatically between stages.

A complete DevOps toolchain typically covers:

• Source control and configuration management – e.g., Git, version-controlled infrastructure
• Continuous integration and continuous delivery (CI/CD) – automated build, test, and deployment pipelines
• Test automation – unit, integration, performance, and security testing integrated into the pipeline
• Issue tracking and change management – linking work items to commits and deployments
• Collaboration and communication – tools that keep distributed teams aligned
• Deployment and infrastructure automation – enabling consistent, repeatable releases

When these components are integrated rather than bolted together ad hoc, automated deployments become reliable and the entire pipeline becomes observable and auditable.

5. Balance Speed with Stability

Agile and DevOps both prize rapid delivery, but speed without stability creates technical debt, outages, and eroded customer trust. The goal is to maximize the rate of safe change—not simply to deploy as fast as possible regardless of risk.

Achieving this balance means choosing the right testing strategies, deployment patterns (such as feature flags, canary releases, or blue-green deployments), and rollback mechanisms so that fast delivery does not come at the cost of reliability. Organizations that get this right can respond to changing business conditions and seize opportunities quickly while maintaining the quality standards their users expect.

6. Measure Continuously and Act on Feedback

Feedback loops are central to any agile or DevOps practice. Without measurement, there is no reliable way to know whether the model is working or where to improve. For every deployment or sprint, teams should capture baseline metrics across at least three dimensions:

• Quality – defect rates, escaped bugs, mean time to detect (MTTD)
• Speed – lead time, deployment frequency, mean time to recover (MTTR)
• Cost – infrastructure spend, rework effort, manual toil

Over time, trends in these metrics reveal whether agility is improving and highlight areas that need attention. Equally important—though harder to quantify—are cultural metrics: team cohesion, psychological safety, time to resolve interpersonal or cross-team conflicts, and the degree of shared ownership and pride in the product. These cultural indicators are leading signals for the long-term health and effectiveness of a DevOps organization.

Bringing It All Together

Achieving lasting DevOps agility is not a destination—it is an ongoing commitment. Culture, organizational structure, policies, toolchains, and measurement practices must all be aligned and evolving together. Mature DevOps organizations are characterized by cross-functional teams that collaborate naturally, pipelines that deliver reliably at speed, and a data-driven culture of continuous improvement. When all these elements work in concert, DevOps becomes a genuine competitive advantage.