This article is based on analysis originally published at forbes.com (March 2018). Technical claims reflect the state of the industry at that time.
The Current State of the Enterprise Data Center
Ask any enterprise technologist or tech investor to name the forces reshaping the data center, and you will likely hear the same core list: cloud computing, containers and virtualization, microservices, machine learning and data science, flash storage, edge computing, NVMe, and GPUs. These technologies collectively underpin today’s digital transformation initiatives.
The more difficult question is: what comes next? Which emerging paradigm shifts are poised to become the next big thing, and how will they reshape the hardware and software markets? One strong candidate is a discipline called DataOps.
DevOps: A Quick Definition
To understand DataOps, it helps to start with DevOps. First described around 2008, DevOps is an IT practice designed to maximize automation and repeatability in the process of building and deploying software applications. Its core thesis: when software developers and operations professionals collaborate tightly, delivery becomes faster and less expensive.
The primary goals of DevOps include:
- Agility — the ability to respond quickly to changing requirements
- Faster time to market — shorter cycles from code to production
- Continuous delivery — automated pipelines that ship working software reliably
Vendors like VMware, Docker, Puppet, and Chef all built significant businesses by riding the DevOps wave.
The DevOps Disillusionment
Despite its early momentum, DevOps has not fully delivered on its original promise—particularly for data-heavy workloads. A 2017 study of 2,197 IT executives found that only 17% cited DevOps as having had a strategic impact on their organizations. By comparison, 41% cited big data and 39% cited public cloud infrastructure-as-a-service as strategically impactful.
One widely cited explanation: DevOps methodologies were designed with lightweight, stateless applications in mind. They were not architected to handle the complexities of data-intensive workloads—a gap that has become increasingly visible as enterprises lean harder on machine learning, data science, and real-time analytics.
The Rise of Data as a Strategic Asset
If there is one trend affecting virtually every enterprise, it is the growing emphasis on data as a driver of competitive advantage. Whether the goal is to improve customer experience, streamline operations, or unlock new revenue streams, data is central to modern business strategy.
This reality puts pressure on IT organizations to think differently about how data-intensive applications are developed, deployed, secured, and maintained. The conventional DevOps toolchain was simply not designed for this problem space.
Why Data-Intensive Applications Are Different
When you shift focus from lightweight web applications to data-intensive workloads—machine learning models, data science pipelines, streaming analytics—a new set of requirements emerges:
- Training data at scale — ML and data science applications require access to large volumes of training data throughout the development lifecycle, not just at deployment time.
- Data locality — Performance-sensitive workloads must run close to where data is stored, meaning the operations team must think about data placement, not just compute placement.
- Access control and governance — When multiple teams share data assets, strict policies must govern who can access what data, when, and under what conditions. This is a security and compliance requirement, not just an engineering preference.
- Mixed data types — Modern data environments mix legacy structured data (relational databases) with unstructured data (documents, images, logs) and streaming data, each requiring different handling.
These requirements collectively make the case for a new operational discipline: DataOps.
What Is DataOps?
DataOps is an agile methodology for developing and deploying data-intensive applications. It borrows the spirit of DevOps—tight collaboration, automation, repeatability—but extends it to accommodate the unique demands of data-driven workloads.
Where DevOps primarily connects software developers with operations engineers, DataOps requires collaboration across a broader set of stakeholders:
- Software developers and architects
- Data scientists and data engineers
- Security and governance professionals
- Operations teams
The goals of DataOps include repeatability, productivity, agility, and self-service access to data—ultimately enabling continuous deployment of data science models and analytics pipelines in a governed, scalable way.
The DataOps Technology Stack
As DataOps practices mature inside large organizations, a recognizable technology stack is beginning to take shape. It builds on some familiar DevOps components—containers, virtualization—while adding capabilities that are specific to data-intensive environments.
Tools Layer
At the tools layer, a DataOps practice depends on a data science platform that supports the languages and frameworks data scientists actually use: Python, R, Jupyter notebooks, and version control systems like Git. Critically, the platform must also enforce data access and governance policies at every stage of the pipeline—not just at the perimeter. Data-as-a-service and self-service data marketplace tools are increasingly central to this layer.
Platform Layer
At the platform layer, DataOps requires a unified data fabric—infrastructure capable of managing and providing consistent access to massive volumes of data across structured, unstructured, and streaming formats. Key platform requirements include:
- Support for multiple compute engines, including containerized workloads
- Cross-location data management (handling data distributed across physical sites or cloud regions)
- Optimization for data locality to minimize latency in data-intensive processing
DataOps vs. DevOps: A Side-by-Side Comparison
| Dimension | DevOps | DataOps |
|---|---|---|
| Primary focus | Application development & deployment | Data-intensive application development & deployment |
| Key collaborators | Developers, operations engineers | Developers, data scientists, data engineers, security, ops |
| Core artifact | Application code | Data pipelines, ML models, analytics workloads |
| Key concern | Compute placement, uptime | Data locality, governance, model lifecycle |
| Tooling examples | Docker, Kubernetes, Jenkins, Puppet | Data science platforms, data fabric, self-service catalogs |
What DataOps Means for the Market
If DataOps follows the adoption trajectory of DevOps, it will reshape the vendor landscape over the next decade. The companies that will be well positioned are those that make it significantly easier to develop, deploy, govern, and iterate on data-intensive applications at enterprise scale.
Key market opportunity areas include:
- Unified data platforms that can handle structured, unstructured, and streaming data under a single management plane
- Data governance and access control tooling that integrates natively into data science workflows rather than sitting outside them
- Self-service data marketplaces that allow teams to find, access, and use data assets without creating bottlenecks on central data engineering teams
- ML platform and MLOps tooling that operationalizes model training, deployment, and monitoring in a repeatable, auditable way
Key Takeaways
- DevOps transformed software delivery but was not designed for data-intensive workloads.
- The explosion of machine learning and data science inside enterprises has exposed those limits.
- DataOps extends DevOps principles—automation, collaboration, repeatability—to cover the full lifecycle of data-intensive applications.
- A DataOps stack requires both tooling (data science platforms, governance tools) and platform infrastructure (unified data fabric, data locality optimization).
- Organizations that invest in DataOps practices now are likely to be better positioned to deploy and scale AI and ML initiatives.