DataOps (data operations) is an emerging discipline that unites DevOps teams with data engineers and data scientists to provide the tools, processes, and organizational structures that data-driven enterprises need. Originally attributed to agile and lean manufacturing influences, DataOps is increasingly recognized as a necessary evolution as organizations inject machine learning and advanced analytics into their products and services.
“You’ve got the modern trend for development of DevOps, but more and more people are injecting some sort of data science capability into development, into systems, so you need someone on the DevOps team who has a data frame of mind.”
Ted Dunning, Chief Applications Architect at MapR Technologies, and co-author of Machine Learning Logistics: Model Management in the Real World
What is DataOps?
DataOps is a collaborative approach to data management and analytics delivery. It applies the cultural and technical practices of DevOps — continuous integration, automation, shared ownership, and rapid iteration — to the data pipeline lifecycle. The goal is to reduce the time it takes to move from raw data to actionable analytic insight, while improving quality and reliability along the way.
Where traditional data workflows have kept data scientists, data engineers, and operations staff in separate silos, DataOps deliberately breaks down those barriers. Cross-functional teams share responsibility for the entire data lifecycle: ingestion, transformation, modeling, deployment, and monitoring.
DataOps Principles
Like DevOps itself, DataOps draws heavily from agile methodology and lean manufacturing thinking. The following principles characterize the DataOps approach:
- Continuous delivery of analytic insights: The primary goal is satisfying the customer with a steady stream of reliable, timely insights — not just periodic batch reports.
- Measurement by outcomes: DataOps teams value analytics that work. Performance is measured by the quality of the insights delivered, not by process compliance or output volume.
- Embrace change: Requirements evolve. DataOps teams continuously monitor and respond to changing customer needs rather than locking down specs upfront.
- Self-organizing teams: DataOps teams organize around shared goals and reduce reliance on heroism — the unsustainable practice of a single expert holding critical knowledge or carrying excessive workload.
- End-to-end orchestration: Data, tools, code, and environments are managed holistically from ingestion to insight. Reproducibility is non-negotiable.
- Analytic pipelines as lean production lines: Just as lean manufacturing eliminates waste and maximizes flow, DataOps applies the same thinking to data pipelines — reducing bottlenecks, handoff friction, and rework.
Where DataOps Fits in the Modern Enterprise
Enterprises today are embedding machine learning into a wide range of products and services. DataOps is particularly well suited to supporting the end-to-end requirements of machine learning workflows — especially the often-neglected handover between model development and production deployment.
“This style makes it more feasible for data scientists to have the support of software engineering to provide what is needed when models are handed over to operations during deployment.”
Ted Dunning and Ellen Friedman, Machine Learning Logistics
However, DataOps is not limited to machine learning. As Dunning and Friedman note, the approach is valuable for any data-oriented work, making it easier to take advantage of capabilities such as building a global data fabric — a unified, governed data architecture that spans on-premises and cloud environments.
DataOps also maps well onto microservices architectures, where small, independently deployable services generate, consume, and transform data continuously. The same principles of automation, modularity, and shared observability that make microservices manageable also underpin DataOps pipelines.
DataOps in Practice: Breaking Down Silos
Traditionally siloed roles — the data scientist working in isolation from the engineering team, or the DBA disconnected from production operations — are too rigid and slow for organizations undergoing digital transformation at scale. DataOps addresses this directly by restructuring how people work together.
The DevOps movement already demonstrated that bridging software development and operations shortens delivery cycles and aligns engineering work with business objectives. DataOps extends this model by adding data science and data engineering to the cross-functional mix. The result is a team where operations professionals, software engineers, data engineers, and data scientists share the same goals, the same feedback loops, and ideally the same physical or virtual workspace.
Dunning is direct about what this means structurally:
“The most important thing to do here is to not stick with the more traditional Ivory Tower organization where data scientists live apart from dev teams. The most important step you can take is to actually embed data scientists in a DevOps team. When they live in the same room, eat the same meals, hear the same complaints, they will naturally gain alignment.”
Ted Dunning
Embedding data scientists in DevOps teams also has a natural knowledge-transfer effect. Over time, team members absorb data-oriented thinking and some engineering team members grow into data engineer roles. The data scientist can then rotate to another team, carrying those capabilities forward across the organization.
“Typically, there’s a data scientist embedded in the team for a time. Their capabilities and sensibilities begin to rub off. Someone on the team then takes on the role of data engineer and kind of a low-budget data scientist. The actual data scientist embedded in the team then moves along. It’s a fluid situation.”
Ted Dunning
How to Build a DataOps Team
Building a DataOps team does not necessarily require a new wave of hiring. Many organizations already have the nucleus of a DataOps team within their existing DevOps groups. The practical starting point is identifying projects that require data-intensive development and locating — or developing — someone with appropriate data skills, whether that’s a full data scientist or a data engineer.
“When you’re covering these different skills, and putting them together toward this common goal, that doesn’t necessarily mean you’re having to hire a bunch of people to fill these roles. Often you have these people with the key skills. It just requires a realignment to understand what the key roles are.”
Ellen Friedman, Principal Technologist at MapR
Roles within a DataOps team commonly include:
- Data engineer: Builds and maintains data pipelines, storage systems, and transformation logic.
- Data scientist: Develops and validates analytical models; ideally embedded within the team rather than consulted remotely.
- Software engineer: Contributes application development expertise; may also cover data engineering responsibilities.
- Operations engineer: Ensures reliability, latency, and uptime of data services in production.
- Product manager / analyst: Translates business objectives into data-driven goals the team can act on.
In practice, roles overlap. A software engineer with strong data skills may also serve as a data engineer. Data scientists frequently handle their own data engineering. The one overlap that is rare, according to Dunning and Friedman, is between data science and operations — these two disciplines tend to require sufficiently different expertise that dual coverage is uncommon.
Aligning DataOps Teams Around a Common Goal
The most important structural ingredient in a functional DataOps team is a shared goal. Without it, the natural tensions between different disciplines — operations engineers optimizing for reliability and response time, data scientists optimizing for model accuracy — can pull a team apart.
“Once there’s a common goal, solving a problem, then the team organizes itself very often toward solving that problem… If they’re trying to solve the same problem and they’re willing to compromise on how it’s solved, I think it’s a pretty easy social structure to build up.”
Ted Dunning
Practically, this means DataOps teams should be organized around the data-driven needs of a specific service or business outcome — not around technology stacks or job titles. When everyone on the team is accountable to the same metric, the organizational friction between disciplines tends to resolve itself through natural collaboration.
Key Takeaways
- DataOps extends DevOps principles — agility, automation, cross-functional collaboration — to the full data and analytics lifecycle.
- It is relevant for any data-intensive work, not just machine learning.
- The most important organizational change is embedding data scientists within DevOps teams, not isolating them in separate departments.
- DataOps teams do not require large new headcounts; realigning existing talent around shared data goals is often sufficient to start.
- Common goals, shared environments, and willingness to compromise across disciplines are the social foundations of an effective DataOps team.
Quotes and insights in this article are drawn from Ted Dunning (Chief Applications Architect, MapR Technologies) and Ellen Friedman (Principal Technologist, MapR), co-authors of Machine Learning Logistics: Model Management in the Real World, as originally reported by cio.com.au.