Originally sourced from DevOps.com (December 2019). The predictions below reflect the state of the data engineering landscape heading into 2020 and are presented as industry opinion, not verified outcomes.
2019 saw more data organizations running advanced analytics, AI, and machine learning workloads than ever before. Heading into 2020, the expectation was a spike in both the number of technologies and the data teams required to support those workloads internally. AI and analytics teams were predicted to merge into unified data organizations, tackling challenges such as hybrid cloud strategy, Kubernetes-based analytics, and the ongoing evolution of the Hadoop ecosystem.
1. The Rise of the Hybrid Cloud
After years of discussion, hybrid cloud was expected to move from concept to practical reality in 2020. Large enterprises were already refusing to expand on-premises Hadoop deployments, choosing instead to invest in public cloud capacity. However, most were unwilling to migrate core enterprise data off-premises entirely.
The predicted model: data stays on-prem, compute bursts to the cloud. This approach targets peak demand and unpredictable workloads, where elastic cloud compute is more cost-effective than maintaining excess on-premises capacity. Technologies enabling seamless data-local-compute-remote architectures were expected to lead the hybrid cloud wave.
2. One Machine Learning Framework to Rule Them All
By late 2019, operationalizing ML model training had become a priority across organizations of all sizes. Multiple frameworks competed for dominance, but no clear winner had emerged. The analogy drawn was to earlier data tooling consolidations: Apache Spark became the standard for data transformation, and Presto was emerging as the leading choice for interactive querying.
The prediction for 2020 was that the model training space would see similar consolidation, with PyTorch and TensorFlow identified as the leading contenders for the dominant framework position.
3. Kubernetifying the Analytics Stack
Containers and Kubernetes had proven highly effective for stateless workloads — web servers, microservices, self-contained databases — but adoption for advanced analytics and AI workloads lagged behind. The prediction was that 2020 would mark the shift toward AI and analytics workloads becoming mainstream on Kubernetes.
The core technical challenge: solving data sharing and elasticity. Moving data from remote silos into Kubernetes clusters improves data locality, which is critical for performance-sensitive analytics jobs. This required new patterns for stateful workload management that the community was actively developing.
4. HDFS Is Dead — But Hadoop Compute Lives On
The “Hadoop is dead” narrative was common heading into 2020, but the reality was more nuanced. The prediction separated two distinct components of the Hadoop ecosystem:
- HDFS (Hadoop Distributed File System): Predicted to decline due to operational complexity, high cost, and a fundamental inability to support elastic compute scaling. Data stored in HDFS would migrate to cloud object storage (such as S3 or GCS) or on-premises object storage alternatives.
- Hadoop compute frameworks (Spark, Presto): Predicted to remain strong. These tools had already been adopted well beyond the Hadoop ecosystem and were seen as foundational for large-scale data processing regardless of the underlying storage layer.
The key insight: compute and storage decoupling was accelerating, making storage-layer-agnostic compute frameworks more valuable, not less.
5. AI and Analytics Teams Merge Into a Unified Data Organization
The organizational structure of data teams was also expected to evolve. Hadoop platform teams had already transitioned into analytics and data engineering teams. The next step was absorbing AI and ML functions into the same organizational unit, reflecting how deeply intertwined structured data analytics and machine learning had become.
The convergence of statistical modeling and computer science into modern ML meant that data, analytics, and AI teams were often working on the same datasets with overlapping goals. The predicted outcome: organizations would build dedicated, unified data stack teams responsible for storage, compute, and modeling across on-prem and cloud environments.
6. The Talent Gap Will Constrain Data Technology Adoption
A persistent challenge identified for 2020 was the shortage of engineers who could bridge data engineering, cloud architecture, and AI/ML workloads. Building production data stacks that incorporate modern cloud, Kubernetes, and ML tooling requires a rare combination of skills.
The concern: as executive-level interest in data-driven strategies grew, the pipeline of engineers capable of architecting those systems was not keeping pace. Companies making ambitious “data-everywhere” commitments were expected to fall short without a parallel investment in developing or recruiting the necessary talent.
7. China’s Cloud Adoption Leapfrog
While U.S. enterprises had been migrating to public cloud infrastructure for several years, Chinese enterprises had largely invested in on-premises infrastructure for data-intensive workloads. The prediction for 2020 was that this would change dramatically — with Chinese enterprises leapfrogging incremental cloud adoption to deploy advanced workloads directly in the public cloud at significant scale.
Public cloud providers operating in China were expected to see growth rates potentially outpacing those of established Western cloud giants, driven by new use cases, burst compute for peak loads, and migration of existing on-prem workloads.
Key Takeaways for Data Engineering Teams
- Plan for hybrid architectures: Separating data residency from compute elasticity is a practical near-term strategy, not a distant goal.
- Watch the ML framework landscape: Framework consolidation simplifies toolchain decisions; betting on widely-adopted frameworks reduces long-term migration risk.
- Invest in Kubernetes expertise for analytics: Stateful analytics workloads on Kubernetes require different patterns than stateless services — build that knowledge proactively.
- Decouple storage from compute: Moving away from HDFS toward object storage unlocks the elasticity needed for modern cloud-native analytics.
- Build unified data teams: Siloed analytics and AI functions create duplication; unified teams sharing infrastructure and data assets are more efficient.
- Address the talent gap directly: Upskilling existing engineers in cloud-native and ML tooling is often faster than competing for a limited pool of specialists.
The trends outlined here represent the data engineering inflection points anticipated heading into 2020 — a year when advanced analytics and AI were expected to move from experimental to operational across a much wider range of organizations.