This article is based on reporting originally published on SearchITOperations (TechTarget) in September 2019.
Google Cloud Dataproc Integrates Apache Spark with Kubernetes
Google Cloud Dataproc released an alpha preview of Apache Spark running on Kubernetes in September 2019, positioning it as the first managed cloud service from a major provider to offer this integration. While other vendors — including Databricks (founded by the creators of Apache Spark) and D2iQ (formerly Mesosphere) — support Spark on Kubernetes, Dataproc’s move signals that the major cloud platforms are beginning to treat Kubernetes as a first-class resource orchestrator for data-intensive workloads, not just stateless microservices.
Why Run Apache Spark on Kubernetes Instead of Hadoop YARN?
Historically, Apache Spark has run on top of Hadoop YARN, the resource manager and job scheduler that ships with the Hadoop ecosystem. Kubernetes, by contrast, was designed for containerized, often stateless applications. So why shift?
There are several operational benefits to using Kubernetes as the resource orchestration layer under Spark:
- Portability: Containers run consistently across on-premises, hybrid, and multi-cloud environments, making it easier to move Spark workloads without re-platforming.
- Configuration consistency: Container images capture dependencies and runtime configuration, reducing environment drift between development, staging, and production.
- Automated provisioning and scaling: Kubernetes-native horizontal scaling and Helm charts enable dynamic resource allocation without manual cluster resizing.
- Resource efficiency: Containers typically pack more workloads onto a given host than virtual machines, reducing infrastructure costs for bursty analytical jobs.
Doug Henschen, an analyst at Constellation Research, framed the business case clearly: “Analytical workloads, in particular, benefit from the ability to add rapidly scalable cloud capacity for spiky peak workloads, whereas companies might want to run routine, predictable workloads in a virtual private cloud.”
Where Hadoop’s Container Story Falls Short
Apache Spark added a native Kubernetes scheduler in version 2.3, and Apache Hadoop introduced native container support in Hadoop 3.0.3 — both released in May 2018. However, Hadoop’s container integration remains tightly coupled to HDFS and is widely regarded as complex to operate in practice.
James Malone, product manager for Cloud Dataproc at Google, offered a candid assessment: “People have gotten Docker containers running on Hadoop clusters using YARN, but Hadoop 3’s container support is probably about four years too late. It also doesn’t really solve the problems customers are trying to solve — customers don’t care about managing Hive or Pig, and want to use Kubernetes in hybrid clouds.”
Google’s strategy with Dataproc is therefore to bypass YARN as the orchestration layer and route Spark jobs directly through Kubernetes, using GKE (Google Kubernetes Engine) as the underlying cluster infrastructure.
What the Alpha Integration Actually Covers
At alpha release, the Dataproc Spark-on-Kubernetes integration was split across several parallel workstreams, according to Malone:
- Dataproc cluster updates: Enabling Cloud Dataproc itself to provision and manage Kubernetes clusters rather than traditional YARN-based Hadoop clusters.
- Upstream Spark Kubernetes operator contributions: The Spark Kubernetes operator — which defines how Spark applications are submitted and managed as Kubernetes custom resources — was still in the experimental stage within Spark Core at the time of release.
- Performance add-ons: Features such as external shuffle service support, which enables dynamic resource allocation by allowing shuffle data to persist beyond the lifetime of individual executor pods, were not yet fully integrated.
Google also signalled plans to extend Dataproc’s Kubernetes support beyond Spark to other Apache data analytics frameworks, including Flink (stream processing), Druid (low-latency queries), and Presto (distributed SQL).
Known Technical Friction Points
Python vs. Scala Executor Experience
Michael Bishop, co-founder and board member at Alpha Vertex — a fintech startup running machine learning workloads on a multi-cloud Kubernetes infrastructure — highlighted one practical rough edge: the Python experience in Spark on Kubernetes has historically lagged behind Scala.
“The Python experience of Spark in Kubernetes has always lagged the Scala experience, mostly because deploying a compiled artifact in Scala is just easier logistically than pulling in dependencies for Python jobs,” Bishop said. “This is getting better and better, though.”
The core issue is that Scala Spark jobs compile to a JAR file that the executor can load predictably, whereas PySpark jobs require coordinating Python environments and dependency resolution across distributed executor pods — a challenge that container image layering partially addresses but does not eliminate.
Scheduler Impedance Between Spark and Kubernetes
A deeper architectural tension exists between how Spark and Kubernetes each think about scheduling. Spark’s scheduler is data-locality-aware: it tries to schedule tasks on the nodes where the data they need already resides, minimising network transfer. Kubernetes, by contrast, is primarily concerned with pod resource requirements and node capacity — not data locality.
Bishop described this as a fundamental impedance mismatch: “Spark is intimately aware of ‘where’ is for [nodes], while Kubernetes doesn’t really care beyond knowing a pod needs a particular volume mounted.”
This gap matters most when Spark jobs depend on HDFS or other distributed storage that benefits from data-local reads. In cloud-native deployments using object storage (such as Google Cloud Storage or Amazon S3), data locality is less critical because the storage layer is already remote — which is one reason the Kubernetes model fits cloud-native Spark deployments better than it fits on-premises Hadoop clusters.
What Teams Running Spark on Kubernetes Need Today
As of the alpha release, teams wanting to run production Spark workloads on Kubernetes outside of a fully managed service needed to assemble several components independently:
- The upstream Spark Kubernetes scheduler (built into Spark since 2.3, but still maturing).
- A supported Spark distribution, such as that provided by Databricks.
- A Kubernetes cluster, either cloud-managed (GKE, EKS, AKS) or self-hosted.
- For hybrid cloud portability, a distributed storage layer capable of running across clusters — vendors such as Robin Systems and Portworx were among those offering solutions at the time.
Assembling these pieces is possible, but it requires significant operational expertise and lacks the managed-service conveniences — automated upgrades, integrated monitoring, IAM integration — that a platform like Dataproc is designed to provide once the integration matures.
Key Takeaways for DevOps and Platform Engineering Teams
- Kubernetes is becoming a viable resource orchestration layer for Apache Spark, particularly in cloud-native and hybrid cloud environments where object storage replaces HDFS.
- The move from YARN to Kubernetes reduces Hadoop operational overhead but introduces new challenges around scheduler data-locality assumptions and Python dependency management.
- Google Cloud Dataproc’s alpha integration in 2019 represented a significant signal of direction, even if the implementation was incomplete at launch.
- Teams evaluating this architecture should assess whether their Spark jobs are data-locality-sensitive and whether they require external shuffle service support before committing to a Kubernetes-based execution layer.
- Upstream Spark and Kubernetes project maturity, not just managed service features, determines when this stack becomes production-ready.