Originally reported by Steef-Jan Wiggers on InfoQ. Original Google announcement published on the Google Cloud Blog, August 2020.
What Is Google Cloud Dataflow?
Google Cloud Dataflow is a fully managed service for executing Apache Beam pipelines on Google Cloud Platform (GCP). It handles provisioning, scaling, and operational management of distributed data processing jobs, supporting both batch and streaming workloads. Developers write pipelines using the Apache Beam SDKs — available in Java, Python, and Go — and Dataflow executes them at scale.
What Is Runner V2?
In mid-2020, Google announced Runner V2, a redesigned version of the Dataflow runner for Apache Beam. The core motivation was to move toward a more services-based, portable architecture that could support all of Dataflow’s language SDKs consistently — rather than maintaining separate, language-specific execution paths that diverged in capability over time.
Runner V2 is built on top of Apache Beam’s portability framework, an initiative within the Apache Beam project to enable pipelines written in any supported SDK to run on any compliant runner without SDK-specific runner code. Google packaged this framework together with two existing Dataflow services:
- Dataflow Shuffle — for optimizing batch job execution
- Streaming Engine — for offloading streaming job state management from worker VMs
By combining these components under the portability framework, Google can now deliver a consistent feature set across all language SDKs, share bug fixes, and roll out performance improvements uniformly.
Key Features of Runner V2
Multi-Language Pipeline Support
Perhaps the most significant capability introduced in Runner V2 is multi-language pipeline support. Prior to this release, a Beam pipeline written in Python could not natively call a transform or I/O connector written in Java, and vice versa. Runner V2 changes this by using the portability framework’s cross-language framework, which allows transforms from one SDK to be embedded in a pipeline written in another.
In practice, this means engineering teams can build reusable pipeline components in their preferred language — say, a high-performance Java I/O connector — and expose those components to teams writing pipelines in Python or Go. The result is a single, high-performance, distributed processing pipeline that spans language boundaries without sacrificing performance or correctness.
Increased SDK Parity
Historically, Beam’s Java SDK was the most feature-complete. Runner V2 significantly narrows the gap between SDKs by delivering:
- State and timer support in Python — enabling stateful processing patterns (e.g., sessionization, aggregation with custom windowing) that were previously only reliable in Java
- More I/O connectors for Python via the cross-language framework, including Kafka I/O — allowing Python pipelines to read from and write to Apache Kafka using the mature Java Kafka I/O implementation
Portable, C++-Based Worker Architecture
Runner V2 introduces a more efficient and portable worker architecture, with critical components rewritten in C++. The worker Virtual Machines (VMs) run the complete pipeline and provide access to the various language SDKs via the portability framework’s gRPC-based worker API. This design separates the runner logic from the SDK-specific user code, making it possible to support new languages without rewriting runner internals.
Custom Container Support
Runner V2 adds support for custom containers. Teams can supply their own Docker container images for the Beam SDK worker environment. This is particularly valuable when pipelines depend on native libraries, non-standard Python packages, or specific OS-level dependencies that cannot be satisfied through standard Dataflow worker images alone.
Increased Throughput via SplittableDoFns
SplittableDoFns (SDF) is an Apache Beam API that allows a single DoFn (a fundamental processing unit in Beam) to be split and processed in parallel across multiple workers. Runner V2 improves the execution of SplittableDoFns, enabling better parallelism for I/O-bound operations such as reading from large files, databases, or message queues. This translates directly to higher throughput for pipelines that use SDF-based I/O sources.
Architecture Overview
The high-level architecture of Runner V2 centers on the worker VM. Each worker VM:
- Runs the portable Dataflow runner (C++-based harness)
- Communicates with language-specific SDK worker containers via gRPC
- Offloads shuffle state (for batch) to Dataflow Shuffle and streaming state to Streaming Engine
- Executes user-defined transforms inside isolated SDK environments, including custom containers
This separation of the runner harness from SDK worker code is the architectural shift that makes multi-language pipelines possible. The runner does not need to understand the semantics of user code; it simply coordinates execution by sending and receiving data via a well-defined portable API.
Why This Matters for DevOps and Data Engineering Teams
For teams running production data pipelines on GCP, Runner V2 offers several practical benefits:
- Reduced maintenance overhead — bug fixes and performance improvements are applied across all SDKs simultaneously, rather than being ported language by language.
- Greater code reuse — high-quality connectors or transforms written in Java can be used in Python pipelines without rewriting them.
- Dependency flexibility — custom container support makes it easier to manage complex dependency trees in containerized CI/CD workflows.
- Future-proofing — as Apache Beam adds new language SDKs (e.g., Go, Typescript), they will benefit from the portability framework from the start, rather than requiring bespoke runner integration work.
Getting Started with Runner V2
Runner V2 can be enabled on existing Dataflow jobs by passing the --experiments=use_runner_v2 flag when submitting a pipeline. Consult the official Cloud Dataflow documentation for the latest guidance on enabling Runner V2, supported regions, and any feature-specific prerequisites such as Streaming Engine or Dataflow Shuffle enablement.
Teams evaluating Runner V2 for production workloads should test pipeline behavior carefully, particularly around stateful operations, I/O connectors, and any custom dependencies, before migrating from the default runner.