Originally reported via technotification.com (2019). Details reflect the state of the project at the time of writing.
What Is Ballista?
Ballista is a distributed computing platform introduced by software engineer Andy Grove. It is written in Rust and built on top of Kubernetes, using a Rust implementation of Apache Arrow as its in-memory data model. The project was positioned as an attempt to bring Apache Spark-style distributed query execution to the Rust ecosystem.
Background: From DataFusion to Ballista
Ballista grew out of Grove’s earlier work on the DataFusion project — an in-memory query engine built on Apache Arrow. Grove started DataFusion approximately eighteen months before the Ballista announcement, with the original goal of creating a full distributed computing platform in Rust capable of competing with Apache Spark. That goal proved overly ambitious at the time.
“Unsurprisingly, this turned out to be an overly ambitious goal at the time and I fell short of achieving that. However, some very good things came out of this effort. We now have a Rust implementation of Apache Arrow with a growing community of committers, and DataFusion was donated to the Apache Arrow project as an in-memory query execution engine and is now starting to see some early adoption.”
— Andy Grove
After stepping back from DataFusion and Arrow to focus on other work, Grove returned with a new proof-of-concept: Ballista. This second attempt benefited directly from the groundwork laid by DataFusion and the Rust Arrow implementation.
How Ballista Works
A Ballista cluster is composed of individual pods running inside a Kubernetes cluster. Applications are deployed using the Ballista CLI, which handles communication with Kubernetes. Service discovery within the cluster is handled natively via Kubernetes service discovery mechanisms.
At the time of announcement, Ballista did not yet include a distributed query planner. This meant that application developers were responsible for manually constructing the query plans to be distributed and executed across cluster nodes.
Roadmap for v1.0.0
Grove outlined the following milestones on the Ballista roadmap for the v1.0.0 release:
- Distributed query planner: Automate the construction and distribution of query plans across the cluster.
- Full DataFusion plan and expression support: Cover all DataFusion logical plans and expressions within the distributed execution layer.
- User-defined code execution: Allow custom user code to run as part of a distributed query.
- Interactive SQL over gRPC: Enable interactive SQL queries to be issued against a live cluster using gRPC.
- Arrow Flight protocol and Java bindings: Add support for the Arrow Flight wire protocol and expose Java language bindings.
Impact on the Apache Arrow Ecosystem
Even in its early proof-of-concept stage, Ballista had already produced tangible contributions to the broader Apache Arrow project. Development work on Ballista resulted in three DataFusion pull requests being merged into the Apache Arrow codebase, helping to shape the requirements and direction of the DataFusion query engine.
Why It Matters for DevOps and Data Engineering Teams
Ballista represents an important experiment in bringing memory-safe, high-performance distributed data processing to cloud-native infrastructure. By running natively on Kubernetes and leveraging Rust’s performance characteristics alongside Apache Arrow’s columnar memory format, the project aimed to offer a modern alternative to JVM-based frameworks like Apache Spark — with lower overhead and tighter integration with container orchestration.
For teams already running Kubernetes workloads and looking to reduce JVM dependency or improve resource efficiency in data processing pipelines, the direction Ballista pointed toward remains relevant — even if the project has since evolved significantly within the Apache Arrow DataFusion ecosystem.
Note: This article was written based on the original 2019 announcement. The Ballista and DataFusion projects have continued to develop under the Apache Arrow umbrella. Readers should consult the Apache Arrow project website for current status and documentation.