This article is based on content originally published on thenewstack.io. It has been expanded and rewritten for a DevOps engineering audience.
Why Storage Is Different in Kubernetes
Managing stateful workloads in Kubernetes is fundamentally different from deploying and scaling stateless microservices. Traditional storage systems were designed for static, long-lived servers — not ephemeral, scheduler-driven pods. Whether the backing store is a shared filesystem such as NFS or GlusterFS, or block storage such as Amazon EBS, Azure Disks, or GCE Persistent Disks, storage has historically been an external, often monolithic resource. Even Kubernetes Persistent Volumes (PVs) have historically been tightly coupled to underlying OS kernel modules, making them difficult to move, scale, or manage in a truly cloud-native way.
Container Attached Storage (CAS) is an emerging architectural pattern that attempts to resolve this tension by treating storage the same way Kubernetes treats compute — as a collection of loosely coupled, independently scalable microservices.
What Is Container Attached Storage (CAS)?
CAS enables Kubernetes users to treat storage entities as microservices rather than as monolithic external dependencies. The model has two distinct planes:
- Control plane: Deployed as a set of Custom Resource Definitions (CRDs) that manage low-level storage entities — provisioning, policies, snapshots, and metrics.
- Data plane: Runs as a collection of pods co-located with the application workload. It handles actual I/O transactions, translating requests into read and write operations on the underlying storage medium.
This clean separation delivers several key benefits:
- Portability: Persistence is decoupled from the underlying storage infrastructure, making workloads easier to move between environments.
- Scale-out: Storage capacity and performance can be expanded dynamically, independent of the compute layer.
- Hyper-convergence: Data (PV) and compute (Pod) are co-located on the same node, reducing network latency and improving throughput.
- Fault tolerance: Storage replicas are managed at the application layer, so pod rescheduling does not result in data loss.
What Is OpenEBS?
OpenEBS is an open-source, CNCF Sandbox project that implements the CAS model for Kubernetes. It can be deployed in clusters running on-premises, in managed Kubernetes services in public clouds, and even in air-gapped environments. Like other cloud-native platforms such as the Istio service mesh, OpenEBS follows established cloud-native design principles: declarative configuration, Kubernetes-native APIs, and integration with standard tooling including kubectl and Helm.
OpenEBS Architecture Overview
OpenEBS has three logical layers: a control plane, a data plane, and observability/tooling integrations. Together they implement the full lifecycle of cloud-native storage — from block device discovery through volume provisioning, snapshotting, cloning, and metrics export.
The OpenEBS Control Plane
The control plane lives close to the storage infrastructure and the Kubernetes master. Its responsibilities include:
- Provisioning and deprovisioning storage volumes
- Initiating snapshots and clones
- Creating and enforcing storage policies
- Exporting volume metrics to external systems such as Prometheus
The control plane is exposed externally through an API server that runs as a Pod and exposes a REST API for managing volumes and policies. Administrators submit declarative YAML manifests to this API server, which orchestrates the resulting workflow and makes scheduling calls to the Kubernetes API server to place volume pods on the data plane.
The provisioner component implements dynamic volume provisioning through the standard Kubernetes external storage plugin interface. When an application creates a PersistentVolumeClaim (PVC) referencing an OpenEBS StorageClass, the provisioner creates a corresponding PV based on the parameters defined in that StorageClass and binds it to the PVC.
The Node Device Manager (NDM) is one of the most important control plane components. NDM runs as a DaemonSet on every node in the cluster. Its job is to discover new block storage devices attached to each node, apply configurable filters, and report qualifying devices to the NDM operator, which registers them as block device resources. NDM maintains a real-time inventory of all registered block devices in etcd, making etcd the single source of truth for available storage in the cluster.
The OpenEBS Data Plane
The data plane runs close to application workloads on the worker nodes, not near the Kubernetes master. It hosts the storage engine — the component directly in the I/O path of a persistent volume. The storage engine intercepts read and write operations from application pods and translates them into operations on the underlying storage medium.
A key design principle of OpenEBS is that users can choose different storage engines for different workloads. A high-IOPS transactional database has very different storage requirements from a read-heavy content management system. OpenEBS exposes this flexibility through StorageClass configuration.
OpenEBS Storage Engines
OpenEBS provides three production-grade storage engines, each suited to a different class of workload.
1. cStor — The Recommended Engine for HA Workloads
cStor is the primary storage engine recommended for production, high-availability workloads such as databases. It is lightweight yet feature-rich, offering:
- Synchronous data replication across multiple replicas
- Snapshots and clones for backup and test/dev workflows
- Thin provisioning to avoid over-allocating disk space
- High resiliency and data consistency
- On-demand capacity and performance expansion
cStor’s synchronous replication model means that writes are acknowledged only after all configured replicas have committed the data. A cStor volume configured with three replicas can tolerate the loss of one node without data loss or application interruption. When a pod is rescheduled to a different node, the volume continues to serve I/O from the surviving replicas.
2. Jiva — The Lightweight Option for Simpler Use Cases
Jiva was the first storage engine included in OpenEBS. It runs entirely in user space and provides standard block storage capabilities, most notably synchronous replication. Because it does not require additional block storage devices to be attached to nodes, Jiva is well suited to smaller or resource-constrained environments.
However, Jiva’s simplicity comes with trade-offs. It lacks the advanced features of cStor — such as snapshots and clones — and is generally not recommended for mission-critical workloads that demand high IOPS or enterprise-grade storage capabilities.
3. Local PV — High-Performance Local Storage
OpenEBS Local PV is the simplest storage engine. It creates persistent volumes from locally-attached disks or host paths on individual worker nodes. Because I/O never crosses the network, Local PV can deliver the highest throughput and lowest latency of the three engines.
The trade-off is that Local PV volumes are tied to a specific node. If that node fails, the volume becomes unavailable until the node recovers. This means Local PV is best suited to applications that manage their own replication and high availability at the application layer — for example, a StatefulSet running a distributed database such as Cassandra or a distributed message queue such as Kafka.
Cloud-native applications that do not require storage-layer features such as replication, snapshots, or clones can safely use Local PV and benefit from near-disk I/O performance.
Additional and Emerging Storage Engines
Beyond the three production engines, the OpenEBS community has been developing additional options:
- MayaStor: A low-latency data engine written in Rust, designed for applications that require near-disk performance and direct block storage access. MayaStor was in active development at the time of the original article’s publication.
- ZFS Local PV: A variant of Local PV that adds RAID functionality, local snapshot support, and clone support — addressing some of the durability concerns associated with standard Local PV.
For the most current status of these engines, refer to the official OpenEBS documentation.
Choosing the Right Storage Engine
| Engine | Replication | Snapshots & Clones | Best For |
|---|---|---|---|
| cStor | Yes (synchronous) | Yes | HA databases, stateful production workloads |
| Jiva | Yes (synchronous) | No | Smaller workloads, nodes without extra block devices |
| Local PV | No (application-level) | No (ZFS variant: Yes) | Distributed apps with self-managed HA, high-throughput workloads |
Summary
OpenEBS brings the Container Attached Storage model to Kubernetes by implementing a cloud-native control plane and a flexible, per-workload data plane. Its architecture — modeled on the same microservices principles that underpin Kubernetes itself — delivers portability, dynamic scalability, and fault tolerance to stateful applications without requiring legacy storage infrastructure.
For teams running stateful workloads in Kubernetes and looking for a storage solution that fits natively into a cloud-native operations model, OpenEBS is a well-architected option worth evaluating. Start with the OpenEBS documentation to understand which storage engine best matches your workload’s requirements.