A Blueprint for Running Stateful Services on Kubernetes

This article is based on a post originally published on thenewstack.io, with additional technical context added for clarity.

Why Stateful Applications Are Difficult to Run on Kubernetes

Managing stateful applications has been a challenge for engineering and operations teams long before Kubernetes existed. The core tension is this: Kubernetes is built around the concept of ephemeral, interchangeable containers, while stateful applications fundamentally require continuity — of identity, data, and network address.

Understanding how to bridge that gap is essential for any team running databases, caches, message queues, or content management systems on Kubernetes.

What Does “State” Mean in a Cloud-Native Context?

State refers to the condition an application is in at a particular point in time. A stateful application changes its behavior based on previous interactions — it maintains a memory of past transactions. Common examples include:

• Databases (PostgreSQL, MySQL, Cassandra, MongoDB)
• Caches (Redis, Memcached)
• Message queues (Kafka, RabbitMQ)
• Content management systems (WordPress)

For a stateful application to work correctly, it must have a reliable location to persist its data throughout its lifespan. On a single-server deployment, this is straightforward: write data directly to the host filesystem. On a distributed, multi-node Kubernetes cluster, the problem becomes significantly more complex.

The Core Challenge: Ephemerality vs. Persistence

Kubernetes containers are designed to be ephemeral — they can be stopped, rescheduled, or replaced at any time. When a Pod is rescheduled to a different node, any data written to its local filesystem is lost by default.

When scaling a stateful application to multiple instances, two additional challenges arise:

1. Independent state per instance: Each replica must maintain its own consistent dataset (common in databases with leader/follower replication).
2. Shared state across instances: Multiple replicas must access the same data concurrently, which requires shared or distributed storage.

Kubernetes supports both patterns, but neither works out of the box — both require deliberate infrastructure and configuration decisions.

Infrastructure Considerations: Start with Your Cluster Design

The effectiveness of stateful services on Kubernetes starts at the infrastructure layer. Poor cluster design will undermine even the most carefully written application configuration. Key decisions include:

Managed vs. Unmanaged Clusters

Managed Kubernetes services (such as GKE, EKS, and AKS) offload control-plane management and often provide tighter integration with cloud-native storage backends. Unmanaged clusters give you more control but place the burden of upgrades, etcd management, and high availability on your team.

Node Hardware

For I/O-intensive stateful workloads, node hardware matters significantly. Consider dedicating nodes with local NVMe SSDs to database workloads, or using instance types optimized for memory and storage throughput. Node taints and tolerations can enforce workload placement on the appropriate hardware.

Storage Backend

The choice of storage backend directly affects latency, throughput, and durability. Common options include:

• Cloud block storage (AWS EBS, GCP Persistent Disk): Durable, easy to provision, but typically limited to a single node attachment at a time.
• Distributed storage systems (Ceph, Rook, Longhorn): Enable ReadWriteMany access and data replication across nodes.
• Local volumes: Lowest latency, but not portable across nodes — best paired with application-level replication.
• NFS / shared filesystems: Suitable for workloads requiring concurrent multi-pod access.

Multi-Cloud and Data Residency

If your stateful workloads span multiple clouds or must meet data residency requirements, you will need storage solutions that respect regional boundaries and replication policies. This adds significant operational complexity that should be planned at the architecture stage.

Deploying Stateful Workloads: StatefulSets

The primary Kubernetes API object for managing stateful applications is the StatefulSet. While a Deployment manages stateless Pods that are interchangeable, a StatefulSet provides guarantees that stateful applications require:

• Stable, unique Pod identity: Each Pod receives a persistent, ordinal-based name (e.g., app-0, app-1, app-2) that persists across rescheduling.
• Stable network identity: Each Pod gets a consistent DNS hostname via a headless Service, enabling direct per-pod addressing (e.g., app-0.myservice.default.svc.cluster.local).
• Ordered deployment and scaling: Pods are created, updated, and deleted in a predictable sequential order, which is critical for applications like databases that require a primary to be ready before replicas start.
• Persistent volume per Pod: Using volumeClaimTemplates, each Pod in a StatefulSet gets its own PersistentVolumeClaim (PVC), which is retained even if the Pod is deleted and rescheduled.

Example: A Minimal StatefulSet Definition

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-database
spec:
  selector:
    matchLabels:
      app: my-database
  serviceName: "my-database"
  replicas: 3
  template:
    metadata:
      labels:
        app: my-database
    spec:
      containers:
      - name: db
        image: my-database-image:latest
        volumeMounts:
        - name: data
          mountPath: /var/lib/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 50Gi
Code language: JavaScript (javascript)

In this example, three Pods (my-database-0, my-database-1, my-database-2) are created sequentially. Each Pod gets its own 50 GiB PersistentVolumeClaim, provisioned automatically from the fast-ssd StorageClass.

Pod Identity and Ordinal Indexing

Every Pod in a StatefulSet is assigned a zero-based ordinal index. This index serves multiple purposes:

• It forms part of the Pod’s unique name and DNS hostname.
• It determines the order in which Pods are deployed, updated, and terminated.
• Applications can use the ordinal to self-configure — for example, Pod app-0 can elect itself as the primary database node.

This predictable identity makes StatefulSets particularly well-suited for clustered applications like Kafka, Elasticsearch, Cassandra, and Redis Sentinel, which rely on stable member identities for cluster formation and leader election.

Update Strategies for StatefulSets

StatefulSets support two update strategies, configured via spec.updateStrategy:

• RollingUpdate (default): Pods are updated one at a time, in reverse ordinal order (highest index first). This allows databases to update replicas before the primary, reducing risk.
• OnDelete: Pods are only updated when manually deleted. This gives operators full control over when each Pod receives the update, which is useful for critical production databases.

The partition field within RollingUpdate allows canary-style rollouts: only Pods with an ordinal at or above the partition value are updated, leaving lower-ordinal Pods on the previous version until you are ready to proceed.

Persistent Volumes and Dynamic Provisioning

Kubernetes PersistentVolumes (PVs) decouple storage from Pod lifecycle. A PersistentVolumeClaim (PVC) requests storage from the cluster, and the cluster fulfills it either from pre-provisioned volumes or dynamically via a StorageClass.

Dynamic provisioning — where a StorageClass automatically creates a PV when a PVC is submitted — significantly reduces operational overhead for stateful workloads. Most managed Kubernetes platforms ship with StorageClasses backed by their native block storage products, making dynamic provisioning the default recommendation for cloud deployments.

Important: PVCs created by volumeClaimTemplates in a StatefulSet are not deleted when the StatefulSet is scaled down or deleted. This is intentional — it protects your data from accidental loss. You must manually delete PVCs when you are certain the data is no longer needed.

Headless Services and Internal DNS

A StatefulSet requires a headless Service (a Service with clusterIP: None) to provide stable DNS records for each Pod. Unlike a standard Service that load-balances across all Pods, a headless Service creates individual DNS A records for each Pod.

This enables both service discovery and direct per-pod communication — essential for database replication protocols, cluster gossip, and client-side load balancing strategies used by tools like Kafka clients or Redis Cluster.

apiVersion: v1
kind: Service
metadata:
  name: my-database
spec:
  clusterIP: None
  selector:
    app: my-database
  ports:
  - port: 5432
    name: db

Operational Best Practices

• Use Pod Disruption Budgets (PDBs): Define a PDB to limit how many Pods can be simultaneously unavailable during voluntary disruptions such as node drains or cluster upgrades.
• Set resource requests and limits: Stateful workloads, especially databases, can be sensitive to CPU and memory starvation. Setting appropriate requests ensures the scheduler places Pods on nodes with sufficient capacity.
• Pin Pods to dedicated nodes: Use node affinity rules or taints/tolerations to keep stateful workloads off nodes shared with bursty stateless services.
• Back up PersistentVolumes regularly: Kubernetes does not automatically back up volume data. Use tools like Velero or cloud-native snapshot APIs to implement a backup and recovery strategy.
• Monitor storage I/O: Disk throughput and IOPS limits are a common performance bottleneck for stateful workloads. Instrument your storage layer and set alerts for saturation.
• Test failure scenarios: Deliberately terminate Pods and drain nodes in a staging environment to validate that your StatefulSet recovers correctly and that data is intact.

Summary

Kubernetes was not originally designed with stateful applications as a primary use case, but the platform has matured significantly. StatefulSets, dynamic volume provisioning, headless Services, and StorageClasses together provide a solid foundation for running databases and other stateful workloads in production.

Success depends on deliberate decisions at every layer: the storage backend you choose, the cluster topology you design, the update strategy you apply, and the operational processes you put in place around backup, monitoring, and failure testing. Get those decisions right, and Kubernetes can run stateful services with the same reliability and automation it brings to stateless workloads.