The etcd maintainers released etcd v3.4, a milestone focused on stability, performance, and ease of operation for Kubernetes clusters. Key highlights include fully concurrent backend read transactions, a pre-vote phase for Raft leader elections, a new Learner non-voting member type, and a redesigned client balancer. This article walks through each improvement in detail.
What Is etcd and Why Does It Matter for Kubernetes?
etcd is the distributed key-value store that Kubernetes uses as its primary backing store for all cluster state data — from node registrations and pod specs to service endpoints and configuration. Its reliability and performance directly affect the health of every Kubernetes cluster. etcd implements the Raft consensus algorithm to replicate data reliably across a cluster of nodes, tolerating failures as long as a quorum of members remains available.
Key Features and Improvements in etcd v3.4
1. Better Backend Storage: Fully Concurrent Read Transactions
etcd v3.4 addresses a significant performance bottleneck that affected large-scale Kubernetes deployments: blocking between read and write transactions in the storage backend.
Previously, the backend commit operation — which flushes pending writes — would block all incoming read transactions, even when there were no pending writes to flush. This caused read requests to queue up unnecessarily, sometimes surfacing as log warnings such as:
read-only range request … took too long to executeIn v3.4, the commit operation no longer blocks reads, improving the performance of long-running read transactions significantly.
The team went further and made backend read transactions fully concurrent. Prior to this change, ongoing long-running read transactions would block both writes and subsequent reads. According to the etcd release notes, this change resulted in:
- ~70% increase in write throughput when long-running reads are in progress.
- ~90% reduction in P99 write latency under the same conditions.
The team validated these gains by running a Kubernetes 5,000-node scalability test on GCE and observed consistent improvements across the board.
2. Improved Raft Voting Process: Pre-Vote
etcd uses the Raft consensus protocol for data replication. In a Raft cluster:
- A single leader is elected by the cluster members.
- The leader replicates data to followers and sends periodic heartbeats to maintain authority.
- Followers forward client proposals to the leader.
- An entry is committed once a quorum of members acknowledges it.
In the standard Raft implementation, a leader immediately steps down to follower status upon receiving any message carrying a higher term number — without performing any cluster-wide health checks first. This behavior can be exploited accidentally by a flaky or rejoining node:
- A node drops off and reconnects intermittently, accumulating a higher term.
- When it rejoins, it ignores messages with lower terms and broadcasts its higher term.
- The current stable leader receives this message, steps down, and triggers an unnecessary re-election — disrupting cluster availability.
Network partitions amplify the problem: a partitioned node that regains connectivity can immediately force a leader re-election, even though the existing leader is perfectly healthy.
The fix: Pre-Vote. etcd v3.4 introduces a new pre-candidate node state. Before incrementing its term and launching a real election, a pre-candidate must first solicit informal votes from a majority of peers to confirm it is sufficiently up-to-date. Only if it wins this pre-vote phase does it proceed to increment its term and start a formal election. This extra handshake means:
- Disruptive nodes cannot force re-elections unless they genuinely have the support of a quorum.
- A healthy leader that maintains quorum connectivity stays in power, improving cluster stability.
3. New Raft Non-Voting Member: The “Learner”
Adding or removing members from a running etcd cluster (membership reconfiguration) is one of the riskiest operational tasks. Even when a reconfiguration does not change quorum size, it can expose latent issues that destabilise the cluster. A new member that joins without being caught up on the leader’s log is especially dangerous — it could inadvertently reduce the effective quorum before it has replicated enough state to participate meaningfully.
etcd v3.4 introduces the Learner role to address this. A Learner:
- Joins the cluster as a non-voting member.
- Receives all log updates from the leader, catching up to current state.
- Does not count toward quorum, so it cannot affect leader evaluation of peer activeness.
- Serves only as a standby node until an operator explicitly promotes it to a full voting member.
By decoupling log replication from quorum participation, the Learner role makes membership changes far safer and reduces the window of increased cluster vulnerability during scaling operations.
4. Simplified Client Balancer Failover Logic
etcd clusters are designed to tolerate node and network failures transparently, presenting clients with a single logical cluster view across multiple server endpoints. However, cluster-side fault tolerance does not automatically guarantee client-side liveness — if a client is connected to a failing endpoint, it needs its own mechanism to detect the failure and reconnect.
Historically, the etcd client balancer was tightly coupled to an older gRPC interface. Every gRPC dependency upgrade risked breaking client failover behaviour, and fixing those regressions consumed a disproportionate share of engineering effort. The resulting code was complex and made fragile assumptions about server connectivity.
In v3.4, the balancer failover logic was simplified significantly. Rather than maintaining a complex list of unhealthy endpoints, the new logic adopts a straightforward approach: when the client is disconnected from its current endpoint, it immediately tries the next available endpoint. This change makes the client easier to reason about, easier to test, and more resilient across gRPC version upgrades.
Summary of etcd v3.4 Improvements
| Area | What Changed | Benefit |
|---|---|---|
| Backend Storage | Commit no longer blocks reads; reads fully concurrent | Higher write throughput, lower P99 write latency under load |
| Raft Voting | Pre-Vote phase before term increment | Fewer disruptive re-elections; more stable leadership |
| Membership | Learner non-voting member role | Safer cluster scaling and membership reconfiguration |
| Client Balancer | Simplified failover; no unhealthy endpoint list | More predictable client behaviour across gRPC upgrades |
What This Means for Kubernetes Operators
For teams running large Kubernetes clusters, etcd v3.4 is a meaningful upgrade:
- Large clusters with heavy read traffic will see direct latency and throughput benefits from the concurrent read transaction work.
- Clusters that experience network instability will benefit from the pre-vote mechanism, which prevents flapping nodes from forcing unnecessary leader elections.
- Operators who regularly scale etcd clusters should use the Learner role when adding new members to reduce risk during the catch-up phase.
- Client application developers can expect more predictable reconnection behaviour from the etcd v3.4 client library.
As always, test etcd upgrades in a non-production environment first, review the official etcd changelog for any breaking API changes, and consult your Kubernetes distribution’s compatibility matrix before upgrading etcd on production clusters.
Source: Originally reported via Packt Publishing’s technology news coverage.