Continuous Delivery with Kubernetes: From Simple to Production-Ready

This article is based on a piece originally published on InfoQ and describes the approach the Weaveworks team developed for continuous delivery on Kubernetes. Architecture diagrams are sourced from that original article.

What Is Continuous Delivery and Why Does It Matter?

Continuous delivery is a software engineering practice in which code is produced and released in small, frequent increments rather than large, infrequent batches. Eliminating a “release day” mindset reduces the risk of each individual deployment: smaller changes are easier to reason about, easier to test, and easier to roll back when something goes wrong.

For teams running microservices, continuous delivery carries an additional constraint: because services communicate via APIs, each service must maintain a degree of backwards compatibility across releases. This decouples deployment schedules so that individual teams can ship independently without forcing a synchronized “big bang” release across the entire organization.

Conway’s Law observes that software architectures mirror the communication structures of the organizations that build them. Microservices and containers are therefore as much an organizational shift as a technical one — each microservice can be owned and delivered by a separate, autonomous team.

A Brief Overview of Kubernetes Primitives

Before examining the delivery architectures, it helps to understand a few core Kubernetes concepts.

• Pod: The smallest schedulable unit in Kubernetes. A pod wraps one or more containers that share a network namespace — containers inside the same pod can reach each other via localhost.
• Mortal pods: Pods are not self-healing. If the underlying node disappears (hardware failure, cloud provider interruption), the pods on that node disappear with it.
• Deployment: The recommended way to run stateless workloads. A Deployment declares a desired replica count and keeps that many pods running at all times. If a node fails, the Deployment controller reschedules the affected pods elsewhere. Deployments also support rolling updates and rollback via the Kubernetes API.

Kubernetes does not prescribe a single continuous delivery workflow. It provides the primitives; how you wire them into a pipeline is up to you — and that is exactly what the following architectures explore.

Version 1: CI System Drives Everything

Architecture Overview

In the simplest setup, the entire pipeline lives inside your CI system:

1. A developer pushes code to version control.
2. The CI system detects the push, builds a Docker image, and tags it with the commit’s SHA-1 hash.
3. The CI system pushes the tagged image to a Docker registry.
4. The CI system calls kubectl set image to tell the running Deployment to use the new image tag.
5. Kubernetes performs a rolling update, pulling the new image and replacing pods incrementally.

The initial manual deployment is straightforward:

kubectl apply -f service.yamlCode language: CSS (css)

Once the pipeline matures, the CI system takes over this step automatically using kubectl set image:

# Example: update the front-end container in the "frontend" deployment
kubectl set image deployment/frontend frontend=myregistry/frontend:<new-sha>Code language: PHP (php)

Rollbacks in Version 1

Rolling back in this architecture requires pushing new code. The developer reverts the offending commit on main (or force-pushes a reset), which triggers the CI system to rebuild the old code as a new image and redeploy it. This means rollback time is gated by your full build-and-push pipeline — exactly the wrong moment to be slow.

Downsides of Version 1

• Slow rollbacks: Every rollback triggers a full Docker build and push, which consumes disk I/O and network bandwidth.
• Tight coupling: The same pipeline that deploys to production also gates every environment. Running different image versions across dev, staging, and production is difficult.
• No single source of truth: The live state of the cluster is not recorded anywhere auditable — only the CI logs reflect what was deployed.

Version 2: Centralized Config Repository

Architecture Overview

Version 2 introduces a dedicated config repository — a separate Git repo that holds all Kubernetes YAML manifests for every microservice in the application. Rather than keeping deployment manifests alongside each service’s application code, all manifests are centralised in one place.

This config repo becomes the single source of truth (SSOT) for the cluster. If the production cluster is accidentally deleted, it can be reconstructed entirely from this repo.

Committing a Change

1. Developer pushes code to the application repo.
2. CI system builds and pushes a new Docker image to the registry.
3. CI system clones the config repo, updates the relevant Kubernetes YAML to reference the new image tag, and commits the change back to the config repo.
4. CI system applies the updated manifest to the Kubernetes cluster.
5. Kubernetes pulls the new image from the registry.

Downsides of Version 2

• CI system overloaded: The CI system now handles image building, registry pushes, manifest editing, repo commits, and cluster deployments. This violates the single-responsibility principle and makes the pipeline harder to debug and maintain.
• Rollbacks still require code pushes: To roll back, a developer must manually update the config repo or push a revert commit. Any out-of-band kubectl command that bypasses the config repo immediately breaks the SSOT guarantee.

Version 3: Adding a Release Manager

Architecture Overview

The solution to Version 2’s overloaded CI system is to introduce a dedicated release manager. The Weaveworks team uses Weave Flux, an open-source release manager also available as part of Weave Cloud. Other options exist — Spinnaker is a well-known alternative — but Weave Flux is designed specifically for container-centric workflows and is intentionally simpler than Spinnaker.

With a release manager in place, each component in the pipeline has a single, well-defined responsibility:

• CI system: Build code into a container image and push it to the Docker registry. Nothing more.
• Release manager: Watch the registry for new images, clone the config repo, update the relevant manifests, push the changes back to the config repo, and apply them to the Kubernetes cluster.
• Config repo: Remain the SSOT for the entire cluster state.
• Kubernetes: Pull images and reconcile desired state.

Per-Environment Policies

The release manager supports configurable release policies per environment. For example:

• Staging: Automatically release every new image as soon as it appears in the registry.
• Production: Require a manual promotion via the release manager’s UI before deploying.

This eliminates the tight coupling between individual microservice repos and what actually runs in production. Different services can be at different image versions without any special-casing in the CI pipeline.

Rollbacks in Version 3

Rollbacks are now fast and code-free. The operator instructs the release manager to roll back to a previous image tag. The release manager updates the config repo and applies the change to the cluster — no rebuild, no new commit to the application repo, no CI pipeline run required. Because every change goes through the release manager and is committed to the config repo, the SSOT remains intact.

Comparing the Three Architectures

Capability	V1: CI Only	V2: Config Repo	V3: Release Manager
Single source of truth	No	Yes	Yes
Fast rollbacks (no rebuild)	No	No	Yes
Per-environment policies	No	Partial	Yes
CI system responsibility	High	Very high	Low (build only)
Cluster reconstructable from Git	No	Yes	Yes

Key Principles for CD on Kubernetes

Regardless of which architecture fits your current scale, three principles should guide every continuous delivery implementation on Kubernetes:

1. Automate deployments. Manual kubectl apply commands are fine for bootstrapping, but every production change should flow through an automated pipeline with an audit trail.
2. Establish a single source of truth. Store all Kubernetes manifests in version control. The live cluster state should always be derivable from a Git repo, not from someone’s memory of what kubectl commands were run.
3. Make rollbacks fast and code-free. A rollback should not require rebuilding a Docker image. Tag images immutably at build time (e.g. with the Git SHA), and ensure your pipeline can repoint a Deployment to a previous tag in seconds.

The more complex your microservices application, the more likely you will need the full Version 3 approach — or an equivalent GitOps-style toolchain. But every team benefits from starting with clear automation, version-controlled configuration, and a rollback story that does not involve pushing new code under pressure.