Continuous Deployment with Google Kubernetes Engine and Kubernetes

This article is based on a tutorial originally published on semaphoreci.com. Commands and sample output reflect the original source material, which dates from early 2017. Some tooling details may have changed — see review notes.

Introduction

This tutorial walks through deploying a sample microservices application to Kubernetes and wiring up a continuous deployment pipeline using Semaphore CI. Along the way you will get a practical introduction to Kubernetes concepts, Google Kubernetes Engine (GKE), and the kubectl CLI.

Kubernetes — often abbreviated k8s — is a container orchestration platform. Although it supports multiple container runtimes, this tutorial uses Docker because it remains the most widely used runtime in practice. Kubernetes is a complex distributed system, but you do not need to understand all of it to get started. You only need access to a running cluster, and GKE makes that straightforward.

Prerequisites: Familiarity with Docker and a basic understanding of what container orchestration does. The hands-on sections assume you have the tooling listed below installed and configured.

Kubernetes Concepts You Need to Know

Kubernetes is an open-source container orchestration platform originally inspired by Google’s internal Borg system. It follows a control-plane/worker architecture:

• Control plane (master) nodes schedule workloads and maintain desired state.
• Worker nodes run the container workloads. Each worker runs a kubelet agent that communicates with the control plane and manages the local container runtime.

The four core Kubernetes primitives used in this tutorial are:

1. Pods — The smallest deployable unit. A pod contains one or more containers that always run on the same node and share networking. A modern web application might group a frontend and backend container into one pod.
2. Services — Stable network endpoints that expose pods to other pods or to the public internet. Kubernetes supports several service types, including a ClusterIP (internal only) and LoadBalancer (external, cloud-managed).
3. Deployments — Manage the desired state of a set of pods, including the container image version, environment variables, replica count, and rolling update strategy. Deployments work with ReplicaSets to provide horizontal scaling and built-in liveness/readiness probes.
4. kubectl — The command-line interface for every cluster operation. It accepts YAML or JSON resource definitions via the -f flag.

These four concepts are enough to deploy and operate the sample application below. The official Kubernetes documentation provides extensive guides and API references for deeper exploration.

The Sample Application

The demo application consists of three services:

• server — A Node.js HTTP server. It accepts POST / to increment a counter stored in Redis, and GET / to retrieve the current counter value.
• counter — A loop that continuously sends POST requests to the server to increment the counter by random amounts.
• poller — A loop that continuously sends GET requests to the server and prints the counter value.

Redis is used as the shared data store. You will first run the application locally with Docker Compose to verify it works, then deploy it to Kubernetes on GKE.

Pre-Launch Checklist

Gather the following before starting:

• A Google Cloud Platform (GCP) account with a billing method attached.
• docker and docker-compose installed locally to build and push images.
• gcloud CLI installed and authenticated (gcloud auth login).
• kubectl installed (can be installed via gcloud components install kubectl).
• A Semaphore CI account for the CD pipeline.

Creating the GKE Cluster

Step 1 — Create a GCP project

Create a new GCP project. Replace semaphore-gke-tutorial with your preferred project ID:

gcloud alpha projects create semaphore-gke-tutorial

After creating the project, open the GKE section of the GCP Console, select your project, and link it to a billing account. The GKE API may take a minute or two to activate.

Step 2 — Create the Kubernetes cluster

gcloud container clusters create demo \
    --zone europe-west1-a \
    --project semaphore-gke-tutorial

The cluster is named demo; change this to suit your needs. The --zone flag controls the GCP region. When provisioning completes, you will see output similar to:

Creating cluster demo...done.
Created [https://container.googleapis.com/v1/projects/semaphore-gke-tutorial/zones/europe-west1-b/clusters/demo].
kubeconfig entry generated for demo.
NAME  ZONE            MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
demo  europe-west1-b  1.4.7           104.199.44.242  n1-standard-1  1.4.7         3          RUNNINGCode language: CSS (css)

Step 3 — Fetch cluster credentials

gcloud container clusters get-credentials demo \
    --project semaphore-gke-tutorial \
    --zone europe-west1-bCode language: JavaScript (javascript)

This command writes a kubectl context to your local kubeconfig file and sets it as the active context. Confirm with:

kubectl config get-contextsCode language: JavaScript (javascript)

Expected output (one row per configured cluster):

CURRENT   NAME                                                CLUSTER                                             AUTHINFO                                            NAMESPACE
*         gke_semaphore-gke-tutorial_europe-west1-b_demo      gke_semaphore-gke-tutorial_europe-west1-b_demo      gke_semaphore-gke-tutorial_europe-west1-b_demoCode language: PHP (php)

The asterisk (*) marks the active context. Note the value in the NAME column — you will reference it when passing --context to kubectl. Test that the cluster is reachable:

kubectl get pods --context gke_semaphore-gke-tutorial_europe-west1-b_demoCode language: JavaScript (javascript)

No output is expected because no pods exist yet.

Running the Application Locally

Clone (or fork) the sample application repository. Start all services with Docker Compose:

docker-compose up --build

Once the build completes, you will see streaming output like this:

server_1   | Server running on port 8080!
counter_1  | Incrementing counter by 5 ...
poller_1   | Current counter: 117
counter_1  | Incrementing counter by 2 ...
poller_1   | Current counter: 119
counter_1  | Incrementing counter by 10 ...
poller_1   | Current counter: 129

The counter is incrementing correctly. Now push the images to Google Container Registry (GCR) so the cluster can pull them.

Push images to GCR

Open docker-compose.yml and update the image names with your GCR registry hostname and project ID (e.g. gcr.io/semaphore-gke-tutorial/server). Then authenticate Docker with GCR:

gcloud docker --authorize-only --project semaphore-gke-tutorial

Build and push the images:

docker-compose build
docker-compose push

GKE clusters are automatically granted pull access to the GCR registry associated with the same GCP project, so no additional authentication configuration is required on the cluster side.

Creating the First Pod

Set the default context so you do not need to pass --context to every command:

kubectl config use-context gke_semaphore-gke-tutorial_europe-west1-b_demoCode language: PHP (php)

Create a development namespace to isolate resources. Using namespaces to separate environments (development, staging, production) is a Kubernetes best practice:

kubectl create -f k8s/development-namespace.ymlCode language: PHP (php)

Open k8s/development-pod.yml and fill in the TODO placeholders with your actual image names. The pod manifest is conceptually similar to docker-compose.yml: it lists containers with their images, ports, and environment variables. Create the pod:

kubectl create -f k8s/development-pod.yml --namespace developmentCode language: PHP (php)

Check its status:

kubectl get pods --namespace developmentCode language: JavaScript (javascript)

NAME      READY     STATUS    RESTARTS   AGE
demo      4/4       Running   0          7s

4/4 means all four containers in the pod are running. If you see an error, the image name is likely wrong. Delete the pod and try again:

kubectl delete pod demo --namespace developmentCode language: JavaScript (javascript)

Use kubectl describe for detailed debugging information:

kubectl describe pod demo --namespace developmentCode language: PHP (php)

Check the poller logs to confirm the counter is incrementing:

kubectl logs demo -c poller --namespace developmentCode language: PHP (php)

You have just deployed a microservices application to Kubernetes. However, this single-pod setup has a critical scalability problem: because all four containers (redis, server, counter, poller) are in one pod, horizontal scaling would create multiple independent Redis instances, each with its own counter. The solution is to split the application into separate pods connected by Kubernetes services.

kubectl delete pod demo --namespace developmentCode language: JavaScript (javascript)

Production-Grade Services and Deployments

The sample repository includes a set of production-oriented manifest files under the k8s/ directory:

• k8s/production-redis-pod.yml — Runs the Redis container as a standalone pod.
• k8s/production-redis-service.yml — Exposes Redis to other pods (specifically the server) via a ClusterIP service.
• k8s/production-server-service.yml — Exposes the server pods to the counter and poller via a ClusterIP service.
• k8s/production-server-deployment.yml — A Deployment for the server container, enabling independent horizontal scaling.
• k8s/production-support-deployment.yml — A Deployment grouping the counter and poller containers.

The recommended deploy order is:

1. Create the services (so DNS names are available before pods start).
2. Create the Redis pod.
3. Create the server deployment.
4. Create the support deployment.

The repository provides a script/bootstrap script that runs these steps in order. Pass the target namespace as an argument:

script/bootstrap development

Verify the pods:

kubectl get pods --namespace developmentCode language: JavaScript (javascript)

NAME                      READY     STATUS    RESTARTS   AGE
redis                     1/1       Running   0          41s
server-506448125-0lq9m    1/1       Running   0          32s
server-506448125-a4u1g    1/1       Running   0          32s
server-506448125-m7srr    1/1       Running   0          32s
support-592105180-9pl4j   2/2       Running   0          21s
support-592105180-blnse   2/2       Running   0          21s
support-592105180-xcqbn   2/2       Running   0          21s

Three server replicas and three support replicas are running — all sharing a single Redis instance. Check the poller output from one of the support pods:

kubectl logs support-592105180-xcqbn -c poller --namespace developmentCode language: PHP (php)

Scaling a deployment

To scale the support deployment, edit the replicas value in k8s/production-support-deployment.yml and apply the change:

kubectl apply -f k8s/production-support-deployment.yml --namespace developmentCode language: PHP (php)

The apply command creates the resource if it does not exist, or updates it if it does — making it the right tool for both initial deployments and subsequent changes.

Setting Up Continuous Deployment with Semaphore CI

With a working manual deploy process, you can now automate it. The CD pipeline performs these steps on every successful build:

1. Install kubectl, gcloud, docker-compose, and docker on the CI worker.
2. Authenticate the CI worker with GCP using a service account key.
3. Fetch cluster credentials so kubectl is configured.
4. Authenticate Docker with GCR.
5. Build and push updated Docker images to GCR.
6. Apply Kubernetes resource manifests to the cluster.

Create a GCP service account for CI

In the GCP Console, navigate to IAM & Admin → Service Accounts. With your project selected, create a new service account with the Owner role and download its JSON key file. Store this file securely — you will upload it to Semaphore CI as a configuration file, not a secret embedded in your repository.

Configure Semaphore CI

1. Create a new project in Semaphore CI and configure it to use the Docker platform.
2. Upload the GCP service account JSON key file (e.g. auth.json) as a configuration file. Note the absolute path Semaphore shows you (e.g. /home/runner/auth.json).
3. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to that absolute path. Do not use the ~ shorthand — use the full path.

Add CI scripts

The repository provides two scripts:

• script/ci/setup — Installs tooling and configures gcloud and kubectl authentication. Fill in the TODO placeholders with your project ID, zone, and cluster name. Add this script as the setup command in your Semaphore CI project.
• script/ci/deploy — Builds and pushes Docker images to GCR, then runs kubectl apply against the manifests in k8s/. Using apply means the same script handles both first-time deployments and updates. Add this script as a deploy command in your pipeline.

After committing these scripts and pushing a build, Semaphore will run the full pipeline. When it completes successfully, verify the production deployment:

kubectl get pods --namespace productionCode language: JavaScript (javascript)

You can now make changes — for example, editing the replica count in a deployment manifest — commit them, and watch Semaphore CI apply the changes automatically.

What to Explore Next

This tutorial covered the fundamentals. Here are practical improvements to investigate:

• Stateful storage. Redis data is lost if the Redis pod is deleted. For production, use a persistent volume or a managed database service so state survives pod restarts.
• Image tagging. The tutorial uses images without version tags (:latest behaviour). A production-grade pipeline should tag images with the git commit SHA and set imagePullPolicy: IfNotPresent so each deploy uses an immutable, traceable image.
• Public internet access. The server deployment is not publicly exposed. Kubernetes makes this straightforward: update the server service type to LoadBalancer and GKE will provision a cloud load balancer automatically.
• Liveness and readiness probes. Add HTTP health check probes to your deployments so Kubernetes can detect and restart unhealthy pods and avoid routing traffic to pods that are not yet ready.
• Kubernetes documentation. The official pod, service, and deployment guides are well-written and are the best reference for deepening your understanding of the concepts introduced here.

With a working Kubernetes cluster on GKE and a Semaphore CI pipeline in place, you have a solid foundation for shipping containerised microservices reliably and repeatably.

Original tutorial source: semaphoreci.com. Sample repository: github.com/ahawkins/semaphore-gke-tutorial.