Kubernetes Jenkins Master-Slave: Solving CI/CD Scalability

Large engineering organisations such as Dell run Jenkins as their core CI/CD tool, executing builds against substantial codebases continuously. A common pain point in traditional Jenkins setups is that build agents (slaves) remain running long after a build completes, wasting compute resources and inflating infrastructure costs. The solution is dynamic scalability—provisioning agents on demand and terminating them the moment they are no longer needed. Running Jenkins on Kubernetes makes this straightforward.

The Jenkins Master-Slave Model

Jenkins uses a master-slave (also called controller-agent) architecture. A single central Jenkins instance—the master—is responsible for scheduling jobs, managing configuration, and distributing work to one or more slaves (agents) that perform the actual build execution.

Key capabilities this model provides:

• Run multiple build tasks in parallel across many agents.
• Automatically replace failed agent instances.
• Spin agents up when a build starts and terminate them immediately after, cutting idle resource consumption.

When Jenkins is deployed on Kubernetes, each agent is a short-lived pod created by the Kubernetes plugin at the moment a job is queued, and deleted as soon as the job finishes. This article walks through the installation of a Jenkins master on Kubernetes, the configuration of dynamic Kubernetes agents, and explains the end-to-end workflow.

Prerequisites

Before you begin you will need:

• A running Kubernetes cluster with kubectl configured.
• A custom Jenkins Dockerfile based on the official Jenkins image.
• A Kubernetes Deployment manifest (jenkins-deployment.yaml).
• A Kubernetes Service manifest (jenkins-service.yaml).
• A PersistentVolume and PersistentVolumeClaim to retain Jenkins configuration across pod restarts.

Installing the Jenkins Master on Kubernetes

Step 1 – Build a Custom Jenkins Docker Image

The preferred approach is to bake all required plugins and build tools into a custom Docker image so that the running container needs zero post-start configuration. Start from the official Jenkins base image and add the plugins your organisation requires, plus any build dependencies such as Maven.

FROM jenkins/jenkins:lts

# Install plugins declaratively so the image is self-contained
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt

# Install Maven as a build dependency
USER root
RUN apt-get update && apt-get install -y maven && rm -rf /var/lib/apt/lists/*
USER jenkinsCode language: PHP (php)

Build and push this image to a container registry accessible by your cluster before proceeding.

Step 2 – Create the Deployment Manifest

Create a jenkins-deployment.yaml file that references your custom image, mounts the PersistentVolumeClaim at /var/jenkins_home, and exposes the standard Jenkins ports (8080 for the UI, 50000 for JNLP agent communication).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: your-registry/custom-jenkins:latest
          ports:
            - containerPort: 8080
            - containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvcCode language: JavaScript (javascript)

Apply the manifest and verify the pod comes up:

kubectl apply -f jenkins-deployment.yaml
kubectl get pods -l app=jenkinsCode language: JavaScript (javascript)

Step 3 – Expose Jenkins with a Service

Create a jenkins-service.yaml to expose the Jenkins UI. Using a NodePort service type is the simplest approach for on-premises clusters; for cloud environments a LoadBalancer or Ingress is preferred.

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  selector:
    app: jenkins
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: jnlp
      port: 50000
      targetPort: 50000

After applying the service, retrieve the assigned NodePort and access the Jenkins dashboard at http://<node-ip>:<node-port>.

Configuring Dynamic Kubernetes Agents (Slaves)

Once the Jenkins UI is accessible, install and configure the Kubernetes plugin to enable dynamic agent provisioning.

Install the Kubernetes Plugin

If the plugin was not baked into your Docker image, install it via Manage Jenkins → Manage Plugins → Available and search for “Kubernetes”.

Add a Kubernetes Cloud

Navigate to Manage Jenkins → Configure System → Cloud → Add a new cloud → Kubernetes and fill in the following:

• Kubernetes URL: the API server endpoint (e.g., https://kubernetes.default when Jenkins runs inside the same cluster).
• Kubernetes Namespace: the namespace where agent pods should be created.
• Jenkins URL: the internal service URL so agents can register back to the master (e.g., http://jenkins:8080).
• Jenkins tunnel: the JNLP address for agent-to-master communication (e.g., jenkins:50000).

Define a Pod Template

Within the Kubernetes cloud section, add a Pod Template. This template defines the Docker image used for each dynamically created agent pod, as well as resource requests, labels, and volume mounts.

• Name / Label: a label such as jenkins-slave that jobs will reference to target this agent type.
• Docker image: use the official jenkins/inbound-agent image (formerly jenkins/jnlp-slave) or a custom derivative.
• Working directory: typically /home/jenkins/agent.

Enable JNLP Agent Protocol

Navigate to Manage Jenkins → Configure Global Security → Agents and ensure the JNLP agent port is fixed at 50000 and that the appropriate inbound agent protocol is enabled. This port must match what is exposed in your Service manifest.

Target the Agent in a Job

Create a Freestyle job (or a Pipeline using the agent { label 'jenkins-slave' } directive) and in the General tab check Restrict where this project can be run. Enter the pod template label you configured (e.g., jenkins-slave). Jenkins will now create a dedicated pod for each build run against this label.

End-to-End Workflow

Understanding the lifecycle of a dynamic agent pod helps with troubleshooting and capacity planning:

1. Job triggered: A build is queued on the Jenkins master. The master detects that the job requires a Kubernetes agent (based on the label).
2. Pod creation: The Kubernetes plugin calls the Kubernetes API to create an agent pod using the configured pod template. The pod runs the JNLP agent image.
3. Agent registration: The agent container starts, connects back to the Jenkins master over port 50000 using the JNLP protocol, and registers itself as an available executor.
4. Build execution: The master assigns the queued build to the newly registered agent. The build runs inside the pod.
5. Pod termination: Once the build completes (successfully or not), the agent reports results back to the master and the pod is automatically deleted. No idle agents linger.

Benefits of This Approach

• Cost efficiency: Agents exist only while a build is running. There are no permanently running VMs or containers consuming resources when idle.
• Elasticity: Kubernetes scales the number of agent pods up or down automatically based on build queue depth, constrained only by available cluster capacity.
• Reproducibility: Every build starts from a clean, immutable container image, eliminating state pollution between builds.
• Simplified operations: The Jenkins master configuration is the single source of truth; no manual agent registration is required.

Summary

Running Jenkins on Kubernetes with dynamically provisioned agent pods is one of the most effective ways to solve the scalability and cost challenges inherent in traditional static master-slave deployments. By baking plugins into a custom Docker image, defining a Kubernetes Deployment and Service for the master, and configuring the Kubernetes plugin with a pod template, you get a CI/CD platform that scales with demand and shrinks back to near-zero when idle.