Continuous Integration and Continuous Delivery (CI/CD) have reshaped how modern software teams build, test, and release applications. Yet, as cloud-native architectures grow more distributed, maintaining parity between what developers intend to deploy and what actually runs inside a cluster becomes challenging. Traditional push-based deployment pipelines often rely on centralized deployment scripts holding high-privilege cluster credentials, creating operational bottlenecks and security risks. GitOps solves these operational challenges by turning your Git repository into the canonical single source of truth for your entire system infrastructure and application runtime. Rather than running deployment scripts that push changes into an environment, a specialized software agent inside the target environment continuously observes the repository and pulls the declared state inward.
For students, beginners, and early-career DevOps engineers, reading about these concepts is only the starting point. Developing true competence requires working through GitOps hands-on CI/CD learning exercises where you can trigger deployments, observe reconciliation loops, fix configuration drift, and perform safe rollbacks in a disposable sandbox. If you are learning through structured platforms such as DevOpsSchool or practicing independently on your local workstation, this guide walks you through seven progressive, practical exercises designed to build real-world confidence.
Understand the GitOps Workflow Before Starting
Before running terminal commands, you need a clear mental model of how GitOps alters the traditional deployment paradigm. At its core, GitOps is an operating model for cloud-native applications that relies on four foundational pillars:
+-------------------------------------------------------------+
| GITOPS ARCHITECTURE |
+-------------------------------------------------------------+
[Developer]
│ (git push)
▼
+──────────────────────────+ +──────────────────────────+
| Git Repository | | CI Pipeline |
| - Source of Truth | ────> | - Linting & Syntax |
| - Declarative Manifests | | - Security & Dry-Run |
+──────────────────────────+ +──────────────────────────+
│ │ (Passes)
│ ▼
│ [Approved Pull Request]
│ │
▼ ▼
+─────────────────────────────────────────────────────────────+
| Kubernetes Cluster |
| |
| +────────────────────────+ Continuous |
| | GitOps Controller | <── Reconciliation Loop |
| | (Argo CD / Flux) | |
| +────────────────────────+ |
| │ |
| ▼ (Applies State) |
| +──────────────────────────────────────────────────────+ |
| | Cluster Runtime (Pods, Services, Ingresses) | |
| +──────────────────────────────────────────────────────+ |
+─────────────────────────────────────────────────────────────+
The Desired-State Model and Declarative Configuration
In traditional imperative computing, you issue sequential commands telling the system how to do something (for example: docker run, systemctl restart, or kubectl scale). If a command fails halfway through, the system is left in an unknown, partially configured state.
GitOps relies strictly on declarative configuration. Instead of listing commands, you write configuration manifests (such as Kubernetes YAML files) that describe what the system should look like:
- “There should be three replicas of the web application running image
v1.2.0.” - “The application must listen on port 8080.”
- “A ClusterIP service must balance traffic across these replicas.”
Git as the Single Source of Truth
In a true GitOps workflow, your running infrastructure and deployed workloads should mirror the state stored in Git. If a resource is not defined in Git, it should not exist in the cluster. If an attribute changes in Git, that change must be reflected in the cluster. Because Git preserves an immutable, cryptographically signed commit log, every change is auditable, reviewable, and reversible.
The Reconciliation Loop
How does the system ensure that runtime reality matches the declaration in Git? Through an active software loop known as the reconciliation loop.
A GitOps controller runs inside the target cluster. It continuously polls or receives webhooks from your Git repository, compares the declared desired state in Git against the live state inside Kubernetes, and calculates any difference (known as divergence or drift). When drift occurs, the controller automatically acts to bring the live cluster back into alignment with the repository.
Separating CI from CD in GitOps
One common beginner misconception is assuming that GitOps eliminates CI. It does not. Instead, GitOps redraws the boundary between Continuous Integration and Continuous Delivery:
- Continuous Integration (CI): Focuses on compilation, unit tests, code linting, packaging container images, running security scans, and validating deployment manifests. CI ends when an artifact is built and a declarative configuration file is validated and committed.
- Continuous Delivery / Continuous Deployment (CD): Managed by the GitOps controller inside the cluster pulling configurations from Git. The CI pipeline never needs administrative credentials to the Kubernetes API.
Crucially, GitOps does not mean running kubectl apply from a developer’s laptop. Direct access to the cluster API is restricted. Every change—without exception—travels through a version-controlled Git commit.
Prepare a Safe Learning Environment
To complete the exercises in this guide, you do not need an expensive cloud account or a multi-node production cluster. You can run every exercise locally on Linux, macOS, or Windows (via WSL2).
+--------------------------------------------------------------+
| LOCAL LAB STACK |
+--------------------------------------------------------------+
| [Local Workstation] |
| |
| - Git CLI (Version Control) |
| - Docker / Podman (Container Runtime) |
| - K3d / Minikube / Kind (Local Kubernetes Cluster) |
| - Kubectl CLI (Kubernetes Client) |
| - GitOps Controller (Argo CD running inside the cluster) |
| - Remote Git Hosting (GitHub / GitLab / Gitea account) |
+--------------------------------------------------------------+
Required Tooling and Component Roles
- Git CLI (
git): Tracks your declarative manifests, creates branches, manages commits, and pushes updates to your remote repository. - Container Engine (Docker or Podman): Provides the virtualization runtime required to spin up lightweight local Kubernetes nodes.
- Local Kubernetes Engine (
k3d,minikube, orkind): Creates disposable, lightweight Kubernetes clusters inside container sandboxes. We will usek3d(which runs lightweight K3s in Docker) due to its minimal resource footprint, butminikubeorkindwork just as well. - Kubernetes CLI (
kubectl): Used to inspect cluster state, view controller logs, and observe running pods. - GitOps Controller (Argo CD): The cluster-side agent that monitors your Git repository and reconciles differences.
- Remote Git Account (GitHub, GitLab, or personal Gitea): Hosts your public or private repository where the controller fetches manifests.
Step-by-Step Environment Provisioning
Step 1: Verify Core Tools
Open your terminal and confirm that the basic utilities are installed:
Bash
git --version
docker --version
kubectl version --client --output=yaml
Step 2: Create a Disposable Local Cluster
Using k3d, create a lightweight cluster named gitops-lab:
Bash
k3d cluster create gitops-lab --servers 1 --agents 1 --port "8080:80@loadbalancer"
Code language: CSS (css)Expected output: The command downloads the K3s container image, establishes internal cluster networking, configures your local ~/.kube/config, and switches your current context to k3d-gitops-lab.
Verify cluster communication:
Bash
kubectl get nodes
Code language: JavaScript (javascript)You should see two nodes (k3d-gitops-lab-server-0 and k3d-gitops-lab-agent-0) reporting a Ready status.
Step 3: Establish Safe Operational Boundaries
- Never commit credentials: Do not place API keys, private certificates, or plain-text passwords into your Git repositories.
- Keep the environment disposable: Treat this local cluster as temporary. If an exercise becomes misconfigured, delete the cluster (
k3d cluster delete gitops-lab) and recreate it within sixty seconds.
Exercise One: Store an Application’s Desired State in Git
Goal: Structure a declarative Kubernetes deployment repository and store an application’s desired state under version control.
Prerequisites: Git installed, active account on GitHub/GitLab, terminal access.
In GitOps, repository structure matters. Mixing raw application source code (Java, Go, Python) with operational manifests (Kubernetes YAML) can lead to messy CI loops. A clean pattern for beginners is maintaining an environment/manifest repository dedicated solely to operational state.
gitops-manifests/
├── README.md
└── apps/
└── web-app/
├── deployment.yaml
└── service.yaml
Hands-on Steps
1. Initialize the Manifest Repository
Create a new directory on your machine, initialize Git, and build the directory skeleton:
Bash
mkdir -p gitops-manifests/apps/web-app
cd gitops-manifests
git init -b main
2. Define the Declarative Deployment Manifest
Inside apps/web-app/deployment.yaml, define a deployment using a lightweight, verifiable web server container:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-web-app
namespace: default
labels:
app: sample-web-app
spec:
replicas: 2
selector:
matchLabels:
app: sample-web-app
template:
metadata:
labels:
app: sample-web-app
spec:
containers:
- name: web-server
image: nginxinc/nginx-unprivileged:1.25-alpine
ports:
- containerPort: 8080
resources:
limits:
memory: "128Mi"
cpu: "250m"
requests:
memory: "64Mi"
cpu: "100m"
Code language: JavaScript (javascript)3. Define the Accompanying Service Manifest
Inside apps/web-app/service.yaml, declare how network traffic routes to the pods:
YAML
apiVersion: v1
kind: Service
metadata:
name: sample-web-service
namespace: default
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: sample-web-app
Code language: JavaScript (javascript)4. Review and Commit the Configuration
Before staging, review your files for syntax errors or unintended entries:
Bash
git status
git diff
Stage the files and create your baseline commit:
Bash
git add apps/
git commit -m "feat: add initial manifests for sample-web-app"
Code language: JavaScript (javascript)5. Push to a Remote Git Repository
Create an empty public repository named gitops-manifests on your Git hosting provider (e.g., GitHub), link the remote, and push:
Bash
# Replace <YOUR-USERNAME> with your actual account name
git remote add origin https://github.com/<YOUR-USERNAME>/gitops-manifests.git
git push -u origin main
Code language: PHP (php)Verification Method
Visit your remote repository in a web browser. Confirm that:
- The directory structure
apps/web-app/exists. - Both
deployment.yamlandservice.yamlare present with clean formatting. - The commit history shows your initial commit message.
Common Errors and Troubleshooting
- Error: Git rejects push with
Support for password authentication was removed.Fix: Configure an SSH key or create a fine-grained Personal Access Token (PAT) with repository read/write scopes. - Error: Indentation syntax errors in YAML.Fix: Ensure you use two spaces per indentation level. Never use tabs in Kubernetes manifests.
Extension Challenge
Add a namespace.yaml file declaring a custom namespace named web-apps, and update the metadata.namespace field in both deployment.yaml and service.yaml to use it.
Exercise Two: Make a Change Through a Pull Request
Goal: Modify an application’s desired state using a feature branch and pull request (PR) workflow, simulating team-based peer review.
Prerequisites: Completion of Exercise 1, remote repository pushed.
Directly pushing changes to your production main branch defeats the governance advantages of GitOps. The pull request creates an auditable review boundary where human operators and automated tools can inspect configuration changes before they affect running infrastructure.
+-------------------------------------------------------------+
| PULL REQUEST WORKFLOW |
+-------------------------------------------------------------+
[main branch] ───(Branch)───> [feat/scale-replicas]
│
(Edit manifests)
│
(git commit & push)
│
▼
[Pull Request Opened]
├── Automated CI Checks
└── Peer Code Review
│
(Approved)
▼
[main branch] <───(Merge)─────────────┘
Hands-on Steps
1. Create and Switch to a Feature Branch
To handle increased traffic, assume we need to scale our web application from two replicas to four:
Bash
git checkout -b feat/scale-replicas
2. Modify the Desired State
Open apps/web-app/deployment.yaml in your editor. Locate the replicas key and update it:
YAML
spec:
replicas: 4
3. Audit Changes Locally
Inspect the unified diff to confirm that only the intended line changed:
Bash
git diff apps/web-app/deployment.yaml
Expected output:
Diff
@@ -7,3 +7,3 @@ spec:
- replicas: 2
+ replicas: 4
selector:
Code language: CSS (css)4. Commit and Push the Branch
Bash
git add apps/web-app/deployment.yaml
git commit -m "chore(web-app): scale replicas from 2 to 4"
git push -u origin feat/scale-replicas
Code language: JavaScript (javascript)5. Open and Review the Pull Request
- Navigate to your remote repository interface and open a new Pull Request comparing
feat/scale-replicasagainstmain. - Title the PR:
Scale sample-web-app replicas to 4. - Use the following review checklist in the PR description:
- [x] Target namespace verified
- [x] Replica count aligns with resource quotas
- [x] Pod resource limits remain unchanged
- [x] No sensitive values exposed in manifests
6. Merge the Change
Approve and execute a standard merge into your main branch. Delete the feature branch on the remote host after merging.
Verification Method
Check out your local main branch, pull the latest changes, and confirm the log:
Bash
git checkout main
git pull origin main
git log -n 1 --oneline
The output should display your merge commit reflecting four declared replicas.
Common Errors and Troubleshooting
- Error: Merge conflict when opening the PR.Fix: Your feature branch has diverged from
main. Rungit checkout feat/scale-replicas, rebase againstmain(git rebase main), resolve conflicting lines, and push with--force-with-lease.
Extension Challenge
Update the deployment.yaml on a new branch to add an environment variable (ENV: "staging") using a Kubernetes env block, open a PR, review the diff, and merge it.
Exercise Three: Connect a GitOps Controller
Goal: Install Argo CD into your local Kubernetes cluster and configure it to continuously synchronize your Git repository with the cluster state.
Prerequisites: Running local cluster from Section 2, public Git repository with manifests from Exercise 1.
The GitOps controller acts as the automation engine. It monitors your Git repository at regular intervals, evaluates the live objects in the cluster, and applies the necessary changes to eliminate differences.
+───────────────────────────+
| Remote Git Repository |
| (Desired State: 4 Pods) |
+───────────────────────────+
▲
│ Polling (Every 3m or Webhook)
│
+─────────────────────────────────────────────────────────────+
| K3d Cluster |
| |
| +───────────────────────────────────────────────────────+ |
| | Argo CD Namespace (argocd) | |
| | | |
| | [Repo Server] <──> [Application Controller] | |
| | │ | |
| +───────────────────────────┼───────────────────────────+ |
| │ |
| ▼ Compares & Reconciles |
| +───────────────────────────────────────────────────────+ |
| | Default Namespace | |
| | [Pod 1] [Pod 2] [Pod 3] [Pod 4] | |
| +───────────────────────────────────────────────────────+ |
+─────────────────────────────────────────────────────────────+
Code language: HTML, XML (xml)Hands-on Steps
1. Install Argo CD Components
Create the argocd system namespace and install the official Argo CD manifests:
Bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.4/manifests/install.yaml
Code language: JavaScript (javascript)2. Monitor Controller Readiness
Wait for all controller pods to transition into a Running state:
Bash
kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=180s
kubectl get pods -n argocd
Code language: JavaScript (javascript)3. Expose the Argo CD Interface (Optional but Helpful)
You can port-forward the web dashboard to your local browser:
Bash
kubectl port-forward svc/argocd-server -n argocd 8081:443 &
Note: The default administrative username is admin. Retrieve the auto-generated initial password:
Bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Code language: PHP (php)4. Declare an Argo CD Application Resource
In true GitOps fashion, we do not configure applications by clicking buttons in a UI. Instead, we declare the Argo CD application using a custom Kubernetes manifest.
Create a file named application.yaml on your workstation (outside the apps/ directory, or in a root directory):
YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sample-web-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/<YOUR-USERNAME>/gitops-manifests.git
targetRevision: HEAD
path: apps/web-app
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Code language: JavaScript (javascript)Important configuration fields:
repoURL: Points to your remote Git repository.path: Tells the controller which subdirectory contains the relevant manifests.targetRevision: HEAD: Instructs the controller to track the latest commit on the default branch.selfHeal: true: Enables automatic correction if someone alters the cluster manually.prune: true: Tells the controller to delete cluster objects if their manifests are deleted from Git.
5. Apply the Application Manifest
Replace <YOUR-USERNAME> with your GitHub username, then apply the manifest:
Bash
kubectl apply -f application.yaml
Code language: CSS (css)Verification Method
Inspect the live status using kubectl:
Bash
kubectl get applications.argoproj.io -n argocd
Code language: CSS (css)The status should transition to Healthy and Synced.
Next, verify that the application pods exist in the default namespace:
Bash
kubectl get deployments
kubectl get pods -l app=sample-web-app
Code language: JavaScript (javascript)You should see 4 running pods matching the desired state merged in Exercise 2.
Common Errors and Troubleshooting
- Error: Application status shows
ComparisonErrorwithrepository not found.Fix: Verify therepoURLspelling. If the repository is private, you must provide Git credentials to Argo CD via a Kubernetes secret. - Error: Pods remain in
CrashLoopBackOff.Fix: Runkubectl describe pod -l app=sample-web-appandkubectl logs -l app=sample-web-app. Verify that your container port is non-privileged (e.g., 8080) when running unprivileged container images.
Extension Challenge
Change the targetRevision from HEAD to point to a specific release tag (e.g., v1.0.0). Tag your commit in Git, push it, and observe how the controller pins synchronization to that immutable tag.
Exercise Four: Observe and Fix Configuration Drift
Goal: Manually mutate cluster resources, observe how drift occurs, and watch the GitOps controller reconcile the cluster back to the declared state.
Prerequisites: Completion of Exercise 3, active Argo CD controller running with selfHeal configured.
Configuration drift occurs when an environment’s real runtime state diverges from its formal declaration. In traditional environments, engineers often make quick, unrecorded hotfixes using direct terminal access during emergencies. Over time, nobody knows what is running or why. GitOps systematically eliminates this problem.
+--------------------------------------------------------------+
| CONFIGURATION DRIFT & HEALING |
+--------------------------------------------------------------+
1. State in Git: 2. Unauthorized Cluster Drift:
replicas: 4 replicas: 1 (via manual kubectl)
│ │
│ ▼
│ [Cluster Pods Terminated]
│ │
│ 3. GitOps Controller Detection:
└───────────────> Divergence Detected!
(Git != Live)
│
▼
4. Automated Self-Healing Triggered:
Controller overwrites cluster state
to match Git.
│
▼
[Replicas restored to 4]
Code language: PHP (php)Hands-on Steps
1. Inspect the Baseline Cluster State
Confirm that your web application is healthy with 4 active replicas:
Bash
kubectl get deployment sample-web-app
Code language: JavaScript (javascript)2. Introduce Manual Configuration Drift
Simulate an out-of-band manual intervention. Bypass Git entirely and directly alter the live deployment using kubectl:
Bash
kubectl scale deployment sample-web-app --replicas=1
3. Observe the Cluster Reaction Immediately
List the pods immediately after issuing the command:
Bash
kubectl get pods -l app=sample-web-app
Code language: JavaScript (javascript)You will notice Kubernetes beginning to terminate three of the four pods.
4. Observe the Controller’s Reconciliation Loop
Because our application.yaml was configured with selfHeal: true, the Argo CD reconciliation loop detects that the live state (replicas: 1) does not match the desired state in Git (replicas: 4).
Within seconds to a few minutes (depending on polling cycles), inspect the pods again:
Bash
kubectl get pods -l app=sample-web-app
Code language: JavaScript (javascript)Argo CD detects the drift, overwrites the manual scale-down command, and scales the deployment back up to 4 pods.
5. Safe Demonstration of Manual Drift Without Auto-Healing
To see what happens when automated healing is turned off:
- Temporarily disable self-heal:Bash
kubectl patch application sample-web-app -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"selfHeal":false}}}}' - Manually mutate a label or container image:Bash
kubectl set image deployment/sample-web-app web-server=nginx:latest - Check the application status in Argo CD:Bash
kubectl get application sample-web-app -n argocd -o jsonpath='{.status.sync.status}'Expected output:OutOfSync. The controller highlights the difference without overwriting it, awaiting approval or manual synchronization. - Restore automated healing:Bash
kubectl patch application sample-web-app -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"selfHeal":true}}}}'Notice that the image immediately reverts back tonginxinc/nginx-unprivileged:1.25-alpine.
Verification Method
Review the Argo CD controller logs to observe the reconciliation event being logged:
Bash
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=50 | grep "sample-web-app"
Code language: JavaScript (javascript)Look for log entries indicating that synchronization was initiated due to cluster resource modification.
Common Errors and Troubleshooting
- Symptom: The manual change persists and the controller does not reconcile.Fix: Confirm whether
selfHealis set totrueundersyncPolicy.automated. If omitted, Argo CD marks the app asOutOfSyncbut will not alter running pods automatically.
Extension Challenge
Simulate an accidental resource deletion by running kubectl delete service sample-web-service. Observe how quickly the controller recreates the deleted service manifest from Git.
Exercise Five: Add CI Validation Before Deployment
Goal: Implement a Continuous Integration (CI) pre-merge pipeline that validates manifest syntax and tests policies before changes can be merged into main.
Prerequisites: Remote GitHub repository with manifests, basic familiarity with GitHub Actions.
A GitOps controller blindly applies whatever valid YAML lands in the tracked branch. If someone merges a manifest with invalid indentation, missing mandatory fields, or dangerous security parameters, the controller will fail during sync or deploy an insecure configuration.
To prevent this, we introduce pre-commit and pre-merge CI validation.
+-------------------------------------------------------------+
| CI VALIDATION GATES |
+-------------------------------------------------------------+
[Pull Request Submitted]
│
▼
┌───────────────────────────────────────────────┐
│ 1. YAML Syntax & Linting (yamllint) │
└──────────────────────┬────────────────────────┘
▼
┌───────────────────────────────────────────────┐
│ 2. Schema Validation (kubeconform) │
└──────────────────────┬────────────────────────┘
▼
┌───────────────────────────────────────────────┐
│ 3. Dry-Run Policy / Security Scan (kube-score)│
└──────────────────────┬────────────────────────┘
▼
[All Checks Pass?]
│ │
(No) │ │ (Yes)
▼ ▼
[Block PR] [Allow Merge to Main]
│
▼
[GitOps Sync]
The Boundary Between CI and GitOps CD
- CI Responsibilities: Run tests, execute linters, validate schemas, ensure security compliance, and return pass/fail statuses on PRs. CI never needs cluster access.
- GitOps CD Responsibilities: Read clean, validated declarations from merged branches and apply them to the cluster runtime.
Hands-on Steps
1. Define the Local Validation Workflow
Create a workflow directory in your Git repository:
Bash
mkdir -p .github/workflows
2. Create the GitHub Actions CI Pipeline
Create .github/workflows/validate-manifests.yaml:
YAML
name: Validate Kubernetes Manifests
on:
pull_request:
branches:
- main
paths:
- 'apps/**'
jobs:
lint-and-validate:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python for YAML Linting
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Run yamllint
run: |
pip install yamllint
yamllint -d "{extends: relaxed, rules: {line-length: {max: 120}}}" apps/
- name: Download Kubeconform for Kubernetes Schema Validation
run: |
curl -sL https://github.com/yannh/kubeconform/releases/download/v0.6.4/kubeconform-linux-amd64.tar.gz | tar xz
sudo mv kubeconform /usr/local/bin/
- name: Validate Manifests Against Kubernetes OpenAPI Schemas
run: |
kubeconform -strict -summary -kubernetes-version 1.28.0 apps/
Code language: JavaScript (javascript)3. Test the Validation Mechanism Locally
You can test the same commands on your local workstation before committing:
Bash
# Example if yamllint is installed locally:
yamllint -d "{extends: relaxed, rules: {line-length: {max: 120}}}" apps/
Code language: PHP (php)4. Intentionally Trigger a Pipeline Failure
- Create a test branch:Bash
git checkout -b test/invalid-manifest - Introduce an intentional schema error into
apps/web-app/deployment.yaml. For example, changereplicas: 4to a string value:YAMLreplicas: "four" - Commit and push:Bash
git add apps/web-app/deployment.yaml git commit -m "test: invalid replica data type" git push -u origin test/invalid-manifest - Open a Pull Request targeting
main.
Verification Method
Watch the GitHub Actions tab in your repository. The pipeline will fail on the Validate Manifests Against Kubernetes OpenAPI Schemas step with an error message indicating that spec.replicas expects an integer, not a string.
Because the CI check fails, branch protection rules can block the PR from being merged, protecting your cluster from malformed configurations.
Bash
# Clean up your test branch
git checkout main
git branch -D test/invalid-manifest
Code language: PHP (php)Common Errors and Troubleshooting
- Error:
kubeconformreportscould not find schema for custom resource.Fix: By default, schema validators validate native Kubernetes types (Deployments, Services). For Custom Resource Definitions (CRDs) like Argo CDApplicationmanifests, tellkubeconformto ignore unknown CRDs using the-ignore-missing-schemasflag.
Extension Challenge
Add a security scanning step to the workflow using trivy config apps/ to alert developers if a container runs as root or lacks memory limits.
Exercise Six: Promote a Change Across Environments
Goal: Structure multi-environment configurations using Kustomize overlays, promoting changes from development to staging without duplicating manifest files.
Prerequisites: Completion of earlier exercises, local k3d cluster running.
In real-world engineering, you rarely deploy directly to production. A change must progress through development, staging, and production. A frequent beginner mistake is copying and pasting YAML across directories (dev/, staging/, prod/). This produces configuration drift between environments when someone updates a setting in one folder and forgets the others.
The recommended GitOps solution is using a configuration tool like Kustomize (built directly into kubectl) to create a shared base with environment-specific overlays.
apps/web-app/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
└── overlays/
├── dev/
│ ├── kustomization.yaml
│ └── replica_patch.yaml
└── staging/
├── kustomization.yaml
└── replica_patch.yaml
Hands-on Steps
1. Reorganize Your Manifests into Base and Overlays
Restructure your local repository:
Bash
cd gitops-manifests
mkdir -p apps/web-app/base
mkdir -p apps/web-app/overlays/dev
mkdir -p apps/web-app/overlays/staging
# Move original manifests into base
mv apps/web-app/deployment.yaml apps/web-app/base/
mv apps/web-app/service.yaml apps/web-app/base/
Code language: PHP (php)2. Create the Base Kustomization
Create apps/web-app/base/kustomization.yaml:
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
Code language: HTTP (http)3. Define the Development Overlay
In development, we want a lightweight single pod with a dev- name prefix.
Create apps/web-app/overlays/dev/kustomization.yaml:
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev-environment
namePrefix: dev-
resources:
- ../../base
patches:
- path: replica_patch.yaml
Code language: HTTP (http)Create apps/web-app/overlays/dev/replica_patch.yaml:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-web-app
spec:
replicas: 1
4. Define the Staging Overlay
In staging, we want two replicas and higher resource allocations.
Create apps/web-app/overlays/staging/kustomization.yaml:
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: staging-environment
namePrefix: staging-
resources:
- ../../base
patches:
- path: replica_patch.yaml
Code language: HTTP (http)Create apps/web-app/overlays/staging/replica_patch.yaml:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-web-app
spec:
replicas: 2
5. Verify Overlays Locally
Use kubectl kustomize to render the manifests locally without applying them:
Bash
# Test the Dev configuration rendering:
kubectl kustomize apps/web-app/overlays/dev
# Test the Staging configuration rendering:
kubectl kustomize apps/web-app/overlays/staging
Code language: PHP (php)Notice how both environments inherit the common container specifications from base, while customizing namespaces, name prefixes, and replica counts without duplicating boilerplate code.
6. Commit and Push Multi-Environment Architecture
Bash
git add apps/web-app/
git commit -m "refactor(web-app): convert architecture to Kustomize base and overlays"
git push origin main
Code language: JavaScript (javascript)7. Update Argo CD Applications
You can now create separate Argo CD applications pointing to apps/web-app/overlays/dev and apps/web-app/overlays/staging. Promoting an image or configuration change simply means updating the version tag in base/ or altering an overlay patch via a reviewed Pull Request.
Verification Method
Inspect the live namespaces to confirm clean separation:
Bash
kubectl create namespace dev-environment
kubectl create namespace staging-environment
kubectl apply -k apps/web-app/overlays/dev
kubectl apply -k apps/web-app/overlays/staging
kubectl get pods -n dev-environment
kubectl get pods -n staging-environment
Code language: JavaScript (javascript)You should see one pod prefixed with dev- in dev-environment, and two pods prefixed with staging- in staging-environment.
Common Errors and Troubleshooting
- Error:
kustomizefails withaccumulating resources: ... no such file or directory.Fix: Check relative file paths insidekustomization.yaml. Ensure that../../basecorrectly navigates up two directory levels to reach the base folder.
Extension Challenge
Add a common label (environment: dev and environment: staging) automatically across all generated resources using Kustomize’s commonLabels directive.
Exercise Seven: Roll Back a Failed Change
Goal: Diagnose a bad release, execute a clean rollback using native Git operations, and verify automated cluster recovery.
Prerequisites: Completion of Exercise 3, working GitOps deployment.
In traditional environments, a failed deployment often causes panic. Engineers scramble to run imperative commands like helm rollback or kubectl rollout undo. In a strict GitOps model, running kubectl rollout undo directly against the cluster introduces immediate configuration drift—the cluster would run the old version while Git still declared the broken one. The controller would promptly re-deploy the broken version!
In GitOps, every rollback is simply a forward Git commit that reverts the repository to a known good state.
+--------------------------------------------------------------+
| THE GITOPS ROLLBACK FLOW |
+--------------------------------------------------------------+
Commit 1: (Good State) ──> Image: nginx:1.25 [Cluster Healthy]
│
Commit 2: (Bad Release) ──> Image: broken:9.9 [CrashLoopBackOff]
│
│ DO NOT run "kubectl rollout undo"!
│ Run "git revert HEAD"
▼
Commit 3: (Revert Commit) ─> Image: nginx:1.25 [Cluster Restored]
Code language: PHP (php)Hands-on Steps
1. Intentionally Introduce a Broken Release
Create a feature branch and point your container image to a non-existent tag:
Bash
git checkout -b fix/image-update
Edit your deployment manifest (or your base manifest) and update the image:
YAML
containers:
- name: web-server
image: nginxinc/nginx-unprivileged:non-existent-tag-999
2. Commit, Push, and Merge
Bash
git add apps/
git commit -m "feat(web-app): update web server to non-existent tag"
git push origin fix/image-update
Code language: JavaScript (javascript)Merge this branch into main via PR or direct merge for this lab exercise:
Bash
git checkout main
git merge fix/image-update
git push origin main
3. Observe the Failure in the Cluster
Wait for Argo CD to pick up the change, or trigger a refresh:
Bash
kubectl get pods -l app=sample-web-app
Code language: JavaScript (javascript)You will observe pods stuck in ImagePullBackOff or ErrImagePull:
NAME READY STATUS RESTARTS AGE
sample-web-app-798889989-abcde 0/1 ImagePullBackOff 0 45s
4. Execute the Rollback Using Git
Rather than touching kubectl, identify the commit that introduced the breaking change using the Git log:
Bash
git log --oneline -n 3
Create a revert commit that undoes the changes introduced by the broken commit:
Bash
git revert HEAD --no-edit
This generates a new commit that restores the exact file state prior to the broken release.
5. Push the Revert Commit to the Repository
Bash
git push origin main
Verification Method
Watch your cluster as Argo CD pulls the new revert commit:
Bash
kubectl get pods -l app=sample-web-app -w
Code language: JavaScript (javascript)Watch the old, broken pods terminate while new pods running the stable, previously working image (1.25-alpine) start cleanly and reach Running status.
Check the deployment history:
Bash
kubectl describe deployment sample-web-app | grep Image:
The image has safely returned to the working tag. The Git history records the failure, the revert, and the author who executed the rollback, maintaining a transparent audit trail.
Common Errors and Troubleshooting
- Symptom: The cluster continues trying to pull the broken image after pushing the revert commit.Fix: By default, Argo CD polls Git every three minutes. You can trigger an instant sync by running
argocd app sync sample-web-appor pressing the “Sync” button in the Argo CD web console.
Extension Challenge
Simulate an out-of-memory crash by setting container memory limits to an impossibly low value (1Mi). Commit the change, observe the OOMKilled status in Kubernetes, and use git revert to recover the service.
Common GitOps Learning Mistakes
When adopting GitOps, beginners often encounter several anti-patterns. Understanding these early prevents architectural debt and operational headaches.
+───────────────────────────────────+───────────────────────────────────+
| Common Anti-Pattern | Recommended GitOps Solution |
+───────────────────────────────────+───────────────────────────────────+
| Storing plain-text secrets in Git | Use external secret operators or |
| | sealed secrets with encryption |
+───────────────────────────────────+───────────────────────────────────+
| Treating CI as the deployment | Restrict CI to validation/build; |
| agent with direct cluster access | let in-cluster controllers deploy |
+───────────────────────────────────+───────────────────────────────────+
| Bypassing Git during incidents | Implement audited break-glass |
| with manual kubectl mutations | workflows with auto-healing |
+───────────────────────────────────+───────────────────────────────────+
| Monolithic repo mixing app source | Separate application source code |
| code with environment manifests | from deployment manifest repos |
+───────────────────────────────────+───────────────────────────────────+
Code language: JavaScript (javascript)1. Committing Plain-Text Secrets to Git
- The Mistake: Storing API tokens, database passwords, or private keys directly in Git repositories because “it needs to be declarative.”
- Why it’s dangerous: Git histories are permanent. Even private repositories can be cloned, leaked, or exposed.
- The Solution: Keep plain-text secrets out of Git. Use solutions like External Secrets Operator (which pulls secrets from managed vaults at runtime) or Sealed Secrets (which encrypts secrets using asymmetric public-key cryptography so they can be stored safely in public Git repositories).
2. Granting Excessive Controller Permissions
- The Mistake: Giving the GitOps controller unrestricted cluster-admin privileges across all namespaces.
- Why it’s dangerous: If a misconfigured or malicious manifest is merged into Git, the controller can overwrite critical cluster-wide resources, including RBAC rules and network policies.
- The Solution: Scope the controller’s ServiceAccount permissions using Kubernetes Role-Based Access Control (RBAC). Restrict the controller to managing only the specific namespaces dedicated to application workloads.
3. Confusing CI with Reconciliation
- The Mistake: Triggering a webhook that runs
kubectl applyfrom inside a Jenkins, GitLab CI, or GitHub Actions runner. - Why it’s dangerous: This is traditional “push-based” deployment, not GitOps. It requires storing sensitive cluster credentials inside your CI tool and fails to detect or correct configuration drift if someone modifies the cluster out of band.
- The Solution: Let CI build artifacts and validate YAML. Let the in-cluster GitOps controller pull declarations and manage reconciliation.
4. Bypassing Review During Emergencies
- The Mistake: During an outage, operators run direct imperative commands (
kubectl edit,kubectl delete) to fix an issue quickly, intending to “update Git later.” - Why it’s dangerous: The changes are almost never backported to Git. The next automated reconciliation run will overwrite the emergency fix, causing the outage to reoccur.
- The Solution: If you must use a break-glass emergency procedure, do it through a dedicated emergency PR with expedited review, or document the exact commit required immediately following cluster intervention.
5. Failing to Verify the Deployed State
- The Mistake: Assuming that because
git commitandgit pushsucceeded, the application is running successfully. - Why it’s dangerous: Git only knows that text was saved. It does not know if your pod crashed, failed a readiness probe, or ran out of disk space.
- The Solution: Implement automated health checks, configure alerts for unsynced or degraded applications in your GitOps controller, and verify pod statuses after rollouts.
11. Practice Project and Self-Assessment
Now that you have worked through each individual phase of the GitOps lifecycle, test your understanding by completing an end-to-end integration project independently.
+--------------------------------------------------------------+
| PRACTICE PROJECT INTEGRATION MAP |
+--------------------------------------------------------------+
[Microservice Manifests]
│
▼
[GitHub Actions CI Gate] ──(Fails?)──> [Halt & Notify]
│ (Passes)
▼
[Pull Request Approved & Merged]
│
▼
[Argo CD Reconciliation] ──> [Deploys to "production" Namespace]
│
├── Test 1: Manual Drift Injection ──> [Auto-Healed?]
└── Test 2: Bad Config Injection ──> [Revert via Git]
Code language: PHP (php)The Challenge Scenario
You are tasked with deploying a microservice named order-service across your local cluster. Your implementation must satisfy the following operational requirements:
- Repository Layout: Create a standalone manifest repository configured with a Kustomize structure (
base/and an overlay forproduction/). - Automated CI Gate: Configure a GitHub Actions workflow that automatically runs
yamllintandkubeconformon every pull request targetingmain. - Branch Protection: Open a PR to introduce the
order-servicemanifests. Require the CI workflow to pass before merging. - Declarative GitOps Delivery: Declare an Argo CD
Applicationthat monitors theproduction/overlay and deploys into a custom namespace namedproduction. - Drift Resilience: Manually edit the live production deployment to change its environment variables. Confirm that Argo CD detects the divergence and automatically overwrites it within three minutes.
- Simulated Incident and Rollback: Push an invalid container port or broken configuration commit to
main. Observe the service enter a degraded state. Execute a clean rollback usinggit revert, push to remote, and observe the application return to a healthy state without usingkubectl.
Completion Checklist
Use this checklist to evaluate your implementation:
Manifests are organized into reusable base/ and overlay/ directories.
No plain-text passwords or secret keys exist in the Git commit history.
CI workflow executes successfully on Pull Requests, blocking malformed YAML.
Argo CD Application is declared using a version-controlled YAML manifest.
Application status in Argo CD reports "Healthy" and "Synced".
Manual cluster mutations trigger an automated self-healing reconciliation.
Rollbacks are executed exclusively through Git commits (git revert).
Code language: PHP (php)Reflection Questions
- Why does the pull-based model of a GitOps controller improve security compared to a push-based CI pipeline?
- If an application requires dynamic database credentials that rotate every 24 hours, how should this be handled within a GitOps workflow without violating the declarative principle?
- What are the operational tradeoffs between enabling automatic self-healing (
selfHeal: true) versus leaving it in manual notification mode?
Frequently Asked Questions (FAQs)
What is GitOps in a CI/CD workflow?
In a modern CI/CD workflow, GitOps is an operating model where Git serves as the single source of truth for declarative infrastructure and application configurations. While Continuous Integration (CI) continues to test code, build container images, and validate manifests, the Continuous Delivery (CD) aspect is handled by an automated controller running inside the target environment. The controller continuously pulls state declarations from Git and reconciles differences, replacing push-based deployment scripts with continuous reconciliation.
How is GitOps different from traditional deployment pipelines?
Traditional pipelines use a push-based model: an external CI/CD server (such as Jenkins or GitHub Actions) holds administrative access credentials to the target cluster and runs deployment commands (kubectl apply, helm install) outward. GitOps uses an in-cluster pull model: an internal agent monitors the repository and synchronizes state inward. This eliminates the need to expose cluster API credentials to external systems, ensures continuous drift detection, and keeps Git as an exact mirror of the running environment.
Do students need a cloud account to practice GitOps?
No. You do not need accounts with AWS, Azure, or Google Cloud to master GitOps concepts. Lightweight local Kubernetes distributions like k3d, minikube, or kind can run directly on a personal laptop inside Docker or Podman. These tools provide fully functional Kubernetes environments that support controllers like Argo CD and Flux, allowing you to practice realistic production workflows locally without infrastructure costs.
What is the role of a GitOps controller?
A GitOps controller (such as Argo CD or Flux) is a cluster-side software agent responsible for the continuous reconciliation loop. It performs three primary duties:
- Observing: Periodically reading declared manifests from a designated Git repository.
- Comparing: Inspecting the live state of cluster resources and identifying divergence or drift.
- Synchronizing: Automatically applying changes to bring the live cluster state into alignment with the repository.
How does GitOps help detect configuration drift?
Because the GitOps controller continuously evaluates both the desired state in Git and the actual runtime state in the cluster, any manual, unrecorded modification (such as someone running kubectl scale or kubectl edit directly on the cluster) creates immediate divergence. The controller flags this divergence as an OutOfSync state. If configured with automated self-healing, it will immediately overwrite the manual modification, restoring the cluster to the version declared in Git.
How can a team roll back a GitOps deployment?
In GitOps, you roll back by changing the repository, not by running imperative rollback commands against the cluster. To revert a failed deployment, an engineer runs git revert <commit-hash> locally and pushes the revert commit to the tracked branch (ideally via a quick pull request). The GitOps controller detects the updated commit, compares it to the running cluster, and gracefully transitions the workloads back to the previous, working state while preserving a transparent, auditable history of the change.
Conclusion
GitOps replaces error-prone imperative scripts and unrecorded cluster tweaks with a disciplined, auditable workflow built on declarative manifests, automated CI validation, and continuous reconciliation. As demonstrated across these exercises, the true power of GitOps lies in enforcing that every operational action—from a routine replica scale-up to an urgent production rollback—travels strictly through version-controlled commits. Continuing to experiment with these hands-on patterns, whether in your own local sandbox or through structured training platforms like DevOpsSchool, is the fastest way to build the practical confidence and muscle memory required to manage production Kubernetes environments safely and reliably.