This article is based on a post originally sourced from TechRepublic. The original publication date was April 2020.
Kubernetes has become the dominant container orchestration platform for good reason: it can scale applications elastically in response to demand. But that capability does not come out of the box. Kubernetes requires deliberate configuration at multiple layers before it will scale reliably under production load. Miss one layer and you risk overloaded nodes, wasted cloud spend, or cascading failures across environments.
This article walks through three distinct scaling dimensions in Kubernetes: application-level scaling, cluster-level scaling, and managing multiple clusters. Each layer builds on the previous one, and each introduces its own operational complexity.
Factor 1: Application Scaling
Define Resource Requests and Limits
The foundation of application scaling in Kubernetes is making your pods resource-aware. By default, if you do not specify CPU and memory requests in your pod spec, the Kubernetes scheduler treats the resource needs as zero. This means it may schedule far too many pods onto a single node, overloading it and causing instability.
Before you can set accurate resource requests, you need to profile the application in a realistic environment. Observe actual CPU and memory consumption under normal and peak load, then encode those observations into your pod spec:
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"Code language: JavaScript (javascript)The requests values tell the scheduler how much capacity to reserve on a node. The limits values cap what the container can actually consume. Setting these correctly is the single most impactful thing you can do to make application scaling predictable.
Horizontal Pod Autoscaler (HPA)
Once resource requests are defined, you can use the Horizontal Pod Autoscaler (HPA) to automatically scale the number of pod replicas in response to observed metrics. The HPA can act on CPU utilization, memory utilization, or custom metrics exposed through the Kubernetes Metrics API.
A basic HPA targeting CPU utilization looks like this:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: my-app-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60Running at least two replicas of a pod also enables rolling upgrades: Kubernetes can update pods one at a time while keeping the application available, allowing you to maintain uptime close to 100% through deployments. The official Kubernetes documentation on scaling deployments covers manual scaling as a starting point before you introduce autoscaling.
Factor 2: Cluster Scaling
Application scaling handles the distribution of workload across the pods and nodes that already exist in your cluster. But what happens when the cluster itself runs out of capacity? Kubernetes has no built-in mechanism to provision new virtual machines—it is only aware of the resources already registered with it.
Cluster Autoscaler
The answer is the Cluster Autoscaler, a separate component that monitors the cluster for pending pods that cannot be scheduled due to insufficient node capacity. When it detects this condition, it requests additional nodes from the underlying infrastructure provider—a public cloud API, a private cloud like OpenStack, or a virtualization platform. When nodes are underutilized for a configurable period, the Cluster Autoscaler can also scale them back down to reduce cost.
All major cloud providers—Amazon Web Services, Microsoft Azure, Google Cloud, and IBM Cloud—offer native integration between their managed Kubernetes services and their node auto-provisioning APIs. The open-source kubernetes/autoscaler project supports multiple cloud backends if you are not using a fully managed service.
OpenShift and Hybrid Cloud Scaling
Red Hat OpenShift is a Kubernetes distribution that includes built-in machine management and autoscaling capabilities through its Machine API. OpenShift can manage resources across on-premises infrastructure and public cloud environments simultaneously, making it a practical option for organisations with hybrid-cloud requirements who want cluster autoscaling without assembling the components themselves.
Factor 3: Managing Multiple Clusters
Once you operate Kubernetes in a serious production context, a single cluster is rarely enough. The need for multiple clusters emerges from several independent pressures:
- Environment isolation: Separate development, staging, and production clusters prevent a load test from competing with or destabilising production workloads.
- High availability: Distributing workloads across clusters in different availability zones or regions protects against zone-level outages.
- Geographic distribution: Routing users to a cluster in their region reduces latency and may satisfy data-residency regulations.
- Organisational boundaries: Different business units or teams may require separate clusters for security, billing, or compliance reasons.
These requirements multiply quickly. A multinational organisation that applies each of these constraints across its business units can end up managing dozens of Kubernetes clusters. Each cluster may have its own version, configuration drift, and resource utilisation profile. Getting a unified operational picture across all of them is a significant engineering challenge—let alone managing costs at scale.
Multi-Cluster Management Tooling
To manage its own large-scale Kubernetes infrastructure, IBM developed Razee, a continuous delivery and multi-cluster configuration management tool that it later released as open source. Razee is designed to push configuration and deployments across many clusters consistently. It is available at razee.io.
Other tools in this space include the CNCF project Argo CD for GitOps-based delivery, Fleet from Rancher for large-scale cluster management, and the emerging Open Cluster Management project. The multi-cluster management tooling ecosystem was still maturing at the time the original article was written and has continued to evolve.
When Managed and Specialised Services Make More Sense
Configuring all three layers of Kubernetes scaling—application, cluster, and multi-cluster—requires substantial engineering investment. For many organisations, fully managed Kubernetes services from cloud providers offload the cluster-scaling and infrastructure-management concerns, reducing the problem to application-level configuration.
For specific workloads, specialised managed services may be a better fit than a general-purpose Kubernetes cluster. Database-focused providers, for example, offer managed services built on Kubernetes operators that handle the operational complexity of running stateful workloads at scale—including automated failover, backup, and vertical or horizontal scaling—in a way that a general Kubernetes configuration would require significant custom work to replicate.
The relevant question is not whether to use Kubernetes, but which layers of the stack your team should own versus delegate to a managed service. The answer depends on your team’s size, operational maturity, and whether the workload in question is a differentiator or a commodity for your business.
Summary
| Scaling Layer | Mechanism | Key Concern |
|---|---|---|
| Application | Resource requests/limits, HPA | Correct profiling of workload resource needs |
| Cluster | Cluster Autoscaler, managed node groups | Infrastructure API integration and cost control |
| Multi-cluster | Federation tools, GitOps, Razee, Argo CD | Consistency, visibility, and cost across clusters |
Kubernetes scaling is not a single dial to turn up. It is a layered system where each layer depends on the one below it being configured correctly. Getting application resource requests right is the prerequisite for the HPA to work. Having the HPA drive pod counts up is the prerequisite for the Cluster Autoscaler to need to add nodes. And managing multiple clusters is impossible without visibility tools that span all of them. Work through the layers in order and your cluster will scale the way Kubernetes promises it can.