This article is based on analysis originally published on HPCwire.
The Question Every HPC Team Is Asking
“To Kube or not to Kube?” That question has been circulating in High Performance Computing (HPC) circles for several years. Kubernetes (K8s) has become the dominant platform for running containerized workloads in cloud environments, but HPC has its own deeply rooted toolchain, scheduling models, and operational expectations. Understanding where Kubernetes fits — and where it falls short — is essential for any team managing scientific, financial, or engineering compute workloads.
A Brief Kubernetes Primer
Kubernetes was originally developed at Google and was heavily influenced by their internal cluster management system, Borg. It was open-sourced in 2014 and has since grown into one of the most widely adopted open-source projects in existence, with thousands of contributors worldwide.
At its core, Kubernetes was built to orchestrate cloud-native applications composed of loosely coupled, containerized services — an architectural style known as microservices. It does far more than simple scheduling: it provides a complete runtime environment including service discovery via DNS, persistent storage management, secret management, rolling updates, auto-scaling, and health monitoring.
To deploy an application, a DevOps engineer writes a YAML or JSON manifest describing the application’s components, resource requirements, Docker images, port mappings, volume mounts, and scaling policies. Kubernetes then manages the lifecycle of the “Pods” that make up the application.
How HPC Workloads Differ from Cloud-Native Workloads
The types of applications that run well on Kubernetes are fundamentally different from traditional HPC workloads, and this is the root cause of friction when the two communities try to share infrastructure.
| Dimension | Kubernetes / Cloud-Native | HPC |
|---|---|---|
| Application lifecycle | Long-running services (web apps, caches) | Run-to-completion jobs (simulations, genomics pipelines) |
| Workload type | Microservices, APIs, stateless services | MPI jobs, parametric sweeps, multi-step workflows |
| Container style | Minimalist, fast to start | Large images with complex library dependencies |
| Scheduling needs | Pod placement, auto-scaling | Backfill scheduling, topology-aware scheduling, advanced reservations, preemption |
| Resource density | Many small containers per host | Few large, CPU/memory-intensive containers per host |
HPC workload managers like IBM Spectrum LSF and SLURM were designed with these requirements in mind. They offer capabilities such as:
- Backfill scheduling — filling idle slots with smaller jobs without delaying larger ones
- Advanced reservations — guaranteeing resource availability at a future time
- Preemption — suspending lower-priority jobs to free resources for high-priority work
- Topology-aware scheduling — placing tasks to minimise inter-node latency (critical for MPI)
- Job dependency management — chaining multi-step workflows
None of these are native capabilities of the standard Kubernetes scheduler.
Where Kubernetes and HPC Communities Agree
One area of genuine agreement is the value of containers — though for different reasons.
Cloud-native teams favour containers for speed and modularity: small, purpose-built images that start quickly and can be scaled horizontally. HPC teams are more interested in encapsulation and portability. Scientific applications often have complex, fragile dependency trees (specific compiler versions, MPI implementations, CUDA libraries, and so on). Packaging everything into a container solves the “it works on my cluster” problem and simplifies deployment across different HPC environments.
The result is that HPC containers tend to be significantly larger than their cloud-native counterparts, and are used in much smaller numbers per host — sometimes just one or two per physical node.
Will Kubernetes Take Over HPC?
The honest answer is: not entirely, and not any time soon. The HPC ecosystem has decades of investment in specialised libraries, middleware, and workflow tooling. While HPC applications are increasingly containerised, that does not create a compelling reason to re-architect them from scratch for Kubernetes.
There are, however, notable exceptions and intersection points:
- Financial services: Firms running quantitative models alongside traditional HPC are adopting cloud-native patterns. CI/CD pipelines on Kubernetes help teams iterate on risk models and analytics services continuously — a pattern that pairs well with HPC batch workloads running in parallel.
- AI and deep learning: A common hybrid pattern uses an HPC scheduler (with GPU-aware scheduling) for model training, and Kubernetes to serve trained models at scale for inference. Each platform is used where it excels.
- Data analytics: Frameworks like Apache Spark can run on both Kubernetes and HPC clusters, creating natural overlap in analytics-heavy environments.
Convergence Approaches: Getting the Best of Both Worlds
Several projects and products are working to bridge the gap between Kubernetes and HPC scheduling.
Kube-batch and Volcano
Kube-batch is an open-source batch scheduler for Kubernetes that adds gang scheduling and queue management to the platform. It is used by projects such as Kubeflow to improve the experience of running machine learning training jobs on Kubernetes. The project has since evolved into Volcano, a Kubernetes-native batch system designed specifically for HPC, AI, and big data workloads.
Running a Full HPC Scheduler Alongside Kubernetes
A more powerful strategy is to deploy a production-grade HPC scheduler as a containerised service within a Kubernetes cluster, making it available as an alternative scheduler for workloads that need it.
IBM demonstrated this pattern by running IBM Spectrum LSF on IBM Cloud Private (ICP). Rather than maintaining separate clusters for HPC and cloud-native workloads, Spectrum LSF runs as a service within the Kubernetes environment. Users select LSF as their preferred scheduler in their YAML manifest, immediately gaining access to advanced HPC scheduling capabilities:
- SLA management across users, groups, and projects
- Hierarchical fair-share resource allocation
- Resource reservation for periodic workflows
- Job dependency tracking and multi-step workflow orchestration
- GPU-aware and topology-aware scheduling
This approach allows teams to run HPC and cloud-native workloads on a single shared infrastructure, evolving each application at its own pace rather than forcing a hard migration.
Key Takeaways
- Kubernetes excels at orchestrating long-running, loosely coupled microservices. It is not a drop-in replacement for HPC workload managers.
- HPC schedulers offer capabilities — backfill, preemption, topology-awareness, advanced reservations — that the native Kubernetes scheduler does not provide.
- Both communities value containers, but for different reasons: cloud-native teams prioritise speed and modularity; HPC teams prioritise encapsulation and portability.
- Hybrid architectures — using Kubernetes for inference and service deployment, HPC schedulers for batch training and simulation — are a pragmatic path forward.
- Projects like Volcano and vendor solutions like IBM Spectrum LSF on Kubernetes are narrowing the gap, giving teams access to HPC-grade scheduling within a Kubernetes environment.
For teams navigating this space, the right approach is rarely all-or-nothing. Understanding the strengths of each platform and deploying them where they genuinely fit will deliver better outcomes than forcing every workload into a single paradigm.