Kubernetes turned ten years old having never been designed around a GPU, and the workaround stack teams now run on top of it is not a stopgap. It is the production pattern.
Kubernetes turned ten years old having never been designed around a GPU. Its scheduler, its resource model, and its assumptions about workloads all trace back to Borg, Google's internal system for scheduling stateless web services and batch jobs across commodity CPU fleets. That heritage shows up constantly in production AI clusters today: a platform team spins up a GPU node pool, installs the NVIDIA device plugin, and discovers that the scheduler now treats an H100 the same way it treats a virtual CPU core, as an interchangeable, fungible integer.
That abstraction was the right one for a world of stateless microservices. It is close to the wrong one for a world of multi-day distributed training runs, tightly coupled multi-GPU inference servers, and checkpoint-heavy fine-tuning jobs. The gap between what Kubernetes assumes and what AI workloads actually need is not a minor rough edge. It is the reason platform teams build entire secondary scheduling layers on top of Kubernetes rather than trusting it out of the box.
This matters for anyone buying, building, or budgeting GPU infrastructure right now, because the workaround stack (custom schedulers, topology plugins, checkpoint managers, and increasingly a return to bare metal for the largest jobs) is not free. It shows up as engineering headcount, as GPU idle time nobody can fully explain, and as a widening gap between the sticker price of a GPU hour and what a training run actually costs to complete.
Kubernetes exposes accelerators through the device plugin interface, a mechanism introduced to let vendors advertise custom hardware without patching the scheduler itself. The tradeoff baked into that design is opacity. The scheduler sees a count of available GPUs on a node and matches it against a pod's request. It has no native concept of GPU memory headroom, interconnect topology, or whether two requested GPUs sit on the same NVLink domain or across a slower PCIe hop. Two pods can each get a GPU and end up with wildly different effective bandwidth to each other, and the scheduler has no mechanism to know or care.
For CPU workloads, which core you land on rarely changes the outcome. For distributed training, it can change a job's runtime by a large multiple. NVSwitch and NVLink domains, PCIe switch boundaries, and even which rack a node sits in determine how fast gradients synchronize across GPUs. Kubernetes' default scheduler has no built in awareness of any of this. Teams running large training jobs typically bolt on topology aware placement through vendor tooling or custom scheduler extensions, because the alternative is a job that technically schedules successfully and then runs at a fraction of the speed the hardware is capable of.
Kubernetes schedules pods independently, one at a time, and tries to pack them efficiently onto available nodes. Multi-GPU training jobs need the opposite guarantee: every worker pod in the job must start together, on hardware that meets the topology requirement, or the whole job should not start at all. Scheduling seven out of eight required workers and leaving the job half launched wastes GPU hours across every node it touched. This is the classic gang scheduling problem, well understood in the HPC world that ran on Slurm for decades, and it is why projects like Volcano, Kueue, and Apache YuniKorn exist as batch scheduling layers on top of, or instead of, the default Kubernetes scheduler for AI workloads.
Kubernetes was built on the assumption that killing and rescheduling a pod is close to free. For a web service replica, it is. For a training job hours into a run, losing a worker without a recent checkpoint can mean losing that time entirely. Preemption, spot node reclamation, and routine node maintenance all interact badly with long running, stateful GPU jobs unless checkpointing is deliberately engineered into the workload itself. Kubernetes does not do this for you, and the state that needs to survive a preemption event has to land somewhere fast enough not to become the new bottleneck, which is where storage adjacency and bandwidth to the GPU fleet stop being an afterthought.
| Gap in default Kubernetes | Common workaround | What it costs to run |
|---|---|---|
| No topology aware placement | Vendor topology plugin or custom scheduler extension | Extra platform engineering time, vendor lock in risk |
| No gang scheduling | Volcano, Kueue, or Apache YuniKorn as a batch layer | Additional cluster component to operate and upgrade |
| Opaque GPU resource model | NVIDIA GPU Operator plus MIG configuration | Ongoing driver and firmware compatibility overhead |
| Stateless preemption assumptions | Custom checkpoint scheduling and fast adjacent storage | Storage bandwidth cost near the GPU fleet |
In practice, almost nobody runs vanilla Kubernetes for large scale GPU training. The pattern that has emerged across neoclouds and enterprise platform teams alike is a layered stack: Kubernetes stays as the control plane for orchestration, logging, and lifecycle management, while a purpose built batch scheduler (Volcano, Kueue, or a vendor's own layer) handles gang scheduling and queueing, and a separate topology plugin handles GPU aware placement. NVIDIA's GPU Operator automates driver, plugin, and Multi-Instance GPU (MIG) configuration across the fleet so at least that layer is not hand rolled per cluster.
NVIDIA's 2025 acquisition of Run:ai is a useful signal of where the market decided the gap was worth solving commercially rather than leaving to open source alone. Run:ai's product was, in essence, a GPU aware scheduling and fractional allocation layer sitting on top of Kubernetes, sold specifically because the default scheduler could not do fractional GPU sharing, fair share queueing across teams, or topology aware bin packing on its own.
For inference rather than training, the pressure point shifts. Serving large models efficiently often means keeping model weights resident in GPU memory and routing requests to warm instances rather than treating every replica as disposable and interchangeable, which is again a departure from the assumptions the scheduler was built around. Tools like KubeRay and various model serving frameworks exist largely to reintroduce state and session awareness into a system designed to forget both.
The Kubernetes project has not ignored this. Dynamic Resource Allocation (DRA) is the most significant structural change aimed directly at the accelerator problem: rather than the device plugin's flat integer count, DRA lets a workload request a resource with structured, attribute based parameters (memory size, specific topology, sharing mode) and lets the scheduler reason about those attributes directly rather than treating hardware as an undifferentiated pool. It has moved through alpha and beta stages across recent Kubernetes releases, with the pace of graduation to general availability shaped heavily by how much vendor and platform feedback the SIG receives from real GPU clusters, which is precisely the workload category that stresses it hardest.
None of this changes the near term reality. DRA maturing upstream does not retroactively fix a cluster running an older Kubernetes version, and gang scheduling still is not a first class scheduler primitive even where DRA is available. For the next several release cycles, the honest guidance for a platform team is to assume the workaround stack (batch scheduler plus topology plugin plus checkpoint discipline) is the production pattern, not a stopgap.
Every one of these failure modes eventually becomes a storage problem. Checkpointing at scale means writing large model states fast enough that GPUs are not left idle waiting on I/O, which means storage needs to sit close to the compute with real bandwidth, not just capacity. A failed gang scheduling attempt that has to restart from the last checkpoint is only cheap if that checkpoint was written somewhere fast to read back from. And when a job migrates between providers, whether by design or because capacity vanished unexpectedly, the data it depends on has to move with it. Egress fees on that migration can erase whatever the team saved by chasing cheaper GPU pricing in the first place, which is exactly the kind of cross provider move that zero egress storage from providers like Backblaze B2, Wasabi, or Cloudflare R2 was built to absorb.
This is also why the industry's talk of post-Kubernetes agentic infrastructure is less about replacing Kubernetes outright and more about admitting that the control plane and the scheduling and data layers underneath it are becoming distinct concerns again, the way they were before Kubernetes unified so much of the stack for the CPU era.
If you are running or planning GPU workloads on Kubernetes, treat the default scheduler as a starting point, not a finished solution. For any job spanning more than a single GPU, evaluate a batch scheduler (Volcano or Kueue are the most mature open source options) before you hit a production incident caused by a half launched distributed job. Confirm whether your CoreWeave, Nebius, or other neocloud contract includes topology aware placement as a managed feature or whether that responsibility falls on your own platform team, since pricing quotes rarely make that distinction obvious. Build checkpoint frequency into the cost model for every long running training job, and place checkpoint storage for GPU adjacent bandwidth rather than lowest sticker price, since a checkpoint you cannot write or restore quickly enough is not actually protecting the job. Finally, budget the engineering time for this workaround stack explicitly. It is not a one time setup cost, it is an ongoing operational surface that scales with how much of your GPU fleet runs multi-node jobs.
Kubernetes did not fail the GPU era. It simply was never asked the question, and the industry is still building the layer that answers it.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
