For every dollar of accelerator capacity those organizations were paying for, roughly 95 cents bought heat.
Cast AI published a number in April 2026 that should have triggered emergency budget reviews and mostly did not. Across the non-optimized Kubernetes clusters it analyzed on AWS, Azure and Google Cloud, average GPU utilization was 5 percent. Not 50. Five. For every dollar of accelerator capacity those organizations were paying for, roughly 95 cents bought heat.
The same report put CPU utilization at 8 percent and memory at 20 percent, both down year over year. Those numbers are bad. The GPU number is a different category of bad, because of the price tag attached to it. As Cast AI co-founder Laurent Gil put it, "A GPU sitting idle costs dollars per hour." An idle vCPU costs fractions of a cent. The waste rate is comparable. The bill is not.
This is happening while Amazon, Microsoft, Alphabet and Meta guide toward as much as 725 billion dollars in combined 2026 capital expenditure, and while Flexera's 2026 State of the Cloud Report shows estimated wasted cloud spend rising to 29 percent, its first increase in five years, attributed directly to AI workloads. The industry is buying accelerators faster than it is learning to use them.
Before you take this figure into a budget meeting, understand its boundaries. Three qualifications matter.
It comes from a vendor that sells optimization software. Cast AI has genuine telemetry across tens of thousands of clusters, and Data Center Knowledge reported the sample at roughly 23,000 clusters, which is a real dataset. But the company's product exists to raise this number, and the sample is explicitly non-optimized clusters. Treat 5 percent as the floor of the distribution, not the median of all AI infrastructure.
It measures Kubernetes-scheduled GPU nodes, not every GPU in the world. A dedicated training cluster running one job for six weeks is a different animal. Independent estimates land higher for that reason: one analyst quoted by Data Center Knowledge put typical Kubernetes-based AI clusters at 15 to 25 percent, and statically provisioned GPU fleets are commonly benchmarked at 30 to 40 percent.
Every one of those numbers is still a catastrophe. The argument is not whether the average is 5 percent or 20 percent. It is that a 30,000 dollar accelerator is delivering a fifth of what it could, and nobody is being held accountable for the gap.
There is a second measurement problem, and it runs in the opposite direction. Most teams that do check utilization are checking the wrong metric, and it flatters them. Three distinct things get called "GPU utilization", and they answer very different questions.
The practical consequence: if your only evidence that GPUs are working is a green dashboard, you do not have evidence. Instrument at the job level and tie it to throughput per dollar, or you are measuring allocation and calling it productivity.
From 2023 through 2025, H100 capacity was genuinely scarce and on-demand queues ran into weeks. Reserving more than you needed was rational insurance. Teams locked in capacity before the workloads existed to fill it, and utilization telemetry was irrelevant because the goal was acquisition, not efficiency. Those reservations are still running. The scarcity that justified them is not.
The default Kubernetes device plugin allocates one full GPU per container. A model that needs 12 GB of an 80 GB H100 gets the whole card, and the other 68 GB is unreachable by any other workload. Multi-Instance GPU partitioning and time-slicing both exist. Most clusters run neither, because enabling them requires someone to own the scheduling decision.
Internal inference tools follow office hours. Traffic peaks from mid-morning to early evening on weekdays and collapses overnight and on weekends. A statically sized cluster that meets peak demand is idle for roughly two-thirds of every week by construction. Nights and weekends alone account for about 100 of the 168 hours you are paying for.
This is the cause that gets budgeted last and hurts most. A GPU blocked on a data loader is not underutilized because of scheduling. It is starved. Training jobs stall waiting on object storage reads. Checkpoint writes serialize an entire multi-node run behind one storage target. Embedding pipelines pull from a bucket in a different region than the compute, and every batch pays a latency toll.
Sunny Smith of Massed Compute made this point directly on the DataStorage.com podcast: storage has to be adjacent to the GPU with high east-west bandwidth, and moving a petabyte between data centers is prohibitive enough to shape the whole architecture. Storage is not a peripheral concern in a GPU budget. It is the control plane that determines whether the accelerators ever reach full speed.
The NVMe shortage compounds this. With NVMe pricing roughly tripled and scarcity expected to persist into late 2027, teams are under-provisioning local scratch precisely when their data pipelines need it most. Salvaging 40 dollars per terabyte on scratch storage while a 64-GPU cluster idles is not a saving.
Utilization sits between two teams. Infrastructure treats it as a scheduling metric. Finance treats it as a return-on-spend metric. In most organizations, one of those views exists and the other does not, which means no single person is accountable when the two diverge. Cost dashboards show what you spent. Almost none show what you got for it, which is the same structural blind spot behind the hidden costs buried in cloud billing.
Consider a mid-size team running 64 H100s on committed neocloud capacity at 2.50 dollars per GPU-hour, a defensible mid-market rate in mid-2026 when neocloud on-demand H100s run roughly 2 to 3 dollars and hyperscaler rates run several times higher. You can check live rates across providers on the GPU Price Explorer. At 730 hours per month, that is a 116,800 dollar monthly bill.
At 5 percent utilization, the cluster performs the work of 3.2 continuously busy GPUs. Consolidate that same work onto 8 GPUs running at 40 percent, a conservative target that requires no exotic tooling, and the bill becomes 14,600 dollars. Identical output.
| Scenario | Utilization | Monthly bill | Idle spend |
|---|---|---|---|
| As deployed 64 GPUs |
5% | $116,800 | $110,960 |
| Consolidated 8 GPUs |
40% | $14,600 | $8,760 |
| Difference | plus 35 pts | $102,200 saved | per month |
Run the same footprint at hyperscaler on-demand rates near 6.88 dollars per GPU-hour and the as-deployed bill exceeds 321,000 dollars a month. The utilization gap and the provider premium compound. Most teams are paying both penalties simultaneously and treating the resulting number as the cost of doing AI.
For two years, falling GPU prices quietly absorbed a lot of bad utilization. H100 rental rates fell from launch-era highs of around 8 dollars per hour to a 2 to 4 dollar band as supply caught up. Waste was expensive but the trend line was forgiving.
That trend has reversed. Cloud vendors raised H200 prices by 15 percent, which Cast AI's Gil described as breaking a two-decade pattern of falling compute costs, and NVIDIA signaled an approximately 20 percent increase on H100 rentals for 2026. Committed capacity pricing is now flat to rising rather than falling. Meanwhile Flexera found 76 percent of large enterprises spending more than 5 million dollars a month on public cloud, with generative AI now the third most used cloud service at 58 percent adoption, up from 50 percent.
Prices rising and consumption rising at the same time is the condition under which a tolerable inefficiency becomes an unsurvivable one. The teams that never built utilization discipline during the cheap years are about to discover what it costs to lack it.
You cannot fix what your dashboard is flattering. This costs engineering time and no capital, and it usually reveals the answer within a week.
Multi-Instance GPU partitioning on A100, H100 and H200 class hardware, or time-slicing where MIG does not fit, converts one card into several schedulable units. This is the single highest-leverage change for clusters running many small models.
Serving stacks such as vLLM and SGLang routinely move inference utilization from under 20 percent to over 70 percent by batching requests dynamically rather than padding to a fixed shape. Same hardware, several times the throughput.
Nightly embedding regeneration, evaluation suites, batch summarization and fine-tuning runs all tolerate interruption. Queue them into the overnight and weekend window that you are already paying for.
Verify that training data sits in the same region and ideally the same facility as the GPUs. Provision enough local NVMe scratch to keep loaders ahead of the accelerators. Check whether checkpoint writes are serializing your cluster. Salesforce, running on Google Cloud, reported lifting GPU utilization from 48 percent to nearly 100 percent primarily through storage and scheduling optimization, not through new hardware.
Once you know your true steady-state demand, reserve only that and burst the rest, which is where the reserved, on-demand and spot trade-off actually pays. Shorter contracts and a genuine multi-provider posture are worth more now that supply has loosened and buyers have leverage they did not have in 2024. Just budget for the egress: moving training corpora between providers can erase the savings that motivated the move, which is why zero-egress storage belongs in any multi-provider GPU strategy.
High utilization is achievable, and the gap between the median and the leaders is instructive.
None of those organizations bought a different GPU than you can buy. They built the operational capability to keep it fed. The hardware is a commodity. Utilization is the competitive advantage, and right now it is the cheapest performance improvement available to any infrastructure team, because it requires no procurement cycle at all.
The questions infrastructure and finance teams ask most often once the utilization number lands on the table.
The AI infrastructure conversation has been dominated by procurement questions: which GPU, which provider, which contract length. The Cast AI data reframes it. The most expensive line item in most AI budgets is not the hardware you have not bought yet. It is the hardware you already own that is doing nothing 95 percent of the time.
You do not have a GPU shortage. You have a GPU utilization problem, and it is the only infrastructure cost you can cut without asking anyone for approval.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
