Rent the same GPU from two providers, run the same job on both, and you should expect two different bills. Price per hour is not price per unit of work.
Rent 512 H100s from one provider and 512 from another, run the same training job on both, and you should expect two different bills. Not because the hourly rates differ, but because the jobs will not finish at the same time. The GPU on the invoice is a SKU. The GPU you actually get is that SKU plus a power budget, a cooling loop, a PCIe path, a network fabric, a storage system, and a set of neighbors you never see.
This is measured, not anecdotal. When researchers at the University of Wisconsin-Madison instrumented more than 90% of the GPUs in five large supercomputing clusters, including Oak Ridge's Summit and TACC's Frontera, they found an average performance variation of 8%, and as much as 22%, between GPUs of identical architecture and SKU (Sinha et al., SC22, 2022). On September 23, 2026, SemiAnalysis published ClusterMAX 3.0, its hands-on rating of GPU clouds. Of the 77 providers it reviewed, only 19 earned a medallion rating, and its testing found some providers' storage bandwidth degrading by almost 40% under sustained load compared with the initial benchmark (SemiAnalysis, 2026).
For anyone comparing providers on price per GPU-hour, that is the whole problem in one sentence: price per hour is not price per unit of work. This article breaks down where GPU performance variance comes from, which of it is genuinely caused by other tenants and which is not, and the acceptance tests and contract terms that stop you paying for performance you never receive.
Two effects stack. The first is silicon. No two chips are physically identical, and a GPU's power management hides that until the chip hits its power or thermal limit. At that point dynamic voltage and frequency scaling (DVFS) lowers clock speeds, and each chip throttles at a slightly different point. The SC22 study attributes most of its same-SKU variation to exactly this: manufacturing variability exposed by power management.
The second effect is synchronization, and it is what turns small variation into large bills. Distributed training is lockstep. Every step waits for the slowest rank to finish its compute and its collective communication before anyone moves on. Meta put it plainly in its Llama 3 paper: even a single straggler can slow down thousands of other GPUs (Meta, 2024). Variance across a fleet does not average out inside a job. The job runs at the speed of its worst member, so the tail of the distribution becomes the bill.
Inference is more forgiving, because replicas run independently, but the same variance shows up as uneven tokens per second per replica and a fatter p99 latency. If you size an inference fleet from a benchmark on one well-behaved node, you will under-provision.
"Noisy neighbor" is a useful label, but it hides five distinct mechanisms. Each needs a different test and a different contract clause.
Some low-cost and fractional GPU offerings are not whole GPUs at all. NVIDIA's own GPU Operator documentation is explicit that time-slicing provides no memory or fault isolation between the tenants sharing a card, and that requesting two time-sliced replicas does not guarantee twice the compute (NVIDIA, 2026). Multi-Instance GPU (MIG) is different: it partitions the card into instances with memory and fault isolation at the hardware layer. If a listing is cheap enough to look suspicious, ask whether you are buying a whole GPU, a MIG slice, or a time-slice.
An 8-GPU server shares its CPU sockets, memory bandwidth, PCIe switches, NICs and local NVMe across all eight cards. When a provider carves that server into 1, 2 or 4 GPU virtual machines for different customers, your data loader, tokenizer and checkpoint writes compete with someone else's for those shared paths. Whole-node allocations remove this layer entirely, which is why the single-GPU price and the per-GPU price of a full node are not the same product.
Collective operations such as all-reduce and all-to-all are where multi-node jobs spend much of their time, and they are only as fast as the shared fabric underneath. ByteDance's production training cluster avoided contention-driven stragglers by design: large jobs never shared a server, and the network was overprovisioned and tuned so congestion did not occur (Lin et al., OSDI 2025). Most rented clusters give you no such guarantee. ClusterMAX 3.0 reports big performance differences on certain Ethernet networks under load, and notes that job placement relative to rack and scale-out boundaries materially changes results. In one cluster it tested, the nodes visible to each scheduler spanned multiple racks with nothing preventing a job from needlessly crossing the rack boundary.
Storage is where the neighbor effect is easiest to measure and least discussed. Parallel file systems and object stores are almost always shared, and their performance under your load depends on everyone else's. SemiAnalysis found near 40% bandwidth degradation under sustained load on some providers, and roughly 10% variation in object storage tests. The same report shows how wide the gap between providers gets on simple operations: running import torch takes 1 to 2 seconds on most providers but 10 to 20 seconds on some file systems with small-file problems, and loading a small model into GPU memory from shared storage takes 10 to 40 seconds on the best providers and over 2 minutes on others. Every one of those seconds is billed GPU time with the GPUs idle.
Racks share power feeds and cooling loops with the racks beside them. Meta measured a 1 to 2% diurnal swing in Llama 3 405B training throughput, caused by higher mid-day temperatures pushing GPUs into DVFS (Meta, 2024). That was on one of the most carefully engineered clusters in the world. SemiAnalysis makes the practical point: GEMM (general matrix multiply) numbers rarely distinguish providers on a short burst, but they do during an extended burn-in, when a chip that is not properly cooled drops its clocks. A five-minute benchmark will not find this. An eight-hour one will.
| Layer | What is shared | Symptom | Test that catches it |
|---|---|---|---|
| GPU | Card via time-slicing or MIG | Erratic throughput, memory contention | Whole GPU confirmed in writing; per-GPU GEMM sweep |
| Host | CPUs, PCIe, NICs, local NVMe | Slow data loading and checkpoints | Whole nodes; data loader throughput test |
| Network | Scale-out fabric | Slow or uneven collectives | NCCL all-reduce and all-to-all at full size, by time of day |
| Storage | Parallel file system, object store | Bandwidth drop under load, slow model loads | fio sweep from 1 client to all nodes, held 15+ min |
| Power and cooling | Rack feeds, cooling loops | Clock throttling, diurnal swings | 8-hour combined GPU and network burn-in |
Here is the uncomfortable finding. ByteDance and New York University analyzed 3,079 production training jobs of 128 GPUs or more over five months. 42.5% of those jobs ran at least 10% slower than they should have because of stragglers, and at the tail, stragglers wasted 45% of a job's allocated resources (Lin et al., OSDI 2025). Yet in a cluster with no resource contention, the researchers found that hardware or software faults in individual servers were not a significant cause. The main culprits were uneven work partitioning across pipeline stages, uneven sequence lengths across microbatches, and pauses from Python's automatic garbage collector. When machine faults did occur, though, they caused severe slowdowns.
The software stack matters too. Sunny Smith, founder and CTO of Massed Compute, told the DataStorage.com Podcast (Episode 7, Inside the GPU Carrier Layer) that generic containers tuned for one GPU environment leave 30 to 50% of performance on the table when deployed on another. ClusterMAX 3.0 shows the same effect at the network layer: on one hyperscaler's GB300 cluster, the default NCCL library picked an unroutable address and hung until an environment variable was pinned, and enabling the provider's own NCCL plugin lifted peak all-to-all throughput by 4.4%.
The practical rule: identical GPUs on different clouds are not an identical test. Before you attribute a slowdown to a neighbor, rule out your own pipeline balance, data loading and communication library configuration. Then hold the provider to account for the layers only they control.
Take a 512-GPU H100 training run lasting 30 days. That is 368,640 GPU-hours. At $3.40 per GPU-hour, close to the September 2026 on-demand median of about $3.41 across 44 providers tracked by GetDeploying (2026), the planned run costs $1,253,376.
Now let one slow rank stretch the job. An 8% slowdown, the average same-SKU variation in the SC22 study, adds 29,491 GPU-hours and $100,270. A 22% slowdown, the study's maximum, adds $275,743. For comparison, negotiating the hourly rate down by 20 cents saves $73,728 on the same run. A procurement team can win a rate cut that is worth less than one badly cooled node.
This is the editorial point we keep returning to on DataStorage.com: a cheaper unit price is not a cheaper bill. A cheaper GPU-hour is not a cheaper training run if the hours stretch. The metric that matters is cost per completed job, and the only way to know it before you commit is to test.
Most GPU contracts include some kind of acceptance window. Use it. The following sequence mirrors the testing that separates top-rated providers from the rest in ClusterMAX 3.0, scaled to what a buyer can run in a few days.
Write the acceptance thresholds into the agreement, along with the right to have nodes that fail them replaced, and service credits when replacement misses its window. For rack-scale NVL72 systems, where a failed tray cannot simply be hot-swapped, SemiAnalysis reports the most common structure is an "NVL64+" SLA: per-node terms, with the whole rack counted as down if three or more of its 18 nodes fail at once. Know which structure you are signing before the first incident.
Track per-rank step time, not just job throughput, so a straggler shows up as an outlier rather than a vague slowdown. Watch GPU clock throttle reasons in DCGM (NVIDIA Data Center GPU Manager). Rebalance pipeline stages and control garbage collection in your own code. Some providers now do part of this for you: CoreWeave offers GPU straggler detection built on NCCL telemetry.
In ClusterMAX 3.0, Nebius joined CoreWeave in the Platinum tier, and SemiAnalysis notes that Nebius now consistently commands premium pricing. What the premium buys is not faster silicon. Both rent the same NVIDIA parts as everyone else. It buys detection and replacement. CoreWeave, for example, runs 20 to 30 minute burn-ins in every idle hour on its Kubernetes nodes, pre-empted by customer work, so faults are found by the provider rather than by a production job.
That is the right frame for the price comparison. A provider charging 10% more per GPU-hour that consistently delivers 10% more completed work per hour is not more expensive. One charging 10% less that hands you an untested node and a support queue routed through two resellers is. Compare current rates in our GPU Price Explorer, then run the numbers on completed jobs, not hours.
The GPU on the invoice is a SKU. The GPU you get is a system, and you only get the price you were quoted if the whole system shows up.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
