A training run stalls every twenty minutes while the GPUs sit idle waiting on a checkpoint write. The vendor's spec sheet promised one million IOPS. IOPS was never the metric that mattered.
This happens constantly in AI infrastructure teams, and it is not a hardware failure. It is a vocabulary failure. IOPS, throughput, and latency measure three different things, they trade off against each other, and a storage system can post an outstanding number on one while quietly failing the workload on another. Vendors are not lying when they lead with the metric that flatters their product. They are just picking the metric that is easiest to make look good in a synthetic benchmark, which is rarely the metric that governs how your training job or your inference service actually behaves.
This matters more now than it did two years ago because AI workloads have inverted the access patterns that most enterprise storage strategy was built around. General business applications lean on small transactional reads and writes spread evenly over time. Training and inference concentrate demand into brutal, predictable bursts: a massive sequential write when a checkpoint fires, a wall of random small reads when a shuffled dataset loads, a latency floor that a vector search either meets or a user notices. Buying storage for AI the way you bought storage for a database cluster is how teams end up with a very expensive number on a spec sheet and a GPU cluster that is still waiting.
IOPS, input output operations per second, counts how many individual read or write operations a storage system can complete in a second. It matters most when a workload issues many small, discrete requests, such as reading thousands of small image files during a training data shuffle.
Throughput measures how much data moves per second, typically in gigabytes per second, regardless of how many operations that took. It matters most when a workload moves large, contiguous blocks, such as writing a multi hundred gigabyte model checkpoint in one pass.
Latency measures how long a single operation takes to complete, usually in microseconds or milliseconds. It matters most when a workload cannot proceed until that one operation returns, such as a retrieval augmented generation lookup that a user is waiting on in real time.
A storage system can be exceptional at one of these and mediocre at the other two, and vendors size their marketing numbers accordingly. High IOPS is easy to advertise using tiny, cache friendly requests. High throughput is easy to advertise using large sequential transfers that never happen in production. Low latency is the hardest and most expensive to deliver honestly, which is exactly why it is the metric most often left off the spec sheet.
Four points in a typical AI pipeline each stress a different metric, and a storage system tuned for one will disappoint at the others. This is also why storage has become the anchor of the AI infrastructure stack rather than an afterthought behind the GPU line item.
Loading a shuffled dataset across thousands of small files, common in vision and multimodal training, generates a flood of small random reads. This is an IOPS bound workload. A system with generous throughput but weak random IOPS will starve the GPUs even though the aggregate data volume is modest.
Saving model state at scale is a large sequential write, and it needs to happen fast enough that the GPUs are not idle while it completes. This is a throughput bound workload. IOPS is nearly irrelevant here, since the write is one long contiguous operation rather than many small ones.
A RAG lookup or a vector database query has to return before the model can continue generating a response. This is a latency bound workload, and it is unforgiving. A system that is excellent on throughput and IOPS but has a long tail on latency will produce a service that feels slow even when every other metric looks fine on a dashboard.
As context windows grow, some inference architectures offload key value cache to fast local storage between requests. This workload wants both low latency and high throughput at once, which is why it is usually the first place a storage architecture decision gets expensive.
The four figures above should be treated as internal working estimates pending live verification against current vendor specification sheets, since specification sheets and pricing move quickly in this market.
A modern enterprise NVMe drive can sustain more than 1,000,000 random read IOPS, but that number is measured with small, queue depth optimized requests under ideal lab conditions, not the access pattern of a real training shuffle. PCIe Gen5 NVMe drives publish sequential throughput up to roughly 14,000 megabytes per second per drive, which is the number most checkpoint bound teams should actually be shopping on. Latency is where the real gap opens: local NVMe typically returns a read in 20 to 100 microseconds, while network attached block storage in the same cloud region commonly runs 1,000 to 3,000 microseconds, a 20 to 50 times difference that a synthetic IOPS or throughput benchmark will never surface.
The fourth figure is a modeled estimate rather than a vendor claim: a 70 billion parameter model checkpoint, stored with optimizer state in mixed precision, commonly exceeds 1 terabyte once gradients and optimizer moments are included alongside the weights themselves. That single number is why checkpoint throughput, not IOPS, decides how much GPU idle time a training run absorbs every time it saves.
Every storage benchmark on a spec sheet was run against a specific access pattern, and that pattern is chosen to make the product look as good as possible. A drive rated for a huge sequential throughput number will not deliver anywhere close to that figure under the small random reads a data loader generates, and a drive rated for enormous random IOPS will not deliver anywhere close to that figure once request sizes grow past a few kilobytes.
The practical fix is to stop asking a vendor for their best number and start asking for their number under your access pattern. A shuffled dataset of small files, a checkpoint write of a known size, and a vector lookup at your expected concurrency are three different tests, and a credible vendor can run all three without flinching.
The table below maps each stage of a typical training and inference pipeline to the metric that actually governs it, and the storage tier that tends to fit.
| Pipeline Stage | Bottleneck Metric | Recommended Tier |
|---|---|---|
| Training data ingestion, shuffled reads | IOPS | Local NVMe or parallel file system |
| Checkpoint writes | Throughput | Local NVMe scratch, replicated after write |
| Vector search or RAG lookup at inference | Latency | Local NVMe or in memory cache |
| KV cache offload, long context inference | Latency and throughput | Local NVMe adjacent to GPU |
| Cold model artifacts and archives | Cost, not performance | Object storage |
Local NVMe scratch space attached directly to the GPU node is the right answer for checkpoint writes and KV cache offload, where throughput and latency both matter and the data does not need to survive the node. This is also why storage adjacency to compute has become a genuine architectural constraint rather than a nice to have: moving a large checkpoint or a shuffled dataset between data centers over the network erases whatever performance the local drive would have delivered.
Parallel file systems or high throughput network storage fit training data ingestion at scale, where the dataset is larger than any single node's local capacity and many GPUs need to read it concurrently. Object storage remains the right home for cold model artifacts, training data archives, and anything that does not sit in the hot path of a running job, since its economics and durability profile are far better than any performance tier for data that is not being read every second.
This is also the point at which storage tiering and cloud economics intersect with GPU strategy directly. Neoclouds and GPU focused providers such as CoreWeave and Lambda Labs have built their offerings around pairing GPU instances with local NVMe precisely because storage adjacency, not raw compute, is what determines whether a training cluster hits its promised utilization. Sunny Smith, founder and CTO of Massed Compute, has made a similar point about GPU infrastructure economics: the constraint teams do not budget for is rarely the GPU itself, it is what sits directly next to it. Related territory came up on the DataStorage.com Podcast in Episode 5 with Russ Artzt, on why neoclouds exist and how GPU economics are reshaping compute strategy: listen to the episode.
Egress economics compound this. If training data or checkpoints need to move between providers because of GPU availability, egress fees on that movement can erase whatever price advantage motivated the switch in the first place, which is why zero egress storage options are worth evaluating even for teams whose primary concern is performance rather than cost.
Five questions separate a credible storage evaluation from a spec sheet comparison.
First, ask for the benchmark methodology behind every headline number, specifically the request size, queue depth, and read write mix used to produce it. Second, ask for a number measured against your actual access pattern rather than the vendor's best case pattern, even if that means running your own test dataset against the system before signing anything. Third, ask for latency at the 99th percentile, not the average, since an average latency figure hides exactly the tail behavior that will show up as a slow response for a real user. Fourth, test at the concurrency you will actually run in production, since a system that performs well with one GPU reading can degrade sharply once dozens of GPUs contend for the same volume at once. Fifth, confirm whether the number was measured on a cold or warm cache, since a cache warmed benchmark can flatter a system that will never see that same request twice in production.
A spec sheet number is not a promise about your workload. It is a promise about someone else's benchmark.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
