A stalled checkpoint write on a large GPU training cluster can burn thousands of dollars an hour in idle silicon, and the culprit is almost never the GPUs.
A 512-GPU H100 training cluster running on a typical multi-year committed contract costs somewhere in the range of 3,000 to 6,000 dollars an hour in aggregate compute spend, depending on provider and term (directional pricing from our GPU Price Explorer, 2026, pending live verification). When that cluster stalls waiting on a checkpoint write or a slow shuffled-epoch read, the GPUs are not resting. They are burning the same hourly rate while producing nothing. Storage is rarely the line item anyone questions first when a training run runs long, but it is very often the reason storage sits quietly at the anchor of the AI infrastructure stack.
This is the argument at the center of a fight that most infrastructure teams have not had explicitly: parallel file systems versus object storage for AI training. Both categories are mature, both have passionate vendors, and both are frequently the wrong choice when applied to the wrong stage of the pipeline. Getting this decision right is now a FinOps question as much as an architecture one.
Object storage built its reputation on a workload pattern that AI training does not resemble: write once, read occasionally, tolerate latency in the hundreds of milliseconds, optimize for durability and cost per terabyte over raw throughput. That pattern describes backups, media archives, and static assets extremely well. It does not describe what happens inside an active training job.
A training run reads the same dataset repeatedly across epochs, shuffled into a different order each pass, from thousands of GPU workers simultaneously. Every few minutes to every few hours, depending on model size and failure tolerance, the cluster pauses to write a full checkpoint of model state, optimizer state, and gradients back to persistent storage. Neither pattern looks like a media archive. Both punish a storage layer that was designed for infrequent, sequential, large-object access.
Parallel file systems, WekaFS, VAST Data, DDN, IBM Storage Scale, and Lustre among them, distribute both data and metadata across many nodes and present a single POSIX-compliant namespace to every GPU worker at once. The design goal is not cost per terabyte. It is sustained throughput and metadata operations per second under massively concurrent access, which is exactly the shape of a training job's I/O pattern.
Object storage APIs treat every GET and LIST as a discrete network operation with its own latency floor, typically tens of milliseconds even on a well-tuned S3-compatible endpoint. A shuffled-epoch read touching millions of small files per second across a large cluster turns that latency floor into the actual bottleneck of the training run. Parallel file systems distribute metadata handling across dedicated nodes and can sustain metadata operation rates an order of magnitude or more above what object storage APIs are built to deliver at the same cluster scale, per vendor benchmark disclosures from WekaFS and VAST Data covering 2025 to 2026, directional and vendor-published, pending independent verification.
The other half of the design is physical proximity. Parallel file system deployments lean on RDMA and NVMe over Fabrics to move data between storage nodes and GPU nodes with minimal CPU overhead and microsecond-scale latency, and they are almost always racked adjacent to the compute they serve. Moving a petabyte of training data between data centers is not a config change, it is a multi-day operation that most teams cannot afford mid-training. That adjacency requirement is precisely why storage architecture and GPU procurement cannot be planned separately, a point echoed across infrastructure operators who have built and rented GPU fleets at scale.
None of this makes object storage the wrong choice. It makes it the wrong choice for one specific stage of the pipeline. For the raw training corpus before preprocessing, for dataset versions you are not actively training against, and for the long-term home of completed checkpoints and model artifacts, object storage remains dramatically cheaper and operationally simpler.
Parallel file system deployments, whether self-managed on NVMe or purchased as a managed high-performance tier, typically run several multiples above object storage cost per terabyte per month. Running an entire multi-petabyte training corpus permanently on parallel-file-system-grade storage is the single most common way teams overspend on AI infrastructure without realizing it, and it is worth running the numbers through a cost calculator before committing to either tier at scale.
GPU capacity is scarce and it moves. A team training on AWS today may be on CoreWeave or Lambda Labs next quarter simply because that is where the next allocation of H200s or B200s is available. Every one of those moves drags data with it, and egress fees on a multi-petabyte training corpus can erase the savings that motivated the provider switch in the first place. This is exactly why zero-egress providers such as Backblaze B2 and Wasabi have become a genuine architecture decision for AI teams, not just a cost optimization for archival workloads. Keeping the durable, cold layer of your data estate egress-free is what makes multi-provider GPU strategy financially viable at all.
Checkpointing is where the two categories collide most visibly. As models have grown from tens of billions to hundreds of billions of parameters, checkpoint sizes have grown with them, and a slow checkpoint write does not just cost the time it takes: the entire GPU cluster typically sits idle for the duration, because training cannot safely proceed until the checkpoint is confirmed durable. Teams running large models on general-purpose object storage frequently report checkpoint writes stretching into many minutes, compounding into a meaningful percentage of total wall-clock training time over a multi-week run. Moving hot checkpoint scratch onto a parallel file system or NVMe-oF tier is consistently one of the highest-leverage single changes a training team can make, because it converts wasted GPU-hours directly into finished training steps.
In practice, the winning architecture is rarely either category alone. It is a tiered pipeline: object storage as the durable, cost-efficient system of record for raw and cold data, a parallel file system or NVMe-oF scratch tier sitting adjacent to the GPU cluster for the active working set and checkpoint writes, and an automated or semi-automated process moving data between the two as jobs start, pause, and complete. Getting the tiering boundary wrong in either direction is expensive: too much on the fast tier and the cost per terabyte problem dominates the bill, too little and GPU idle time from storage stalls dominates it instead.
Before signing a contract for either category, run the numbers your own cluster actually produces rather than trusting vendor benchmarks.
Start by measuring GPU idle time attributable to storage I/O on your current training runs. If checkpoint writes or data loading account for more than a low single-digit percentage of wall-clock time, the fast tier will very likely pay for itself. Next, separate your data estate by access pattern rather than by age: what is actively read every epoch belongs closer to compute, what is written once and read rarely belongs on the cheapest durable tier you can find. Finally, price the egress cost of a future provider switch into your storage decision now, not after the switch is already underway. A storage layer that locks you into one GPU provider is a hidden cost that only shows up the day you need to leave.
| Dimension | Parallel File System | Object Storage |
|---|---|---|
| Best fit workload | Active training reads, checkpoint writes | Raw corpus, cold data, completed checkpoints |
| Metadata operations | Very high | Lower |
| Typical access pattern | POSIX, RDMA, NVMe-oF | HTTP API, S3-compatible |
| Cost per TB per month | Several times higher | Lowest of the two |
| Egress on provider switch | Not typically applicable, adjacent to compute | Significant unless zero-egress |
The storage layer you choose does not just store your training data. It sets the ceiling on how much of your GPU bill actually turns into finished training steps.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
