A training checkpoint can run six to seven times the size of the model it will eventually ship as, and almost no GPU budget accounts for it.
Every training budget starts with GPU hours. Rightly so: on a large pretraining run, compute is the dominant cost by an order of magnitude. But GPU budgets are usually built with a hidden assumption baked in: that checkpoint storage is a rounding error. It is not. It is a cost that scales with every parallel experiment, every ablation, every hyperparameter sweep, and every week a checkpoint sits in hot storage after the run that produced it has already shipped.
The reason it gets missed is structural, not careless. Compute is procured, tracked, and capped by a single team against a single budget line. Checkpoint storage accumulates quietly across dozens of runs, often in whatever bucket someone set up on day one, with no retention policy and no owner. By the time someone notices the bill, the fix is a cleanup project, not a budget adjustment.
The first mistake in estimating checkpoint storage is assuming a checkpoint is roughly the size of the deployed model. It is not close. A deployed model in bf16 stores two bytes per parameter. A training checkpoint has to store enough state to resume training exactly where it left off, and with the Adam optimizer, that means carrying two additional full precision tensors alongside the weights themselves.
Microsoft's ZeRO paper laid out the arithmetic the industry still uses to reason about this: mixed precision training with Adam requires roughly sixteen bytes of state per parameter once you add a fp32 master copy of the weights, fp32 first moment and second moment optimizer tensors, and the fp16 or bf16 working copy used for the forward and backward pass (Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Microsoft Research, 2020). A full checkpoint capturing the resumable training state, weights plus optimizer moments, typically lands around twelve bytes per parameter even before you count the working copy.
Run that arithmetic on a real model and the gap becomes concrete. A 70 billion parameter model deployed in bf16 is about 140 gigabytes. A full training checkpoint of that same model, with fp32 optimizer state included, is close to a terabyte. That is not a rounding difference. It is a six to seven times multiplier between what ships and what has to be stored to keep the run recoverable.
| Model Scale | Deployed Size (bf16) | Full Training Checkpoint |
|---|---|---|
| 7B parameters | 14 GB | ~84 GB |
| 70B parameters | 140 GB | ~840 GB to 1 TB |
| 405B parameters | 810 GB | ~4.9 TB |
The obvious cost lever is to checkpoint less often. In practice, large training clusters checkpoint frequently because they have to. At the scale of thousands of GPUs, hardware interruptions are a routine operating condition, not an edge case. Meta's technical report on the Llama 3 models describes frequent unplanned interruptions during the pretraining run on a 16,000 GPU H100 cluster, an environment where the sheer number of components in play means something is statistically likely to fail on any given day (Meta AI, The Llama 3 Herd of Models, 2024).
Checkpoint frequency is the insurance policy against losing GPU hours to that failure. A run that checkpoints every three hours loses at most three hours of compute to any single interruption. A run that checkpoints once a day can lose a full day of a multi million dollar GPU allocation to the same failure. The incentive runs entirely in one direction: checkpoint more often to protect the compute investment, and accept that storage grows as a direct consequence. This is the same asymmetry that defines the broader storage as Achilles heel pattern across AI infrastructure: the constraint nobody sizes for is the one working quietly in the background of every other decision.
Here is a worked model using the assumptions above, built to show where the bill actually accumulates. Take the 70 billion parameter model, a full resumable checkpoint of roughly 1 terabyte, and a 45 day pretraining run.
A reasonable retention policy keeps three rolling checkpoints for fault recovery, one milestone checkpoint per day for evaluation and rollback, and a final archived checkpoint at the end of the run. That works out to roughly 49 terabytes of checkpoint data stored across the life of the run: 3 terabytes in the rolling buffer, 45 terabytes across the daily milestones, and 1 terabyte archived at completion.
At standard hot object storage pricing around 23 dollars per terabyte per month, a rate consistent with published hyperscaler object storage list pricing, storing that 49 terabytes for the roughly 1.5 month duration of the run costs on the order of 1,700 dollars. Set against a GPU bill in the millions of dollars for a run at this scale, that number looks trivial in isolation. It is the wrong number to stop at.
The real cost shows up in multiplication. A lab running 20 concurrent training and ablation jobs at this scale in a given month is not storing 49 terabytes, it is storing roughly 980 terabytes, which is close to 22,500 dollars a month in storage for checkpoints alone, calculated at the same 23 dollar per terabyte per month rate. That is a real, recurring infrastructure line item that most GPU cost models never explicitly account for, and it exists independent of any decision about how long to keep the data once each run finishes.
| Checkpoint Class | Volume | Monthly Cost (at $23/TB) |
|---|---|---|
| Rolling fault recovery buffer (3 checkpoints) | 3 TB | $69 |
| Daily milestone checkpoints (45 days) | 45 TB | $1,035 |
| Final archived checkpoint | 1 TB | $23 |
| Total for one 45-day run | 49 TB | ~$1,127/mo while active |
The monthly figure above assumes checkpoints get cleaned up when a run ends. In most organizations, they do not. Milestone checkpoints from a completed run stay in the same hot bucket they were written to because deleting them requires someone to decide, with confidence, that no future evaluation or rollback will need them. Nobody wants to be the person who deletes the checkpoint that turns out to be needed six weeks later, so the default behavior is to keep everything, indefinitely, in the most expensive storage tier available.
This is where the bill stops being a fixed monthly number and starts compounding. Every month of new training runs adds its own checkpoint volume on top of everything that was never cleaned up from the months before. A team that started the year at 22,500 dollars a month in checkpoint storage and never implemented a retention policy is not still paying 22,500 dollars a month by December. It is paying a number that has grown with every run since January, on data that is overwhelmingly never read again after the first two weeks past a run's completion.
This pattern connects directly to what Russ Artzt described on the DataStorage.com Podcast about the shift in how infrastructure economics get evaluated as workloads move from experimentation to production: the tools that made sense at small scale become liabilities once volume compounds, and nobody notices until the bill forces the conversation.
The fix is not less checkpointing. It is tiered retention with an explicit time to live on every checkpoint class, enforced automatically rather than left to memory.
Rolling fault recovery checkpoints need hot storage with fast read and write, because they exist to get a training job back online in minutes after a hardware failure. That tier should be small by design, three to five checkpoints at most, and self pruning as new ones are written.
Milestone checkpoints used for evaluation during the run have a natural expiration: once the run concludes and the final model is validated, most of the daily milestones stop having any operational value. Moving them to a cooler storage tier after run completion, or deleting all but a handful of key evaluation points, removes the majority of the accumulated volume without losing anything a team will actually use again.
Archived final checkpoints are the one category worth paying to keep long term, since they represent a reproducible, resumable version of a model that shipped. Even here, egress matters more than most teams expect. Checkpoint archives get pulled for fine tuning, evaluation reruns, and incident investigation, sometimes onto a different cluster or a different provider than where they were written. Egress free storage such as Backblaze or Wasabi removes the penalty for that movement entirely, which matters more for a large archived checkpoint corpus than it does almost anywhere else in an AI infrastructure stack, since these are exactly the assets most likely to need to move between environments without warning.
A GPU budget without a checkpoint retention policy is not a complete GPU budget. It is a compute budget with an open ended storage liability attached to it.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
