A coding agent that ends its task with 830,000 tokens in context has re-sent 17.6 million. Which price you pay for that depends almost entirely on caching.
A coding agent that finishes its task with 830,000 tokens of context in its window has not processed 830,000 tokens. Across 40 turns it has re-sent its growing history every time, for a total of 17.6 million input tokens. At Claude Opus 5.5 list prices that one session costs $72.00 uncached. Cached properly, it costs $9.10. Most teams find out which of those two numbers they are paying when the invoice lands.
That gap is the real story of million-token context windows. The headline feature is how much a model can read. The infrastructure consequence is how much your stack has to hold, move, cache and re-send, and who pays for it. This guide breaks down what long context does to GPU memory, API bills and storage, with worked numbers, so you can decide when a million tokens is worth it and how to stop paying for the same context forty times.
Three things changed this year. First, pricing flattened. Anthropic's pricing documentation (2026) states that Claude 4.6 and later models, except Haiku 5.5, bill the full 1M-token window at standard rates, so a 900K-token request costs the same per token as a 9K-token request. Google's Gemini API pricing page (October 2026) still charges Gemini 3.1 Pro $2.00 per million input tokens at or under 200K and $4.00 above it, with output rising from $12.00 to $18.00. The cliff that used to force discipline is gone on one major platform and intact on another.
Second, the hardware industry started treating context as its own storage class. NVIDIA announced BlueField-4 STX at GTC in March 2026, with a context memory platform built specifically to hold KV cache in flash, and said partner systems would ship in the second half of 2026. CoreWeave, Lambda Labs, Nebius, Vultr and Oracle Cloud Infrastructure were named as early adopters.
Third, the flash that context memory runs on got expensive. TrendForce (September 30, 2026) forecasts NAND contract prices up another 15 to 20 percent quarter over quarter in 4Q26 and names enterprise SSD as the only memory category where price growth is accelerating, driven by AI inference and agentic caching.
Every token a transformer has already read is stored as keys and values in the KV cache (key-value cache), so the model does not recompute the whole history to generate the next token. The KV cache grows linearly with context length and lives, by default, in GPU high-bandwidth memory (HBM).
The arithmetic is simple. A 70B-class model built on the Llama 3.1 70B architecture (80 layers, 8 key-value heads, head dimension 128) stores about 328 KB of KV cache per token at FP16 precision. That is roughly 42 GB at 128K tokens and 328 GB at 1M tokens, for a single session. Quantizing the cache to FP8 halves it, to about 164 GB.
| Context | FP16 | FP8 | Share of one 270 GB GPU |
|---|---|---|---|
| 128K tokens | 42 GB | 21 GB | 16% |
| 200K tokens | 66 GB | 33 GB | 24% |
| 500K tokens | 164 GB | 82 GB | 61% |
| 1M tokens | 328 GB | 164 GB | 121% |
An 8-GPU NVIDIA B300 node in the Weka and Nebius benchmark (2026) carried about 270 GB of HBM3e per GPU. Even with that much memory, the benchmark authors note that HBM holds only a few dozen long-context sessions before it starts evicting. Once a session's cache is evicted, its next turn pays the full prefill cost again: the GPU recomputes context it has already seen. NVIDIA's own technical blog (2026) makes the same point: KV cache capacity grows in proportion to context, while the compute needed to rebuild it grows much faster.
This is why long context is not just a model feature. A provider serving million-token sessions either buys far more HBM per user, accepts slow and expensive recompute, or adds a cache tier underneath the GPU. All three cost money, and all three end up in your price or your latency.
Long-context costs rarely come from one giant request. They come from re-sending. Agents and chat sessions are stateless at the API: each turn ships the full history back to the model. A session that grows steadily from 50K to 830K tokens over 40 turns sends 17.6 million input tokens in total, more than 21 times its final context size.
Remove the surcharge at 200K and teams stop trimming. That is good for capability and bad for spend, the same pattern we see across AI infrastructure: a cheaper token is not a cheaper bill. Note too that Anthropic says Claude 4.7 and later models use a new tokenizer that produces roughly 30 percent more tokens for the same text, so a per-token price comparison against older models understates the change.
The table below models one such session across current list prices, then scales it to a 200-engineer team running five sessions a day.
| Scenario | Pricing basis | Per session | Monthly |
|---|---|---|---|
| Claude Opus 5.5, no caching | $4 in / $20 out | $72.00 | $1,512,000 |
| Claude Opus 5.5, cached, 6 cache misses | write 1.25x, read 0.05x | $23.79 | $499,590 |
| Claude Opus 5.5, cached, no misses | write 1.25x, read 0.05x | $9.10 | $191,100 |
| Gemini 3.1 Pro, no caching | $2 / $4 in, $12 / $18 out | $69.82 | $1,466,220 |
| Claude Sonnet 5.5, cached, no misses | write 1.25x, read 0.05x | $4.55 | $95,550 |
Prompt caching is what turns $72.00 into $9.10. On Claude Opus 5.5 and Sonnet 5.5, Anthropic prices a cache hit at 5 percent of the base input rate, a 5-minute cache write at 1.25 times base, and a 1-hour write at 2 times base. The catch is expiry. A developer who steps away for longer than the cache lifetime comes back to a cold cache, and the next turn rewrites the entire context at the write premium. In our model, just 6 misses out of 40 turns lift the session from $9.10 to $23.79, and the monthly bill from $191,100 to $499,590.
Explicit caches can also carry a holding charge. Google lists Gemini 3.1 Pro context caching storage at $4.50 per million tokens per hour. Holding an 830K-token context for an 8-hour workday costs about $29.88 in storage alone, before a single token is read. That is reasonable for a shared corpus queried by hundreds of users and wasteful for one engineer's session.
Treat cache hit rate as a FinOps metric, not an engineering detail. It belongs on the same dashboard as reserved instance coverage and egress, and your Cloud Cost Calculator baseline should include it.
Teams running open-weight models on their own GPUs or a neocloud face the memory problem directly. NVIDIA describes a four-tier hierarchy: GPU HBM, host DRAM, local NVMe, and shared network storage, with its new context memory tier slotted between local and shared storage. Each step away from the GPU adds capacity and latency.
The most detailed public test of an external KV tier is the Weka and Nebius benchmark published on StorageNewsletter on October 5, 2026. It replayed more than 5,000 real agentic coding sessions, with contexts approaching 900K tokens, against DeepSeek-V4-Pro on a single 8-GPU B300 node for eight hours per configuration. Running from HBM alone, the node recomputed 60 percent of all context it processed. With Weka's Augmented Memory Grid pooling NVMe across the cluster, recompute fell below 7 percent, output rose from 46K to 169K tokens per GPU-hour, and median time to first token fell from 5.7 seconds to 1.05 seconds.
Weka sells the cache tier and Nebius sells the GPUs, so both benefit from these numbers. The baseline is HBM-only because, by the authors' account, no DRAM offload implementation supported DeepSeek V4 at the time of testing, which flatters the result against the most common cheaper alternative. NVIDIA's own claim of up to 5 times more tokens per second for STX is also a vendor figure. We found no independent benchmark yet. The direction is credible and the mechanism is sound; the multiplier you will see on your workload is unproven. Ask any provider for results on your own session traces before you pay for a context tier.
Context memory is a storage purchase. It just does not look like one, because it sits inside the inference stack rather than in the data lake. TrendForce (2026) projects enterprise SSD bit demand to grow more than 80 percent year over year in 2026 as cloud providers expand inference, and attributes part of that growth to agentic AI driving real-time retrieval and caching. Sunny Smith, founder and CTO of Massed Compute, told the DataStorage.com Podcast (Episode 7) that NVMe prices roughly tripled and that he expects scarcity to last until the end of 2027.
Two architecture rules follow. First, KV cache is disposable. It can always be recomputed, so it does not need the durability, replication and data protection of enterprise storage, a point Supermicro makes in its own BlueField-4 materials. Do not pay premium durable-storage rates for it. Second, it must sit next to the GPU. Context memory that lives in another region or another provider is useless at inference speed. This is data gravity at its sharpest: your context tier, like your training data, is landlocked to wherever your GPUs run.
For GPU buyers, that adds a vetting question to the list. After "do you own the GPU?", ask: "where does my KV cache go when HBM fills, and what am I charged for it?" A provider without an answer will hand you recompute at GPU-hour rates. Compare that line item across providers before signing, using live pricing in the GPU Price Explorer.
Paying to hold a million tokens assumes the model uses them well. The evidence says it often does not. Chroma's Context Rot study (2025) tested 18 frontier models and found that every one degraded as input length grew, often well before the window filled, and that semantically similar distractors made it worse. Chroma builds retrieval infrastructure and has a stake in that finding, but it matches the earlier Lost in the Middle research from Stanford (2023), which found models retrieve facts at the start and end of a long context far better than in the middle.
The infrastructure implication is uncomfortable: some of the context you are paying to store, cache and re-send is lowering answer quality. Bigger is not automatically better, and the cheapest token is the one you never send.
Use this sequence before you standardize on long context for any workload.
Log input tokens per request and report p50, p95 and the share of requests above 200K. Most teams find a long tail of a few sessions driving most of the spend.
If many requests cross 200K, flat-rate pricing on Claude 4.6 and later beats tiered pricing on Gemini 3.1 Pro for input. If requests are short and high volume, watch for the opposite cliff: Claude Haiku 5.5 charges five times more per input token once a prompt passes 100K.
Target a cache hit rate above 90 percent on long agent sessions, set cache lifetimes to match how people actually pause, and alert when hit rate drops.
Summarize or prune history at a fixed threshold, and retrieve documents on demand instead of preloading them. This cuts cost and, given the context rot evidence, may improve quality.
Model HBM per session, decide whether you will recompute or offload, and price NVMe at 2026 rates rather than last year's.
| Workload | Typical context | Strategy |
|---|---|---|
| Customer support chat | Under 32K | Short context, no long-context tier needed |
| RAG over a document library | 32K to 128K | Retrieve on demand, cache the system prompt |
| Agentic coding sessions | 200K to 900K, growing per turn | Prompt caching mandatory, compaction threshold, track hit rate |
| Whole-repository or contract-bundle analysis | 500K to 1M, one shot | Flat-rate long-context model or batch API |
| Self-hosted multi-tenant agents | Hundreds of long sessions | Budget an NVMe KV tier adjacent to the GPUs |
The window got bigger and the token got cheaper. The bill is decided by how many times you send the same context, and where you keep it in between.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
