When one frontier model handles everything, every request, from classifying a ticket to refactoring a service, is priced as if it were the hardest problem the company has.
A platform team ships an agent on one frontier model because it was the fastest way to get to production. Six months later the agent is handling 10 million requests a month, and the finance team wants to know why the inference line went from a rounding error to the third-largest item on the cloud bill. The honest answer is architectural: every request, from "classify this ticket" to "refactor this service," is priced as if it were the hardest problem the company has.
That pattern is now the exception rather than the rule. An a16z survey of 100 Global 2000 CIOs, reported by eMarketer in February 2026, found that 81 percent use three or more model families in testing or production, up from 68 percent a year earlier. The question for infrastructure leaders is no longer whether to run multiple models. It is whether the layer deciding which model handles which request is engineered, measured, and governed, or improvised inside application code.
This guide covers what a production routing layer actually does, how the main strategies compare, what the savings look like on real 2026 list prices, and the hidden costs that vendor case studies leave out.
As of October 2026, Anthropic lists Claude Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens, while Claude Fable 5.1 sits at $10 and $50. On a typical request of 6,000 input tokens and 400 output tokens, that is $0.0008 versus $0.0800: a 100x difference for the same API call shape. OpenAI made the same move explicit when it launched GPT-5.6 on July 9, 2026 as three named tiers (Sol, Terra and Luna), describing them as durable capability tiers that advance on their own cadence. Vendors are now shipping product lines designed to be routed across.
Perplexity, which sells a multi-model enterprise product and therefore has an interest in this trend, reported that two models accounted for more than 90 percent of its enterprise usage in January 2025, while by December four models each held more than 10 percent and the leader captured only about 23 percent of queries. The independent signal points the same way. The OpenRouter and a16z State of AI study, built on more than 100 trillion tokens of traffic, found that open-weight models reached roughly one-third of token volume by late 2025, and that no single open model held more than about a quarter of open-source tokens. A model you hardcode in January may not be the best, or even the cheapest acceptable, option by July.
Gartner predicted in March 2026 that inference on a one-trillion-parameter model will cost providers more than 90 percent less by 2030 than in 2025. In the same release, Gartner warned that token consumption is rising faster than token costs fall, so total inference spend is expected to increase, and advised that routine, high-frequency tasks be routed to smaller models while frontier inference is gated for complex, high-margin work. That is the routing thesis in an analyst's own words. Menlo Ventures put enterprise generative AI spend at $37 billion in 2025, 3.2 times the 2024 figure. Unit prices fall; bills rise. The routing layer is where you decide how much of that growth you actually pay frontier rates for.
A demo router is a classifier and an if statement. A production router is a control plane with five distinct jobs, and most incidents trace back to teams that built the second one and skipped the rest.
There are four practical patterns, and mature stacks usually combine at least two.
Route by endpoint, feature or prompt template: the ticket classifier always goes to the small tier, the code-review agent always goes to the large one. Rules add almost no latency, are fully debuggable, and capture most of the safe savings because most production traffic is already segmented by feature. Their weakness is that they cannot see difficulty inside a single endpoint.
RouteLLM, the open-source framework from UC Berkeley and Anyscale published at ICLR 2025, trained routers on preference data and reported cost reductions of over 85 percent on MT Bench, 45 percent on MMLU and 35 percent on GSM8K versus using GPT-4 alone, while retaining 95 percent of GPT-4 quality. Note the spread across benchmarks: savings depend entirely on how much of your traffic is easy. Treat 85 percent as a ceiling observed on one benchmark, not a forecast for your workload.
Send everything to the cheap tier first and escalate only when verification fails. Cascades work best where correctness is machine-checkable: structured extraction, code that must compile and pass tests, answers that must cite retrieved documents. They cost latency on escalated requests, so they suit asynchronous and agentic work better than interactive chat.
The hyperscalers now sell routing as a feature. Amazon Bedrock Intelligent Prompt Routing routes between models within one model family and AWS claims cost reductions of up to 30 percent without compromising accuracy. Microsoft Foundry made its model router generally available in November 2025 with Cost, Balanced and Quality modes and built-in failover across a broader catalog. Gateways such as OpenRouter, which routes across more than 300 models from over 70 providers, and LiteLLM give you a single OpenAI-compatible endpoint across vendors. The trade-off is control: managed routers optimize against the vendor's catalog, and same-family routing cannot move traffic to a cheaper competitor.
| Strategy | Added latency | Typical savings | Best for |
|---|---|---|---|
| Static rules and tiering | Negligible | Most safe savings | Feature-segmented traffic |
| Learned classifier | One small call | 35% to 85% | Mixed difficulty in one endpoint |
| Cascade with escalation | High on escalation | Workload dependent | Checkable, async tasks |
| Managed router | Low | Up to 30% | Fast adoption in one cloud |
Assume an agent platform serving 10 million requests a month at 6,000 input tokens and 400 output tokens per request, which matches the average prompt and completion sizes OpenRouter reported for late 2025. All prices are Anthropic list prices as of October 2026, before caching or batch discounts.
Running every request on Claude Opus 5.5 costs $320,000 a month. Running everything on Claude Fable 5.1 costs $800,000. Now route 60 percent of traffic to Haiku 5.5, 30 percent to Sonnet 5.5, 8 percent to Opus 5.5 and 2 percent to Fable 5.1. Model spend drops to $94,400. Add a Haiku-based classifier that reads the first 1,000 tokens of each request ($1,025) and assume 5 percent of Haiku-tier requests fail verification and are re-run on Sonnet ($4,800). The all-in routed bill is $100,225, a 69% reduction against the all-Opus baseline.
| Scenario | Per request | Monthly | vs all-Opus |
|---|---|---|---|
| All on Claude Fable 5.1 | $0.0800 | $800,000 | +150.0% |
| All on Claude Opus 5.5 | $0.0320 | $320,000 | Baseline |
| Routed, models only | $0.0094 | $94,400 | -70.5% |
| Routed, all-in | $0.0100 | $100,225 | -68.7% |
The more useful insight is where the money goes after routing. Haiku carries 60 percent of traffic but only 5.1% of model spend. Fable carries 2 percent of traffic and 16.9% of spend. Once routing is in place, the next dollar of savings is almost never in the cheap tier. It is in tightening which requests are allowed to reach the top one.
Anthropic notes that Claude 4.7 and later models use a tokenizer that produces approximately 30 percent more tokens for the same text than earlier models. Tokenizers differ across vendors too. A router that compares list prices per million tokens can pick the model that looks cheaper and costs more. Route on measured cost per completed task from your own logs, not on rate cards.
Prompt caching is one of the largest discounts available: on Claude Opus 5.5 and Sonnet 5.5 a cache hit costs 5 percent of the standard input price. Caches are per model. A router that bounces turns of the same conversation or agent loop between models throws that discount away on every hop. Make routing sticky at the session or task level, and include the model version in the cache key.
Haiku 5.5 is priced at $0.10 per million input tokens only for prompts up to 100,000 tokens; above that the rate rises fivefold to $0.50. A long-context request routed to the "cheap" tier can cost more than expected. The router has to be price-aware at the token level, not just the model level.
Every classifier call adds latency before the first token. Every routing change needs an evaluation run before it ships, which means maintaining a labeled test set drawn from production traffic. And someone has to own the policy when a new model launches every few weeks. Budget engineering time for the router as a product, or the savings erode as the model landscape moves underneath it.
It is tempting to assume routing across clouds creates an egress problem. For API-served models, it mostly does not. At roughly four characters per token, a 6,000-token prompt is about 24 KB, so 10 million requests move around 240 GB a month. At the AWS internet egress rate of $0.09 per GB, that is about $22. The payload is not the problem.
The problem appears in two places. First, the self-hosted tier. Open-weight models served about a third of tokens on OpenRouter by late 2025, and many teams add an open-weight tier on rented GPUs from neoclouds such as CoreWeave, Nebius or Lambda Labs to cut per-token cost further. Retrieval corpora, embeddings and fine-tuning data then need to sit next to those GPUs, and moving a multi-terabyte corpus out of a hyperscaler to follow cheaper capacity is where egress fees bite. Second, the observability store. Logging full prompts and completions for evaluation at this scale produces hundreds of gigabytes a month, and evaluation jobs pull that data repeatedly into whichever cloud has GPU capacity that week.
Both are arguments for keeping routing logs, evaluation sets and retrieval corpora in S3-compatible object storage with zero or low egress, such as Backblaze B2, Wasabi or Cloudflare R2, so the data can follow the compute without a toll on every move. We discussed why GPU capacity shifts between providers, and why most engineers still do not know neoclouds exist, with Russ Artzt, co-founder of CA Technologies, on the DataStorage.com Podcast. Sunny Smith of Massed Compute made the data gravity point directly in Episode 7: storage has to be adjacent to the GPU, and moving a petabyte between data centers is prohibitive.
To size the opportunity before you build, run your current traffic mix through our Cloud Cost Calculator alongside the GPU Price Explorer if a self-hosted tier is on the table.
The model you pick matters less every quarter. The layer that picks for you is becoming the most important piece of AI infrastructure you own.
Compare AWS, Google Cloud, Azure, and alternatives like Backblaze B2 Discover how much you could save in seconds
