Model Routing & Cascading

Model routing is the practice of sending each request to the least expensive model that can answer it well. A router makes that choice before any text is generated. A cascade lets a small model answer first and escalates to a larger one only when the draft fails a check.

Not every request needs your most expensive model. Classifying a support ticket, pulling a date out of an email or answering a known FAQ is easy work for a small model, and teams still send all of it to a frontier model to be safe. As of October 2026, Anthropic's and OpenAI's pricing pages both list a top tier at $10 per million input tokens and $50 per million output, and each vendor's smallest current model at $0.10 and $0.50. That is a 100x spread between models from one vendor, and routing is how you stop paying the top of it on work that never needed it.

Router or cascade#

A router predicts how hard a request is from the request alone and sends it to one model. A cascade runs the small model, inspects the draft, and pays for a second generation only when the draft fails. FrugalGPT formalised the cascade in 2023 and reported matching GPT-4 with up to 98 percent cost reduction on its benchmarks. On the router side, Hybrid LLM trained a difficulty predictor with a tunable quality threshold and made up to 40 percent fewer calls to the large model with no drop in response quality. Read both as upper bounds from benchmark traffic, because your saving depends on how much of your own traffic is easy.

The two trade the same thing in opposite directions. A router never pays twice, but it has to judge difficulty blind. A cascade judges a real answer, which is far easier, and pays an extra generation on every escalation. We recommend starting with a cascade, because checking a draft is a problem you can test, while predicting difficulty from a prompt needs training data you do not have yet. A router earns its place later, for traffic where a second generation would break the latency budget.

How a gate decides what is hard#

The gate is where routing succeeds or fails, and the signals for it come in an order from most to least trustworthy.

Deterministic checks come first. A draft that fails JSON schema validation under structured outputs, leaves out a required citation, or fails a unit test is proven inadequate, with no probability involved. These checks cost nothing to run and do not drift when the model changes.

The model's own confidence comes next. Token logprobs are the usual signal: when the average log probability of the answer, or of the few tokens that carry it, falls below a threshold, escalate. Self-reported confidence is noisy, which is why AutoMix has the small model check its own draft with a few-shot verification prompt and then lets a learned policy decide whether to escalate.

Learned routers sit before generation. RouteLLM trains a classifier on human preference data to predict whether the small model's answer would satisfy the user, and its abstract reports cutting cost by over 2 times in certain cases without hurting quality. The saving depends on the task, so measure it on your own traffic. A separate decision model can also make the call. TypeSafe's Jev answers typed questions about a state with a probability for each option, so "which model handles this turn" becomes a choice with a threshold you can set. Our guide on introducing Jev sets out the order we recommend for giving it that decision, starting in shadow.

Heuristics fill the rest: input length, task type, whether the prompt holds code, the customer tier. They are free and brittle, and they are useful for sending the obviously easy and obviously hard requests past the gate entirely.

A typical composite for extraction runs the small model with a JSON schema, escalates at once on a validation failure, then applies a logprob threshold to the two fields that drive billing downstream. Only drafts that pass both checks ship.

The spread changes the arithmetic#

Take a request of 1,000 input and 500 output tokens at the October 2026 list prices above. The top tier costs $0.01 in plus $0.025 out, $0.035 a call. The smallest tier costs $0.00035. A cascade runs the small model on every request and escalates 20 percent of them, so it costs 0.00035 + 0.2 × 0.035, about $0.0074 a request, or 21 percent of sending everything to the top tier. At 100,000 requests a day that is roughly $735 instead of $3,500.

At a 100x spread the small model's own cost is a rounding error, and the escalation rate is the bill. Each 10 points of escalation adds about 10 percent of the frontier-only cost. That makes the escalation rate the first number to track in a cascade, and it is why a gate that leans toward escalating costs less than it seems to: the expensive mistake is a wrong answer that passed, not a correct one that went up a tier.

Cost per success, against a floor#

Report cost per success, which is total spend divided by the answers that met the quality bar, and report it next to that bar. A leaking gate shows why both numbers matter. If end-to-end accuracy slides from 95 to 88 percent, cost per call does not move, and cost per success rises by only about 8 percent. The 7 points of answers that are now wrong are the real cost, and no ratio shows them. Treat the quality floor as a constraint the cascade has to hold, then minimise cost under it.

The gate that stops firing#

The characteristic failure is a confident wrong answer from the small model that passes the gate. Easy requests stay fine and the hard, high-value ones decay, so the average barely moves. Calibration is the cause. Logprobs are imperfectly calibrated on instruction-tuned models, and a threshold tuned for one model version drifts when the vendor ships an update.

We have tuned against the same failure outside routing. On DocVerse, our ESG filing pipeline, every extracted field carries a confidence score that works as a routing signal: low-confidence fields go to a person for review instead of downstream as fact. For scanned and irregular filings, the tuning work was mostly making sure low confidence actually fires, rather than producing a confident wrong parse. A model cascade is the same design with a larger model where the reviewer sits, and it needs the same test: sample the answers that passed, not only the ones that escalated.

Tune the threshold against a labelled eval set, and re-tune it at every model swap. On the trademark drafting system, production logging compares the base and fine-tuned models on the same inputs, so a model change shows up as a score movement. A cascade gate needs that comparison more than most components do. Watch the escalation rate as a production metric too: a sudden drop usually means the gate broke, not that traffic got easier.

Start with a cascade and a schema check#

A deterministic gate in front of a logprob threshold captures most of the saving with the least machinery, and it fits batch and asynchronous work, where a second generation in series does not matter. Latency-sensitive paths are the exception. An escalated request pays for two generations back to back, which hurts the p95, the latency that 95 of every 100 requests beat, so those paths want a router that decides before generating. Train a learned router only after the cascade has logged a few weeks of which requests escalated, because that log is its training data. At low volume, one well-chosen model is the simpler answer: at 1,000 requests a day the example above saves about $28 a day, which the labelled eval set and a gate to maintain can easily outweigh.