Model Routing & Cascading

Not every request needs your most expensive model. Classifying a support ticket, extracting a date, or answering a FAQ is trivial for a small model, yet teams route everything to a frontier model "to be safe" and pay the premium on requests that never needed it. The gap is wide in mid-2026: frontier tiers list around $10 per million input tokens and $50 per million output, while the same vendor's small model runs about $1 and $5, a roughly 10x spread that repeats across providers. Model routing and cascading exploit that gap by matching each request to the cheapest model that can actually handle it.

One decision, two places to make it#

A router decides before any generation happens. It looks at the incoming request, predicts how hard it is, and dispatches it to a small or large model accordingly. A cascade decides after the fact: it runs the cheap model first, inspects the draft answer against a quality gate, and escalates to a stronger model only when the gate fails. FrugalGPT formalized the cascade in 2023, asking the smallest model first and walking up the ladder until an answer scores well enough; it reported matching GPT-4's accuracy on several benchmarks with up to 98 percent cost reduction. On the router side, the Hybrid LLM work trained a difficulty predictor with a tunable quality threshold and cut calls to the large model by up to 40 percent with no measured quality drop.

The tradeoff is symmetric. Routers never pay twice, but they must judge difficulty blind, before seeing any answer. Cascades get to judge an actual answer, which is much easier, but pay for an extra generation every time they escalate. Production systems usually combine them: heuristics route the obviously easy and obviously hard traffic up front, and a cascade with a confidence gate handles the ambiguous middle.

How systems decide what is hard#

Everything hinges on the difficulty signal. Four families, cheapest first.

Heuristics. Input length, task type, the presence of code or math, a regex on the prompt, the customer tier. Free and brittle, but a rule as dumb as "anything under 30 tokens that matches a known FAQ intent goes to the small model" routinely covers a third of traffic correctly.

Learned routers. RouteLLM trains a small classifier on human preference data, predicting whether the weak model's answer would satisfy the user; it reports cutting costs by more than 2x on standard benchmarks while preserving quality, with the router itself adding only milliseconds.

The model's own confidence. Token-level logprobs are the classic gate: when the average log probability of the answer, or of the specific answer-bearing tokens, falls below a tuned threshold, escalate. AutoMix instead has the small model run a few-shot self-verification pass on its own draft, then uses a POMDP-based router to decide escalation, precisely because self-reported confidence is noisy.

Deterministic checks. A draft that fails JSON schema validation under structured outputs, omits a required citation, or fails a unit test is proven inadequate, no probability needed. These gates are free, exact, and the right first layer in any cascade.

A concrete composite: an extraction pipeline runs the cheap model with a JSON schema, escalates instantly on a validation failure, then applies a logprob threshold on the two fields that drive downstream billing. Only drafts passing both gates ship.

Cost per success, not cost per call#

The framing that makes routing decisions easy is cost per success: total spend divided by the number of answers that actually met the quality bar. Work the numbers for a typical request of 1,000 input and 500 output tokens. The frontier model costs $0.01 in plus $0.025 out, $0.035 per call. The small model costs $0.0035, a tenth as much. A cascade that keeps 80 percent of traffic at the cheap tier and escalates 20 percent costs 0.0035 + 0.2 x 0.035, about $0.0105 per request, 70 percent below frontier-only. At 100,000 requests a day that is roughly $1,050 instead of $3,500, on the order of $900,000 a year.

The metric matters because it keeps quality in the denominator. If the cheap model is right on the traffic it keeps and the gate catches the rest, cost per success drops nearly threefold. If the gate leaks and end-to-end accuracy slides from 95 to 88 percent, cost per call still looks great while cost per success quietly rises and the product gets worse. Always report both.

The quality-floor risk#

The characteristic failure is the confident-but-wrong cheap answer that sails through the gate. The degradation is concentrated and silent: easy requests stay fine, and it is exactly the hard, high-value requests that decay. Calibration is the root cause. Logprobs are imperfectly calibrated on instruction-tuned models, and a threshold tuned for one model version drifts the day you swap in an update, so every model change demands a re-tune.

Treat the gate as a first-class component. Tune its threshold against a labeled eval set rather than vibes, track the escalation rate as a production metric (a sudden drop usually means the gate broke, not that traffic got easier), and periodically sample gate-passing answers for review by humans or an LLM-as-judge. Bias toward escalation on high-stakes routes. Finally, budget the latency: a cascade pays a serial second generation on escalated requests, so p95 suffers, which is why cascades fit batch and async workloads best while latency-sensitive paths prefer a pre-generation router that adds one millisecond-scale hop.

Practical takeaways#

Start with a cascade, not a learned router: cheap-first plus a deterministic gate (schema validation, then a logprob threshold) captures most of the savings with the least machinery. Measure cost per success against an explicit quality floor, and re-tune the gate at every model swap. Train a learned router only after the cascade has logged a few weeks of which requests actually escalate; that log is its training data. And keep one standing rule: when in doubt, escalate. The frontier call is the cheapest insurance in the stack.