Speculative Decoding

Speculative decoding attacks the core slowness of LLM generation: tokens come out one at a time, and each one requires a full pass over the big model's weights. The insight, published independently by Google (Leviathan et al., 2022) and DeepMind (Chen et al., 2023), is that many tokens are easy to predict. A tiny, fast model can guess a run of them ahead, and the big model can check several guesses in a single pass instead of generating them one by one. Done right, it cuts latency 2x to 3x while producing exactly the same distribution of text the big model would have produced alone.

Why decoding is slow in the first place#

The decode phase of prefill vs decode is memory-bandwidth bound. To emit one token, the GPU must stream every model weight from memory through the compute units: for a 70B model in FP16 that is roughly 140 GB of reads per token, which on a 3.3 TB/s H100 pair puts a hard floor of about 40 ms per token no matter how trivial the token is. The arithmetic units sit mostly idle while bytes move.

That idleness is the opening. Running the big model over five tokens at once costs barely more than running it over one, because the weights stream past either way. The problem is that autoregressive generation cannot normally see five tokens ahead: token N+2 depends on token N+1. Speculative decoding manufactures the lookahead by letting a cheap model guess it.

The draft and verify loop#

The setup uses two models: a small draft model (often 10x to 20x smaller, sharing the target's tokenizer) and the large target model you actually want output from. Each round has three steps. First, the draft model autoregressively proposes a short run of candidate tokens, say four; this is fast because the draft is tiny. Second, the target model runs one forward pass over all four candidates in parallel, producing its own next-token distribution at every position. Third, the system accepts the longest prefix of candidates the target agrees with, rejects the rest, and samples one corrected token from the target at the rejection point.

Walk the example through. The draft proposes "the cat sat down". The target's parallel pass agrees with "the", "cat", and "sat" but would have said "on" instead of "down". The round emits four tokens, "the cat sat on": three accepted plus the target's own correction, all for the price of one target forward pass plus four cheap draft steps. A bare target model would have needed four expensive passes for the same output.

Why the output distribution is preserved#

The acceptance rule is a form of rejection sampling, and this is what separates speculative decoding from simply using a smaller, worse model. At each position, a draft token with draft probability q and target probability p is accepted with probability min(1, p / q). On rejection, the replacement is sampled from the normalized residual distribution max(0, p minus q). Both papers prove that the tokens emitted by this procedure are distributed exactly as if the target model had sampled them alone, at any temperature. With greedy decoding the rule degenerates to a simple check: accept while the draft token equals the target's argmax.

The practical consequence: there is no quality knob to tune and no eval regression to hunt for. Speculative decoding is lossless by construction (vLLM documents this up to floating-point and batching nondeterminism). You are spending spare compute, not accuracy.

The speedup math: acceptance rate is everything#

Let a be the per-token acceptance rate and k the draft length. The expected tokens emitted per target pass is (1 minus a^(k+1)) / (1 minus a). At a = 0.8 with k = 4, that is 3.36 tokens per pass; at a = 0.5 it drops to 1.94; at a = 0.3, just 1.4, which the draft overhead can entirely eat. Measured end to end, Leviathan et al. report 2x to 3x on T5 models and Chen et al. 2x to 2.5x on the 70B Chinchilla, numbers that have held up in production systems since.

Three conditions govern whether you land at the good end of that range. First, draft alignment: the draft must share the target's tokenizer and approximate its distribution, which is why distilled or same-family small models (Llama 3 1B drafting for 70B) work well and random small models do not. Second, workload entropy: grounded, predictable text such as code, JSON, summaries, and RAG answers accepts at high rates, while high-temperature creative writing accepts poorly. Third, draft length: longer drafts amplify wins at high acceptance but multiply wasted work at low acceptance; production systems typically use 3 to 5.

Where it breaks: batch size#

Speculative decoding converts idle compute into latency savings, so it needs idle compute to exist. At batch size 1 to moderate concurrency, decode is memory-bound and verification is nearly free. But under continuous batching at high QPS, the GPU's arithmetic units are already busy serving other requests, and every rejected draft token is real compute burned for nothing. vLLM's docs are explicit that the feature targets medium-to-low QPS, memory-bound workloads; at saturation it can reduce total throughput even while individual streams feel faster. The KV cache for the draft model also takes its own slice of GPU memory, shrinking the batch the target can hold.

Modern variants#

The two-model recipe has largely given way to single-model designs that remove the alignment problem. Medusa (Cai et al., 2024) bolts extra lightweight decoding heads onto the target itself; each head predicts one future token, and tree attention verifies many candidate continuations at once, for 2.2x to 3.6x speedups with no separate draft model to host. EAGLE (Li et al., 2024) instead runs a small autoregressive head over the target's hidden features rather than its tokens, reaching 2.7x to 3.5x on LLaMA2-Chat 70B with acceptance rates near 80%; its successors EAGLE-2 and EAGLE-3 push further and are the default choice in most serving stacks today. At the zero-cost end, self-speculation methods like n-gram and suffix decoding draft by copying strings already present in the prompt, which works surprisingly well for editing, extraction, and RAG, where the answer largely restates the context. vLLM ships all three families behind a config flag.

Practical takeaways#

Reach for speculative decoding when per-stream latency matters and your GPUs are not compute-saturated: chat products, coding assistants, agent loops. Measure the acceptance rate before trusting any speedup claim, since it is workload-specific and everything follows from it. Prefer an EAGLE-style trained head when your serving stack supports it, try free n-gram drafting first for grounded workloads, and skip the technique entirely for throughput-oriented offline batch jobs, where filling the batch is the better use of the same FLOPs.