Speculative Decoding

Speculative decoding is an inference technique that makes a large language model answer faster without changing what it says. A small, fast draft guesses the next few tokens, and the large model, the target, checks all of those guesses in a single pass instead of producing them one at a time. It keeps the run of guesses it agrees with and supplies its own token where it disagrees. Google (Leviathan and colleagues, 2022) and DeepMind (Chen and colleagues, 2023) published the method independently, and both proved the output follows exactly the distribution the target would have produced alone.

The speedup is real, two to three times in the original papers, but it is borrowed from spare GPU capacity. When the server is busy, there is no spare capacity to borrow, and the technique can make the whole system slower. Whether to turn it on is a question about your traffic, not about the model.

Why decoding is slow in the first place#

Generating text has two phases, described in prefill vs decode, and the decode phase is limited by memory bandwidth rather than arithmetic. To emit one token, the GPU reads every weight of the model from memory. A 70B model in 16-bit precision is about 140 GB of weights. Split across two H100s, each reading its half at about 3.35 TB/s, that sets a floor of roughly 21 ms per token however trivial the token is, while the arithmetic units sit mostly idle.

That idle arithmetic is the opening. Running the target over five tokens at once costs barely more than running it over one, because the weights stream past either way. Ordinary generation cannot look five tokens ahead, since each token depends on the one before. Speculative decoding supplies the look-ahead by letting a fast model guess it.

How speculative decoding drafts and verifies#

The setup uses two models: a much smaller draft, sharing the target's tokenizer, and the target whose output you want. Each round has three steps. The draft proposes a short run of tokens, say four, which is fast because the draft is small. The target runs one forward pass over all four in parallel and produces its own prediction at every position. The system keeps the longest prefix the target agrees with, discards the rest, and takes the target's own token at the first disagreement.

Walk the example through. The draft proposes "the cat sat down". The target's pass agrees with "the", "cat" and "sat" but would have said "on" rather than "down". The round emits four tokens, "the cat sat on": three accepted plus the target's correction, for the price of one target pass and four quick draft steps. The target alone would have needed four full passes.

Why the output does not change#

The acceptance rule is a form of rejection sampling, and it separates speculative decoding from simply serving a smaller model. At each position, a draft token that the draft rated with probability q and the target with probability p is accepted with probability min(1, p / q). On rejection, the replacement is drawn from the leftover distribution max(0, p minus q), renormalised. Both papers prove the emitted tokens are distributed exactly as if the target had sampled them, at any temperature. With greedy decoding the rule reduces to accepting while the draft's token matches the target's top choice.

So there is no quality setting to tune. As of October 2026, vLLM's documentation calls the standard method theoretically lossless up to the precision of hardware arithmetic, and warns that batch size can shift output probabilities slightly. Expect the same small run-to-run variation you already see from batched serving, not a quality drop.

The speedup depends on the acceptance rate#

Let a be the chance each draft token is accepted and k the draft length. The expected tokens per target pass is (1 minus a^(k+1)) / (1 minus a). At a = 0.8 with k = 4 that is 3.36 tokens per pass. At a = 0.5 it is 1.94, and at a = 0.3 about 1.4, which the draft's own cost can cancel out. End to end, Leviathan and colleagues measured 2x to 3x on T5-XXL, and Chen and colleagues 2x to 2.5x on the 70B Chinchilla.

Three things set the acceptance rate. The draft has to share the target's tokenizer and approximate its predictions, which is why a small model from the same family works and an unrelated small model does not. The text has to be predictable: code, JSON, summaries and answers grounded in retrieved documents accept well, and creative writing at a high temperature accepts poorly. And the draft length has to suit the rate, because longer drafts gain more when acceptance is high and waste more when it is low.

Bigcircle recommends measuring the acceptance rate on your own prompts before trusting any published speedup, because every number above follows from it and it varies by workload.

Where it breaks: a busy server#

The technique turns idle arithmetic into lower latency, so it needs idle arithmetic. At low concurrency, decode is memory-bound and checking the draft costs almost nothing. Under continuous batching at high request rates, the GPU is already busy serving other requests, and every rejected draft token is computation spent for nothing. vLLM measured this in 2024 on Llama 3 70B across four H100s. At one request per second, a draft model gave up to 1.5x faster generation on ShareGPT chat prompts and n-gram drafting up to 2.8x on CNN/DailyMail summaries. At high request rates the same setups ran 1.4x and 1.8x slower. A draft model also needs its own KV cache, which shrinks the batch the target can hold.

So test at your real concurrency, not at one request at a time. A setup that wins in a single-user benchmark can lose at peak traffic, and the loss arrives exactly when the server is under the most pressure. vLLM's documentation, as of October 2026, describes the feature as aimed at medium-to-low request rates and memory-bound workloads, and lists a dynamic mode that adjusts speculation to load. Prefer a setup that can back off under load if your traffic swings.

Choosing a drafting method#

Most current methods avoid the separate draft model, which removes the tokenizer and alignment problem. Medusa (Cai and colleagues, 2024) adds extra prediction heads to the target itself and checks several candidate continuations at once with tree attention, for over 2.2x speedup in its first version and 2.3x to 3.6x in the second. EAGLE (Li and colleagues, 2024) runs a small head over the target's internal features rather than its tokens and reported 2.7x to 3.5x lower latency on LLaMA2-Chat 70B, and EAGLE-2 and EAGLE-3 followed. At the zero-training end, n-gram and suffix decoding draft by copying text already present in the prompt, which suits editing, extraction and retrieval-grounded answers, where the output largely restates the input. As of October 2026, vLLM's documentation lists EAGLE, multi-token prediction heads, draft models, n-gram and suffix decoding among its methods.

Try n-gram or suffix drafting first for an interactive product at modest concurrency, such as a chat assistant, a coding tool or an agent loop, when its answers restate their context. Move to an EAGLE-style head when the serving stack supports one and acceptance from copying is low. Keep speculation off for offline batch jobs and fill the batch instead, because throughput is the goal there and speculation spends the same arithmetic on guesses.