Prefill vs Decode

Prefill vs decode is the difference between the two phases of every LLM request. The model first reads your whole prompt in one pass, then writes its answer one token at a time, and the two phases load the GPU in opposite ways. Prefill is limited by how fast the chip can compute. Decode is limited by how fast it can read memory. Batch size, which latency number you alert on and whether a faster GPU helps all follow from that split, so a team that treats inference as one workload usually tunes the wrong phase.

Prefill reads the prompt in one parallel pass#

Every prompt token is available at the start, so the model can process all of them at once. That means large matrix-by-matrix multiplications, the shape of work GPUs are built for, and prefill keeps the arithmetic units busy. Its cost grows with prompt length.

Put numbers on it. A forward pass costs about 2 floating-point operations per parameter per token, so an 8B model reading a 2,000-token prompt does about 32 trillion of them. An H100 SXM peaks at about 989 trillion dense BF16 operations per second, which puts that prompt near 30 ms of pure math and well under 100 ms in practice. Prefill also writes the attention keys and values for every prompt token into the KV cache, which decode reads on every later step.

Decode writes one token per step#

Each new token depends on the one before it, so the output cannot be computed in parallel. Every step is a matrix-by-vector multiply that yields one token. It still has to stream all of the model's weights from GPU memory, plus the growing KV cache.

The same 8B model in 16-bit precision is about 16 GB of weights. An H100 SXM reads about 3.35 TB/s from its memory, so reading the weights once takes about 5 ms, a floor near 200 tokens per second for a single request. The chip can do almost 300 floating-point operations in the time it takes to fetch one byte, and decode gives it about one. Expect a long answer to cost more time than a long prompt for that reason, and a newer GPU to speed decode mainly through its bandwidth, not its peak compute.

Two phases, two latency numbers#

The split maps onto the two latency metrics a product promises. Time to first token is queueing plus prefill, so it grows with prompt length and decides whether the product feels responsive. Time between tokens, also called inter-token latency, is decode, and it decides how smoothly text streams. End-to-end latency is time to first token plus the output length times the time between tokens.

Alert on the two separately, because one average hides which phase is in trouble. A spike in time to first token points at prefill pressure or a queue. A spike in time between tokens points at decode saturation or a KV cache under pressure.

Before either, check that the model is the slow stage at all. On our insurance voice agent, the model call is one stage among turn detection, transcription, tool calls and speech synthesis, and we timed each stage separately because "the agent felt slow" cannot be debugged otherwise. The same agent loads policy and claim context once at the start of the call, which keeps that variable cost off every turn.

Batching helps decode and barely helps prefill#

A decode step for one request reads 16 GB of weights to produce one token. A step for 32 requests reads the same 16 GB and produces 32 tokens, so throughput climbs almost in line with the batch while each user's wait between tokens barely moves, until compute catches up. Continuous batching exists to keep that decode batch full, by admitting a new request whenever one finishes.

Prefill gains little from batching, because one long prompt already fills the arithmetic units. Mixing the phases carelessly makes things worse. A 4,000-token prefill dropped into a running batch stalls every other user's stream for its duration, and people see the text stutter.

Chunked prefill, introduced by Sarathi-Serve, splits a long prompt into pieces and runs each piece beside the ongoing decode steps, using compute that decode leaves idle. The paper reports up to 2.6x more serving capacity than vLLM within latency targets for Mistral 7B on one A100, and up to 3.7x for Yi-34B on two. Check for prefill interference first when users report stutter while the average latency chart looks healthy, and turn chunked prefill on before trying anything larger. Set batch size for decode, because that is where batching is nearly free.

Splitting the phases across machines#

DistServe took the split to its end point: prefill on one pool of GPUs, decode on another, with the KV cache shipped between them. Each pool is then sized for its own target, prefill for time to first token and decode for time between tokens. As of October 2026, vLLM still labels its disaggregated prefill experimental, and its docs state that it does not improve throughput. It buys separate control over the two latencies. Reach for it only after chunked prefill has failed to hold both targets on one pool, because it adds a network transfer of the cache to every request.

Which fix belongs to which phase#

The other levers sort cleanly by phase. Pick the fix from the phase your alerts point at, not from whichever technique is newest. Prompt caching skips prefill for a prefix the server has already seen, so it cuts time to first token on long, repeated prompts and does nothing for streaming speed. Quantization shrinks the bytes decode must read per step, so it speeds streaming. Speculative decoding drafts several tokens per weight read and helps decode when the server is lightly loaded, but vLLM measured it slowing throughput by about 1.4x at high load, so test it at your real concurrency. Prompt-heavy work such as long documents and retrieval leans on prefill. Agents, code and reasoning models write long outputs and lean on decode.