Prefill vs Decode

LLM inference is not one workload, it is two. The model first reads your entire prompt, then writes its answer one token at a time, and those two phases stress completely different parts of the GPU. Almost every serving decision, from batch size to which latency metric you alert on, follows from that split. Treat them as one workload and you will optimize the wrong thing.

Prefill: the whole prompt in one parallel pass#

Prefill processes the input prompt in a single forward pass. Every prompt token is available up front, so the model can compute attention and feed-forward layers for all of them simultaneously: large matrix-matrix multiplications, exactly the shape of work GPUs were built for. Prefill is compute-bound: it saturates the floating-point units, and its cost scales with prompt length.

Put numbers on it. A forward pass costs roughly 2 FLOPs per parameter per token, so an 8B-parameter model reading a 2,000-token prompt does about 32 trillion FLOPs. An H100 delivers roughly 989 dense BF16 TFLOPS, so that prompt is on the order of 30 ms of pure math, and well under 100 ms with real-world efficiency. A long prompt is cheap in wall-clock terms precisely because all of it is consumed together. Prefill also produces a side effect that matters later: it writes the attention keys and values for every prompt token into the KV cache.

Decode: one token at a time#

Decode is the opposite. Generation is autoregressive: token N+1 depends on token N, so there is no parallelism across the output. Each step is a matrix-vector multiply that produces exactly one token, yet it still has to stream the full model weights, plus the growing KV cache, from GPU memory. Decode is memory-bandwidth-bound: limited by how fast bytes move from HBM, not by arithmetic.

The same numbers show why. That 8B model in FP16 is about 16 GB of weights. An H100 moves roughly 3.35 TB/s from HBM, so just reading the weights once takes about 5 ms, a hard floor near 200 tokens per second for a single request. Meanwhile the actual math per token uses under 1% of the chip's peak compute. The hardware can perform almost 300 floating-point operations in the time it takes to fetch one byte; decode offers it about one. Generating a long answer is fundamentally slower than reading a long prompt, and faster GPUs help decode mainly through bandwidth, not FLOPs.

Two phases, two latency metrics#

The split maps directly onto the two latency metrics that matter. Time to first token (TTFT) is queueing plus prefill: it grows with prompt length and tells you whether the system feels responsive. Time between tokens (TBT, also called inter-token latency or ITL) is pure decode: it sets how smoothly text streams and whether output keeps pace with reading speed. End-to-end latency is just TTFT plus output length times TBT.

The two metrics have different root causes and different fixes, so blending them into one average hides which phase is actually your bottleneck. A TTFT spike points at prefill pressure or queueing; a TBT spike points at decode saturation or KV cache pressure. Track and alert on them separately and you can tell the two failure modes apart at a glance.

Batching: why decode loves company#

Batching is where the asymmetry pays off. A decode step for one request reads 16 GB of weights to produce one token. A decode step for 32 requests reads the same 16 GB and produces 32 tokens: the weight traffic is shared, so throughput rises almost linearly while per-token latency barely moves, until compute finally catches up. This is why continuous batching is the single highest-leverage serving optimization: it keeps the decode batch full by slotting new requests in as old ones finish.

Prefill gains far less from batching because each request alone already saturates compute. Worse, naively mixing the phases hurts: schedule a 4,000-token prefill into a running batch and every other user's token stream stalls for the duration, which users perceive as stutter. Chunked prefill, introduced by Sarathi-Serve, splits a long prompt into pieces and runs each piece alongside ongoing decode steps, exploiting the compute that decode leaves idle. The paper reports up to 2.6x higher serving throughput within latency targets for Mistral 7B on one A100, and up to 6.9x for Falcon-180B across eight.

Disaggregated prefill and decode#

The logical endpoint of the split is to stop running both phases on the same hardware at all. Disaggregated serving, demonstrated by DistServe, sends prefill to one pool of GPUs and decode to another, shipping the KV cache across the interconnect in between. Each pool can then be scaled and parallelized for its own job: prefill capacity sized for TTFT targets, decode capacity sized for TBT targets, with neither phase interfering with the other. The idea has moved from research into mainstream inference stacks, and most large-scale deployments in 2026 either disaggregate or use chunked prefill to manage the interference. The other big lever on each phase is reuse: prompt caching skips prefill for repeated prefixes, and speculative decoding attacks the decode bound by drafting several tokens per weight read.

Practical takeaways#

Measure TTFT and TBT separately; a single latency number cannot tell you which phase to fix. Expect prompt-heavy workloads (RAG, long documents) to stress prefill and TTFT, and generation-heavy workloads (agents, code, reasoning) to stress decode, where output length dominates total latency. Size batches for decode, since that is where batching is nearly free. If users report stutter while a dashboard shows healthy averages, suspect prefill interference and turn on chunked prefill. And when one box can no longer hit both latency targets at once, disaggregation is the structural fix, not bigger GPUs.