Continuous Batching

Continuous batching is the scheduling idea that turned LLM serving from a research demo into an economic business. The problem it solves is mundane: GPUs want big parallel batches, but LLM requests finish at wildly different times, so a naive batch spends most of its life waiting on its single slowest member. The fix is equally simple to state: stop treating the batch as a fixed unit of work and rebuild it on every single decode step.

Why GPUs hate one-at-a-time decoding#

The decode phase of prefill vs decode is memory-bandwidth-bound. To produce one token, the GPU must stream essentially every model weight from high-bandwidth memory through its compute units, then do a comparatively tiny amount of math with each weight. Run the numbers for a 70B model in FP16: about 140 GB of weights divided by roughly 3.3 TB/s of H100 bandwidth gives a hard floor near 24 ms per decode step, which caps a single request around 40 tokens per second no matter how fast the chip's tensor cores are.

Batching is the escape hatch. If 32 requests share a decode step, the weights are streamed from memory once but produce 32 tokens, so the same 24 ms step now yields over 1,200 tokens per second of aggregate output. The arithmetic intensity rises with batch size, and the GPU shifts from starving on bandwidth toward actually using its compute. Everything about modern serving economics follows from this one fact: a GPU serving one request at a time wastes most of what you paid for.

Static batching and the slowest-request problem#

The obvious way to batch is the old deep-learning way: collect N requests, run them together, return all results, repeat. This is static batching, and for LLMs it fails because generation lengths are unknown up front and vary enormously.

Work a concrete case. A static batch of 4 requests generates answers of 100, 200, 350, and 800 tokens. The batch runs for 800 steps because the longest member dictates the schedule. Total slot capacity is 4 x 800 = 3,200 token-steps, but only 1,450 produce real tokens. The other 55% of the machine's time is spent computing padding for sequences that already finished, while new requests queue outside waiting for the entire batch to drain. The longer the tail of your length distribution, the worse it gets, and chat traffic has a very long tail.

Iteration-level scheduling: rebuild the batch every step#

Continuous batching, introduced as iteration-level scheduling by the Orca system at OSDI 2022, makes batch membership a per-step decision. After every decode iteration the scheduler checks: did any sequence just emit its stop token? If so, return it immediately and admit a waiting request into the freed slot on the very next step. Finished work leaves instantly, new work enters within milliseconds, and the batch stays full as long as there is demand.

The headline numbers come from Anyscale's 2023 benchmark study. Continuous batching alone, as implemented in Hugging Face TGI and Ray Serve, delivered about 8x the throughput of naive static serving. Adding vLLM's PagedAttention, which stores the KV cache in small on-demand blocks so freed slots actually release their memory, pushed the combined gain to 23x over naive Hugging Face serving while also lowering median latency. The two techniques are inseparable in practice: per-step admission only works if the memory of a departing sequence can be handed to an arriving one without reshuffling, which is exactly what paged KV blocks provide. Every serious engine since, including vLLM, TensorRT-LLM, and SGLang, ships both as the default.

The throughput versus latency tradeoff#

Continuous batching dominates static batching outright, but it does not repeal physics. Within a continuously batched server there is still a real dial between aggregate throughput and per-request latency, and it has two faces.

First, batch size itself. Every sequence in a decode step shares the same memory-bandwidth budget, and each one's KV cache must be read every step, so a fuller batch means each step takes slightly longer. Inter-token latency creeps up as concurrency climbs: a server doing 5 ms between tokens at batch 4 might do 40 ms at batch 64, while total tokens per second keeps rising. Throughput improves until the GPU saturates; the tail of your per-request experience pays for it the whole way.

Second, prefill interference. When a new request is admitted, its prompt must be prefilled, and prefill is compute-heavy. If the scheduler processes a 4,000-token prompt in one gulp, every decoding request in the batch stalls behind it, producing a visible hiccup in streaming output. Modern engines mitigate this with chunked prefill, splitting big prompts into pieces and mixing each piece into ongoing decode steps, which is enabled by default in vLLM's V1 engine. Smaller chunks smooth inter-token latency; bigger chunks finish prefill sooner and improve time to first token.

Tuning to your SLO#

The tuning surface reduces to two knobs plus a decision about what you are promising users, measured in the vocabulary of latency metrics: time to first token (TTFT), inter-token latency (ITL), and end-to-end p99.

In vLLM terms, max_num_seqs caps how many sequences may share a step, and max_num_batched_tokens caps the total token budget per step across prefill and decode. The vLLM tuning guide is explicit about the direction of each: a smaller token budget around 2,048 gives better ITL because fewer prefill tokens slow down decodes, while a larger budget above 8,192 gives better TTFT and throughput because prompts finish prefilling faster and the GPU stays saturated.

So tune backwards from the SLO. An interactive chat product promising sub-second TTFT and roughly 30 to 50 ms ITL keeps the token budget and concurrency modest, accepts perhaps 60% of peak throughput, and scales horizontally for the rest. An overnight summarization pipeline with no human watching runs the batch as large as KV memory allows, where a 2x to 3x throughput gain directly halves or thirds the GPU bill and nobody notices 80 ms between tokens. Watch one more gauge while you tune: preemption. If the scheduler admits more sequences than the KV cache can hold, it must evict and recompute requests mid-flight, which shows up as latency spikes; persistent preemption warnings mean you should lower concurrency or add cache memory.

Practical takeaways#

Continuous batching is table stakes, not a differentiator: any engine you would deploy in 2026 already does it, so the real work is configuration. Confirm your stack pairs it with paged KV memory, then set the batch knobs from your SLO rather than from a benchmark leaderboard: small token budgets for tight ITL, large ones for throughput and TTFT. Load test with production-shaped traffic, because the whole point of the technique is handling variable lengths, and a benchmark with uniform lengths will lie to you. And when someone quotes a throughput number, ask what ITL it was measured at; without that, the number is half a fact.