Continuous Batching

Continuous batching is a scheduling method for LLM servers that changes which requests share the GPU after every step of generation. Requests to a language model finish at very different times, because one answer is ten words and the next is two pages. A server that groups requests into a fixed batch spends most of that batch's life waiting on its slowest member. Continuous batching rebuilds the batch at each step instead, so finished requests leave and queued ones join without waiting for the rest.

Why one request at a time wastes the GPU#

Generating text happens one token at a time, in the decode phase. Each token requires the GPU to read nearly every model weight out of its memory, and it then does very little arithmetic with each one. The limit is how fast memory can be read, not how fast the chip can compute.

Take a 70B model stored at 16 bits per weight, which is about 140 GB. An H100 SXM reads memory at 3.35 TB/s, so a single decode step cannot take less than about 42 ms. One request on its own therefore gets about 24 tokens per second, however fast the chip's compute is. If 32 requests share that step, the weights are still read once and the step produces 32 tokens, about 760 tokens per second in total.

That figure is an upper bound, because each request also reads its own KV cache every step and the step gets slower as the batch grows. The direction holds: a GPU that serves one request at a time leaves most of its compute idle.

Static batching waits for the longest answer#

The older way to batch is to collect N requests, run them together, and return them together. Generation lengths are unknown in advance and vary widely, so the batch runs for as long as its longest member.

Take a batch of four requests whose answers come to 100, 200, 350 and 800 tokens. The batch runs for 800 steps. That is 4 × 800 = 3,200 slot-steps, and only 1,450 of them produce a real token. For the other 55%, the GPU computes padding for requests that already finished, while new requests queue outside until the whole batch drains. The longer the tail of the length distribution, the larger that waste, and chat traffic has a long tail.

Rebuilding the batch every step#

Orca, published at OSDI 2022, called the fix iteration-level scheduling. After every decode step the scheduler checks whether any sequence has just emitted its stop token. A finished sequence goes back to its caller at once, and a waiting request takes the free slot on the next step. The batch stays full for as long as there is demand.

Anyscale measured the effect in June 2023. Continuous batching alone, in Hugging Face TGI and Ray Serve, gave about 8x the throughput of naive static batching. vLLM, which pairs it with PagedAttention, reached up to 23x, with lower median latency as well. PagedAttention stores the KV cache in small blocks that are handed out as needed, so a departing request's memory can go straight to the next one. Without that, a free slot in the schedule has no free memory behind it.

As of October 2026, vLLM, SGLang and TensorRT-LLM all ship continuous batching with paged KV memory; TensorRT-LLM calls it in-flight batching. Hugging Face has since put TGI into maintenance mode and archived its repository on 21 March 2026, pointing users to vLLM and SGLang, so the 2023 numbers describe the idea more than a current choice of engine.

What a fuller batch costs each user#

Continuous batching beats static batching on every measure, and a trade-off remains inside it. Every sequence in a step shares the same memory reads, and each adds its own cache to them, so a fuller batch makes every step a little slower. Total tokens per second keep rising as the batch grows. The time each user waits between tokens, the inter-token latency, rises with it.

New prompts add a second cost. A request that joins the batch has to have its whole prompt processed first, the prefill, and that work is heavy on compute. Processed in one pass, a 4,000-token prompt stalls every request that was mid-answer, and users see the stream pause. Chunked prefill splits a long prompt into pieces and mixes each piece into the ongoing decode steps. vLLM turns it on by default. Smaller chunks keep streams smooth, and larger chunks get the new request to its first token sooner.

Which knobs set the trade-off#

Two settings carry most of it in vLLM as of October 2026. max_num_seqs caps how many sequences share a step. max_num_batched_tokens caps the tokens processed per step across prefill and decode together. vLLM's tuning guide says a budget near 2,048 gives better inter-token latency, because fewer prefill tokens slow each decode step. A budget above 8,192 gives a faster first token and higher throughput, especially for smaller models on large GPUs.

Pick from the promise the product makes, measured in latency metrics. A chat product judged on a fast first token and a smooth stream keeps the token budget and concurrency modest and adds replicas for the rest. An overnight batch job that nobody watches fills the batch as far as KV memory allows, because only total throughput matters to it.

Watch preemption while tuning. If the scheduler admits more sequences than the cache can hold, it evicts requests mid-answer and recomputes them later, which shows up as latency spikes and as warnings in the vLLM log. When those warnings persist, the guide's fixes are to give the cache more GPU memory or to lower max_num_seqs or the token budget.

Reading a throughput number#

Every engine you would deploy in 2026 already batches continuously, so the work is in the settings and in the test. A benchmark with uniform request lengths hides the problem continuous batching exists to solve, so load-test with the length mix production sends. A throughput figure without the inter-token latency it was measured at does not say whether users would accept the stream.

Before tuning the batch at all, check that the model is where the time goes. On a voice agent, the model call is one stage among several. Bigcircle timed each stage separately on its insurance voice agent, and on the real-estate agent built on the same runtime those timings put the extra Hindi and Hinglish time in transcription, not in reasoning.