TechnologyInference & Serving
Disaggregated Serving
At a glance
Run prefill and decode on separate GPU pools and ship the KV cache between them, so the two phases stop fighting.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Inference & Serving
- Technology
In short
Disaggregated serving is an LLM serving design that runs prompt processing and token generation on separate GPU pools and moves the KV cache between them, so a long prompt stops pausing other users' streams. Bigcircle recommends staying on one shared pool with chunked prefill until the latency percentiles show that stall, because the transfer and the second pool cost more than they save on small models, short prompts, or light traffic. Our guide on how LLM inference works covers where the split sits among the engines in 2026.
Key facts
Definition
Disaggregated serving gives prefill, the parallel pass over the prompt, and decode, the token-by-token answer, their own GPU pools, and sends the KV cache from the first pool to the second for every request.
How Bigcircle does it
- Bigcircle recommends splitting only when time-to-first-token or between-token percentiles move whenever long prompts arrive, because those metrics show the interference before anyone has to guess at it.
- Bigcircle recommends sizing the network between the pools with the same care as the GPU count, because the cache for a long prompt on a large model runs to gigabytes and every request has to move it.
- Bigcircle records latency per stage on its insurance voice agent and recommends the same record across prefill, transfer and decode, because a trace that holds only the final tokens blames the model for a routing fault.
Comparison
Colocated serving runs both phases on one worker and lets a long prompt stall every answer in progress, while disaggregated serving separates the phases and pays for it with a cache transfer on every request.
Key number
DistServe, at OSDI 2024, served up to 7.4 times as many requests within both latency targets as colocated systems, or met targets 12.6 times tighter, with more than 90% of requests inside them.
Disaggregated serving is an LLM serving design that puts the two phases of every request on separate pools of GPUs and moves the model's working state between them. The two phases are different kinds of work. Prefill reads the whole prompt in one parallel pass and builds the KV cache, the stored attention state for every token so far. Decode then generates the answer one token at a time, reading that cache on every step. Prefill is limited by how fast the GPU computes. Decode is limited by how fast it reads memory. On one shared worker the two get in each other's way, because a long prompt arriving mid-stream pauses every answer that worker is generating.
Two pools and a transfer#
Disaggregated serving sends each new prompt to a pool of prefill GPUs, moves the finished KV cache over the network between machines, and generates the answer on a separate pool of decode GPUs. Each pool can then grow on its own: more prefill GPUs when prompts get longer, more decode GPUs when more users are waiting on streamed answers.
DistServe, published at OSDI 2024, made the case in terms of goodput, the number of requests served while still meeting both the first-token target and the between-token target. Against colocated systems it served up to 7.4 times as many requests, or met targets 12.6 times tighter, with more than 90% of requests inside them. Mooncake, the serving platform behind Moonshot AI's Kimi chatbot, built its design around storing and moving the cache, and won the best paper award at FAST 2025.
Where the engines stand in October 2026#
The split has moved from papers into the main open engines. SGLang ships prefill and decode disaggregation, with Mooncake and NIXL as transfer back ends. TensorRT-LLM documents a disaggregated serving mode. NVIDIA Dynamo sits above vLLM, SGLang and TensorRT-LLM and coordinates the separate pools rather than replacing the engines. llm-d, a Kubernetes project for the same job, was accepted into the CNCF Sandbox in March 2026. vLLM still titles its own page "Disaggregated Prefilling (experimental)", so a team on vLLM alone is running a feature its maintainers have not called stable.
What the split costs#
Every request now crosses at least two machines, and its cache has to cross with it. For a long prompt on a large model that cache runs to gigabytes, so the network between the pools decides whether the split pays at all. Sizing two pools against shifting traffic becomes a control problem that one shared pool never had. When a request stalls, it can be stuck in prefill, in the transfer or in decode, and a trace that only records the final tokens will blame the model for a routing fault.
A small model, short prompts or light traffic can lose more to the transfer than the split gains. Chunked prefill is the lighter fix inside one engine: it splits a long prompt into pieces and mixes them into the ongoing decode steps, and vLLM turns it on by default. It does not separate the hardware, but it removes much of the stall that motivates the split.
When to split#
Split the phases when the latency metrics show the interference: a stream that pauses whenever a long prompt arrives, or new requests waiting for a first token because decode has the GPUs full. Both show up in the time-to-first-token and between-token percentiles before anyone has to guess. If neither moves under production traffic with chunked prefill on, a shared pool is the simpler system to run.
Whatever the serving layout, time each stage separately. On the insurance voice agent, latency is recorded per stage, because a report that the agent felt slow cannot be checked against anything otherwise. A split deployment needs the same record for prefill, transfer and decode before anyone can say which pool to grow.