Disaggregated Serving

LLM inference has two phases. Prefill processes the prompt in parallel and builds the KV cache. Decode generates one token at a time and repeatedly reads that cache. Prefill wants compute. Decode wants memory bandwidth and low per-token latency. Running both on the same worker is simple, but the phases interfere.

Separate the phases#

Disaggregated serving sends incoming prompts to a prefill pool, transfers the resulting KV cache over an interconnect, and streams tokens from a decode pool. The scheduler can scale each pool independently: more prefill workers when prompts are long, more decode workers when many users are waiting for streamed output.

When it pays#

The win is strongest for mixed traffic: long prompts, many concurrent generations, and strict time-to-first-token plus time-between-token targets. DistServe, Mooncake, vLLM, SGLang, and NVIDIA Dynamo all explore variants of this separation. The common theme is KV-cache movement as a first-class scheduling problem.

Complexity costs#

Disaggregation is not free. The cache can be large, so interconnect bandwidth matters. Pool sizing becomes a control problem. Small models, short prompts, or low-traffic deployments may lose more to transfer overhead than they gain from specialization. Debugging also gets harder because one request spans multiple workers.

Practical takeaways#

Start with colocated serving until phase interference is measurable. Move to disaggregated serving when prefill bursts hurt decode latency or decode occupancy starves prompt throughput. Model the KV transfer path as carefully as the GPU count; it is the new critical resource.