LLM inference is a pipeline of compute-bound prefill and memory-bound decode, plus the cache, batching, kernel choice, and GPU scheduling those two phases force. Most product teams first meet LLM inference as an API call, where a prompt goes in, tokens stream out, and the hard parts all appear to belong to the model provider rather than to you. Bigcircle sees that illusion break on client systems the first time a customer sends a 90,000 token document, a demo stalls on time to first token, or finance asks why serving cost rose faster than usage.
The useful mental model is simple: every request spends one phase reading the input, then many small phases writing the output. The first phase is prefill vs decode prefill and the second is decode, and everything else, from KV cache memory to continuous batching, exists because those two phases stress hardware in opposite ways.
Why inference is the product bottleneck
LLM inference is the part of the product a user waits on, even when training gets the headlines. People feel it as latency, finance feels it as gross margin, and operators feel it as the GPUs they have to keep busy. A model that looks strong on a short benchmark can still be unfit to ship when each session needs too much memory, the first token is slow, or long prompts pull throughput down.
The metrics also split by user experience, and the split matters more than the totals. Latency metrics such as time to first token measure how quickly the system starts speaking, and time between tokens measures how smooth the stream feels. Total latency measures completion time, but users rarely experience it as one number. A chat assistant can tolerate a slow full answer if the first token arrives quickly, and a batch summarization job can ignore streaming entirely and optimize cost through the Batch API. A voice agent cannot tolerate either a slow first token or jitter between tokens.
That is why inference work is mostly tradeoff work, and why you tune for the shape of the product not for an abstract fastest possible model call.
The two phases: prefill and decode
Prefill and decode are the two phases of an LLM inference request, and they load the hardware in opposite ways. Prefill reads the entire prompt in one parallel pass, so short and medium contexts are usually limited by compute. Decode writes one new token at a time and must reread the weights and the cache on every step, so it is usually limited by memory bandwidth.
During prefill the server processes the whole input prompt, so a 4,000 token prompt is one large parallel computation. The GPU can keep many cores busy because each token position can be processed across large matrix operations, which makes prefill compute-bound for short and medium contexts.
During decode the server generates one new token at a time, and each step depends on the previous token, so the model cannot produce token 200 before token 199 exists. Decode is usually memory-bandwidth-bound because every step reads model weights and the request's KV cache, then appends one more token of cache. This is why generation speed often drops as context grows even when the model fits in memory.
The product consequence is direct enough to plan against. Long prompts mostly hurt prefill and time to first token, long answers mostly hurt decode and total generation cost, and long conversations hurt both because the prompt and cache keep growing.
KV cache: the memory bill hiding behind every token
KV cache memory is the growing bill behind every token an LLM inference server has already read. The server keeps attention keys and values so the next decode step does not recompute them, and that state scales with layers, heads, precision, context length, and how many requests are active together. Size that footprint in bytes before you commit the GPUs, because a cache that does not fit beside the weights makes the server queue, evict, or recompute.
The KV cache stores attention keys and values for every previous token so the model does not recompute them on every decode step. It is the reason modern serving is practical, and it is also the reason context length is expensive. The cache grows with layers, KV heads, head dimension, precision, tokens, and concurrent requests.
A rough sizing exercise should happen before any serious deployment. Estimate bytes per token, multiply by target context length, then multiply by target concurrency. If that number does not fit after model weights are loaded, the server will queue, evict, preempt, or fail. Those behaviors surface as slow first tokens and uneven streaming, not as a neat "out of memory" product error.
Prompt caching changes the economics when many requests share the same prefix. A long system prompt, policy manual, or tool schema can be prefetched and reused, which removes repeated prefill work and reduces cost. It does not make decode free, it does not erase the memory requirement for active conversations, and all it really does is stop you paying twice for identical prefix tokens.
How servers get throughput
LLM serving throughput comes from keeping many requests in flight, because one decode step is too small to fill a GPU. Continuous batching lets a finished request leave and a new one join without holding the batch until every sequence ends. Paged attention then hands out cache in small blocks, so a short prompt does not reserve a slab of memory it will never touch.
The naive way to serve requests is one at a time. That wastes the GPU because decode steps are small and memory-bound. Real serving stacks interleave many requests through continuous batching, so as one request finishes another joins the active batch without waiting for every sequence to end. The scheduler keeps the GPU busy while preserving each user's token stream.
Memory management is the second half of the problem, and it is where most capacity is lost. Early systems reserved large contiguous cache blocks for each request, which wasted memory when prompts and generations were shorter than the reservation. PagedAttention treats the cache more like virtual memory, allocating small blocks on demand. The result is higher effective concurrency because less GPU memory sits empty inside over-reserved slabs.
This is why two deployments with the same model and GPU can behave very differently. The difference is often not the model. It is the scheduler, the cache allocator, the queue policy and the batching strategy.
Why kernels matter
Attention kernels set how much data an LLM inference pass moves for every token. Tiled kernels in the FlashAttention family compute attention in chunks that fit in fast on-chip memory, which cuts trips to slower GPU memory. The gain is easiest to see in prefill on long prompts, where attention over the input decides time to first token. Kernels leave decode sequential, and a shorter prefill still lets each GPU finish more useful work.
Attention is expensive because it moves a lot of data. FlashAttention improves performance by reducing memory traffic and doing attention in tiled chunks that fit better in fast on-chip memory. For inference, kernel choice affects prefill most visibly, especially with long prompts where attention over the input dominates.
Kernels do not remove the sequential nature of decode, but they still matter because every millisecond saved in prefill improves time to first token, and every efficiency gain increases how much useful work each GPU can do. This is also where hardware details leak into product decisions. A model that looks fine in a notebook can fall apart under production mix because the kernels, quantization format, or serving stack do not match the target GPU.
When one GPU is not enough
One GPU stops being enough for LLM inference when the weights, or the weights plus the live cache, no longer fit in memory. Model parallelism spreads those weights across devices and spends part of the gained capacity on communication between them. Disaggregated serving puts prefill and decode on separate pools, because one phase wants large parallel work and the other wants steady bandwidth across many active sequences.
Large models often need model parallelism, splitting weights across GPUs. This solves capacity for weights, but it introduces communication between GPUs. For high-throughput serving, the question is not just "does it fit?" It is "does the added communication still leave enough bandwidth for decode?"
Disaggregated serving goes a step further by separating prefill and decode onto different workers or GPU pools. That can be useful because prefill and decode want different scheduling: prefill likes big parallel work, and decode likes stable memory bandwidth with many active sequences. Splitting them can improve utilization, but it adds routing, state transfer, and operational complexity.
Quantization is the most common practical lever, because lower precision weights reduce memory and bandwidth pressure. KV cache quantization can also help when long contexts dominate, and the tradeoff in both cases is quality and compatibility. We test this against the actual task distribution rather than against benchmark prompts.
What teams can tune
Teams tune LLM inference from the shape of the workload, not from a generic config file. A slow first token points at prompt length, prefill batching, prompt-cache hit rate, and kernel support. A slow stream points at active batch size, cache pressure, decode bandwidth, and precision. Spikes under load point at queue policy, and at long-context requests starving the short ones users are waiting on.
Separate online and offline traffic, because user-facing chat may need tight latency metrics while backfills, eval sweeps and nightly summarization can use the Batch API or a cheaper offline queue. Mixing those two traffic classes on one pool is a common way to make an otherwise good system feel unreliable.
Capacity planning should use percentiles, not averages. Average prompt length hides the one customer workflow that sends a full contract set. Average output length hides the analyst workflow that generates long reports, and average concurrency hides launch moments, retry storms, and internal batch jobs that accidentally run during business hours. A useful serving plan starts with p50, p90, and p99 prompt tokens, output tokens, and concurrent sessions for each product surface.
Queue design matters as much as model choice, and it is chosen far less carefully. A first-in first-out queue is simple, but it can let one huge prefill job delay many small interactive chats, which is why some serving systems separate short and long requests, others reserve capacity for latency-sensitive traffic, and others cap context length by tier. These are product decisions implemented as scheduler policy.
Another common lever is admission control, which decides what the server refuses to start. If the GPU is already near the cache limit, accepting one more long-context request can degrade every active stream. A better system returns a controlled retry, routes to a slower pool, truncates low-value context, or asks the user to run the job asynchronously. The worst system accepts everything and lets latency fail in public.
Failure modes to watch in production
Production LLM inference fails in four ways an average latency chart will not name. Context grows because a chat keeps every earlier turn. The cache fills, the server preempts or recomputes, and the stream turns uneven. Benchmarks use short prompts while the live product sends schemas, policy, and history. Pushing throughput, first-token speed, or precision on its own breaks another constraint the route still has.
The first failure mode is hidden context growth, which arrives without anyone deciding on it. Chat products often append every previous turn because it is the simplest state model. After a few hours of use, ordinary conversations become long-context serving jobs. The symptom is rising time to first token and falling tokens per second for loyal users. The fix is context management: summarize stale turns, preserve important facts, and drop text that no longer helps the task.
The second failure mode is cache fragmentation or preemption, and users feel it before monitoring reports it. If the server cannot allocate enough KV cache for active requests, it may pause some sequences, move cache blocks, or recompute prefixes, which users experience as uneven streaming. Monitoring should include cache usage, evictions, preemptions, and batch composition, not only request latency.
The third failure mode is benchmark drift, and it shows up on the invoice. A model can look cheap in a short-prompt benchmark and become expensive in a real product where prompts include tool schemas, safety policy, retrieved documents and conversation history. Measure the deployed prompt rather than the toy prompt, because the difference is often multiple thousands of tokens before the user has asked anything.
The fourth failure mode is over-optimizing one metric at the expense of the rest. Chasing maximum throughput can make time to first token unacceptable. Chasing the fastest first token can reduce batch size and raise cost. Chasing the cheapest quantized model can increase hallucinations or break structured output. Inference optimization is a portfolio of constraints, and the right answer changes by route.
Production takeaways
Treat inference as a serving system rather than a wrapper around a model. Model the KV cache before buying GPUs, track time to first token and time between tokens separately, and use continuous batching and PagedAttention class serving unless there is a strong reason not to. Turn on prompt caching when prefixes repeat. Reach for quantization only after measuring quality on real tasks.
Most importantly, tune for the product, because a coding assistant, a voice agent, a research tool and a nightly document processor all stress inference differently. The architecture that wins is the one whose bottleneck matches the job in front of it.
Common questions
What is the difference between prefill and decode?#
Prefill reads the whole prompt in one parallel pass, and it usually sets time to first token. Decode generates one token at a time, and it usually sets how smooth the stream feels and what the full answer costs. Long documents stress the prefill pass. Long answers and growing conversations stress decode, because each new token also adds cache the next step has to read.
Why does the KV cache dominate LLM serving memory?#
The cache holds attention state for every token already seen, on every layer, for every request that is still active. That state is why decode can avoid recomputing the prompt, and it is why context length and concurrency compete with the weights for the same GPU memory. When the sum does not fit, the server queues, evicts, or recomputes prefixes, and users feel that as a slow or uneven stream.
Does prompt caching make generation free?#
Prompt caching reuses a shared prefix, such as a system prompt or a tool schema, so repeated prefill on identical leading tokens is skipped. It does not free the memory an active conversation still holds, and it does not skip the token-by-token decode pass. Measure the hit rate on the prefix the product actually sends.
When should prefill and decode run on separate GPUs?#
Separate them when the two phases want different machines and moving the cache between pools costs less than forcing them to share one. Prefill likes large parallel work, and decode likes stable memory bandwidth across many live sequences. The split can raise utilization, and it adds routing, state transfer, and another pool to operate. If the only problem is that the weights do not fit, try lower precision on the real task mix first.
What should we measure before tuning an inference stack?#
Measure the deployed prompt, not a short benchmark. Track time to first token and time between tokens separately, plus p50, p90, and p99 for prompt tokens, output tokens, and concurrent sessions on each route. Watch cache usage, evictions, preemptions, and whether long requests are blocking short ones. Averages hide the contract workflow and the launch spike that set the bill.