Most product teams first meet LLM inference as an API call, where a prompt goes in, tokens stream out, and the hard parts all appear to belong to the model provider rather than to you. That illusion breaks the first time a customer sends a 90,000 token document, a demo stalls on time to first token, or finance asks why serving cost rose faster than usage. In production, inference is not a single model call. It is a pipeline of compute-bound prefill, memory-bound decode, cache allocation, batching, kernel choice, and GPU scheduling.
The useful mental model is simple: every request spends one phase reading the input, then many small phases writing the output. The first phase is prefill vs decode prefill and the second is decode, and everything else, from KV cache memory to continuous batching, exists because those two phases stress hardware in opposite ways.
Why inference is the product bottleneck
Training gets the headlines, but inference is where the product lives: it is the part users feel as latency, the part finance feels as gross margin, and the part operators feel as capacity planning. A model that is impressive in a benchmark can still be impossible to ship if it needs too much memory per user, returns the first token too slowly, or collapses throughput when prompts get long.
The metrics also split by user experience, and the split matters more than the totals. Latency metrics such as time to first token measure how quickly the system starts speaking, and time between tokens measures how smooth the stream feels. Total latency measures completion time, but users rarely experience it as one number. A chat assistant can tolerate a slow full answer if the first token arrives quickly, and a batch summarization job can ignore streaming entirely and optimize cost through the Batch API. A voice agent cannot tolerate either a slow first token or jitter between tokens.
That is why inference work is mostly tradeoff work, and why you tune for the shape of the product not for an abstract fastest possible model call.
The two phases: prefill and decode
During prefill the server processes the whole input prompt, so a 4,000 token prompt is one large parallel computation. The GPU can keep many cores busy because each token position can be processed across large matrix operations, which makes prefill compute-bound for short and medium contexts.
During decode the server generates one new token at a time, and each step depends on the previous token, so the model cannot produce token 200 before token 199 exists. Decode is usually memory-bandwidth-bound because every step reads model weights and the request's KV cache, then appends one more token of cache. This is why generation speed often drops as context grows even when the model fits in memory.
The product consequence is direct enough to plan against. Long prompts mostly hurt prefill and time to first token, long answers mostly hurt decode and total generation cost, and long conversations hurt both because the prompt and cache keep growing.
KV cache: the memory bill hiding behind every token
The KV cache stores attention keys and values for every previous token so the model does not recompute them on every decode step. It is the reason modern serving is practical, and it is also the reason context length is expensive. The cache grows with layers, KV heads, head dimension, precision, tokens, and concurrent requests.
A rough sizing exercise should happen before any serious deployment. Estimate bytes per token, multiply by target context length, then multiply by target concurrency. If that number does not fit after model weights are loaded, the server will queue, evict, preempt, or fail. Those behaviors surface as slow first tokens and uneven streaming, not as a neat "out of memory" product error.
Prompt caching changes the economics when many requests share the same prefix. A long system prompt, policy manual, or tool schema can be prefetched and reused, which removes repeated prefill work and reduces cost. It does not make decode free, it does not erase the memory requirement for active conversations, and all it really does is stop you paying twice for identical prefix tokens.
How servers get throughput
The naive way to serve requests is one at a time. That wastes the GPU because decode steps are small and memory-bound. Real serving stacks interleave many requests through continuous batching, so as one request finishes another joins the active batch without waiting for every sequence to end. The scheduler keeps the GPU busy while preserving each user's token stream.
Memory management is the second half of the problem, and it is where most capacity is lost. Early systems reserved large contiguous cache blocks for each request, which wasted memory when prompts and generations were shorter than the reservation. PagedAttention treats the cache more like virtual memory, allocating small blocks on demand. The result is higher effective concurrency because less GPU memory sits empty inside over-reserved slabs.
This is why two deployments with the same model and GPU can behave very differently. The difference is often not the model. It is the scheduler, the cache allocator, the queue policy and the batching strategy.
Why kernels matter
Attention is expensive because it moves a lot of data. FlashAttention improves performance by reducing memory traffic and doing attention in tiled chunks that fit better in fast on-chip memory. For inference, kernel choice affects prefill most visibly, especially with long prompts where attention over the input dominates.
Kernels do not remove the sequential nature of decode, but they still matter because every millisecond saved in prefill improves time to first token, and every efficiency gain increases how much useful work each GPU can do. This is also where hardware details leak into product decisions. A model that looks fine in a notebook can fall apart under production mix because the kernels, quantization format, or serving stack do not match the target GPU.
When one GPU is not enough
Large models often need model parallelism, splitting weights across GPUs. This solves capacity for weights, but it introduces communication between GPUs. For high-throughput serving, the question is not just "does it fit?" It is "does the added communication still leave enough bandwidth for decode?"
Disaggregated serving goes a step further by separating prefill and decode onto different workers or GPU pools. That can be useful because prefill and decode want different scheduling: prefill likes big parallel work, and decode likes stable memory bandwidth with many active sequences. Splitting them can improve utilization, but it adds routing, state transfer, and operational complexity.
Quantization is the most common practical lever, because lower precision weights reduce memory and bandwidth pressure. KV cache quantization can also help when long contexts dominate, and the tradeoff in both cases is quality and compatibility. We test this against the actual task distribution rather than against benchmark prompts.
What teams can tune
Useful tuning starts with workload shape rather than with a config file. If time to first token is the problem, inspect prompt length, prefill batching, prompt cache hit rate, and kernel support. If tokens stream slowly, inspect active batch size, cache pressure, decode bandwidth, and quantization. If latency spikes under load, inspect queue policy and whether long-context requests are starving short ones.
Separate online and offline traffic, because user-facing chat may need tight latency metrics while backfills, eval sweeps and nightly summarization can use the Batch API or a cheaper offline queue. Mixing those two traffic classes on one pool is a common way to make an otherwise good system feel unreliable.
Capacity planning should use percentiles, not averages. Average prompt length hides the one customer workflow that sends a full contract set. Average output length hides the analyst workflow that generates long reports, and average concurrency hides launch moments, retry storms, and internal batch jobs that accidentally run during business hours. A useful serving plan starts with p50, p90, and p99 prompt tokens, output tokens, and concurrent sessions for each product surface.
Queue design matters as much as model choice, and it is chosen far less carefully. A first-in first-out queue is simple, but it can let one huge prefill job delay many small interactive chats, which is why some serving systems separate short and long requests, others reserve capacity for latency-sensitive traffic, and others cap context length by tier. These are product decisions implemented as scheduler policy.
Another common lever is admission control, which decides what the server refuses to start. If the GPU is already near the cache limit, accepting one more long-context request can degrade every active stream. A better system returns a controlled retry, routes to a slower pool, truncates low-value context, or asks the user to run the job asynchronously. The worst system accepts everything and lets latency fail in public.
Failure modes to watch in production
The first failure mode is hidden context growth, which arrives without anyone deciding on it. Chat products often append every previous turn because it is the simplest state model. After a few hours of use, ordinary conversations become long-context serving jobs. The symptom is rising time to first token and falling tokens per second for loyal users. The fix is context management: summarize stale turns, preserve important facts, and drop text that no longer helps the task.
The second failure mode is cache fragmentation or preemption, and users feel it before monitoring reports it. If the server cannot allocate enough KV cache for active requests, it may pause some sequences, move cache blocks, or recompute prefixes, which users experience as uneven streaming. Monitoring should include cache usage, evictions, preemptions, and batch composition, not only request latency.
The third failure mode is benchmark drift, and it shows up on the invoice. A model can look cheap in a short-prompt benchmark and become expensive in a real product where prompts include tool schemas, safety policy, retrieved documents and conversation history. Measure the deployed prompt rather than the toy prompt, because the difference is often multiple thousands of tokens before the user has asked anything.
The fourth failure mode is over-optimizing one metric at the expense of the rest. Chasing maximum throughput can make time to first token unacceptable. Chasing the fastest first token can reduce batch size and raise cost. Chasing the cheapest quantized model can increase hallucinations or break structured output. Inference optimization is a portfolio of constraints, and the right answer changes by route.
Production takeaways
Treat inference as a serving system rather than a wrapper around a model. Model the KV cache before buying GPUs, track time to first token and time between tokens separately, and use continuous batching and PagedAttention class serving unless there is a strong reason not to. Turn on prompt caching when prefixes repeat. Reach for quantization only after measuring quality on real tasks.
Most importantly, tune for the product, because a coding assistant, a voice agent, a research tool and a nightly document processor all stress inference differently. The architecture that wins is the one whose bottleneck matches the job in front of it.