LLM inference is the work of running a trained model on a new prompt to produce an answer, as opposed to training the model. It runs in two phases: prefill reads the whole prompt and sets how long the user waits for the first token, then decode writes the answer one token at a time and sets how fast it streams. Most product teams first meet LLM inference as an API call, and the decisions that matter to them sit above the GPU: which latency number to budget, how long the prompt may grow, and where the model runs.
Key facts
Definition
LLM inference is running a trained model on a prompt to produce an answer, with prefill setting the wait for the first token and decode setting how fast the answer streams.
How Bigcircle does it
- Bigcircle times every stage of a request separately before tuning the model call.
- On its insurance voice agent, the failures that took the work were in turn detection and integration, not in reasoning.
- On its real-estate voice agent, the extra time on Hindi and Hinglish calls was in transcription rather than in reasoning.
Comparison
A faster model or a tuned server shortens one stage of the budget, while a voice reply also waits on turn detection, transcription, tools and speech synthesis.
Key number
End-to-end response on the insurance voice agent fell from 2.5 to 3 seconds to around 900 milliseconds.
This guide follows the decisions a product team makes and says what we choose at each one.
Which latency number should you budget?
Three numbers describe one request, and a product usually cares about one of them most. Time to first token is how long before anything appears, the time between tokens is how smooth the stream feels, and total time is when the answer is complete. A chat interface can hide a slow total behind a fast first token. A nightly summarisation job can ignore both and go through a batch API. As of October 2026, OpenAI, Anthropic and Google each price batch work at half the synchronous rate, with results inside 24 hours. A voice agent needs the first token fast and the stream steady, because a caller hears every gap.
On AVOX, an insurance voice agent we built, end-to-end response fell from 2.5 to 3 seconds to around 900 milliseconds on noisy bilingual calls. The model call was one stage of that budget, alongside turn detection, transcription, tool execution and speech synthesis. The failures that took the work were conversational and integration failures rather than reasoning failures, starting with turn detection, the decision that the caller has finished speaking. We timed every stage separately, because "the agent felt slow" cannot be checked any other way.
On AVOX Realty, a property-search agent built on the same runtime, replies came back in under 1.2 seconds in English and under 1.3 seconds in Hindi and Hinglish. The per-stage timings showed the Hindi and Hinglish calls spending their extra time in transcription rather than in reasoning. A faster model would not have closed that gap.
So time each stage before you tune the model. A faster model or a better-tuned server shortens one stage, and on both of these builds the stage that mattered most sat outside the model call. Latency belongs to the whole path (model, network, telephony, synthesis), so measure it on your own. how to build a voice AI agent covers where the rest of a voice budget goes.
What are prefill and decode, and why do they behave differently?
Prefill and decode load the hardware in opposite ways, and each one maps to a length a product team controls. Prefill reads the whole prompt in one parallel pass, so it is usually limited by how fast the GPU can compute, and the prompt's length sets the time to first token. Decode produces one token per step and has to read the model's weights and the stored attention state on every step, so it is usually limited by memory bandwidth. Budget the two separately, because the fixes differ. A shorter prompt or a cached prefix cuts prefill, and a shorter answer or fewer bytes read per step cuts decode.
Length is the first thing we change, before hardware. A longer prompt costs time before the first token, and a longer answer costs time after it. A conversation that keeps every earlier turn costs both, because the prompt and the stored state grow together. On AVOX the policy and claim details the agent needed were loaded at the start of the call rather than fetched during a turn. Loading them early took the largest variable cost off each turn's critical path. It was an application decision, and no server setting could have made it.
How much memory does a long conversation hold?
Check the cache size for your own model before you set a context limit, because it decides how many conversations fit on one GPU. The KV cache holds the attention keys and values for every token a request has already processed, so decode does not recompute them. Its size per token is two (one for keys, one for values) times the number of layers, times the number of key-value heads, times the head size, times the bytes per number. For Llama 3 70B, with 80 layers, 8 key-value heads and a head size of 128 in 16-bit precision, that is 327,680 bytes per token. One conversation at 32,768 tokens therefore holds about 10.7 GB of cache on top of the weights, and every concurrent conversation holds its own.
The weights stay the same size however many people use the product, and the cache grows with every user and every turn. A server that runs out of cache memory queues new requests, evicts work in progress or recomputes it, and users see that as a slow first token or a stream that stalls. Trim conversation history before you raise the context limit, and size for the longest prompts your product actually sends, which the average prompt length will hide.
Do not choose a serving engine on how it manages this memory, because vLLM, SGLang and TensorRT-LLM all do it the same two ways. Continuous batching lets a finished request leave the batch and a waiting one join after any step, instead of holding every request until the longest one ends. PagedAttention stores the cache in fixed-size blocks, so no memory sits reserved for tokens a request never writes. Plan for one side effect of batching. A request's numbers shift slightly with the size of the batch it shares, and why LLM temperature 0 is not deterministic traces what that does to repeated calls.
Which LLM inference optimizations changed in 2025 and 2026?
If you buy inference through an API, none of these is yours to turn on, and they reach you as price tiers and latency claims. Start on a hosted API, and self-host only when a data rule requires it or the volume is steady enough to keep the GPUs busy. If you do run your own servers, below is the order to try the 2025 and 2026 changes in, and the traffic each needs to pay. Confirm each step on your own traffic before you keep it.
Start with chunked prefill, because it needs no extra hardware. It splits a long prompt into pieces and schedules them between other users' decode steps, so one long prompt no longer freezes everyone else's stream. vLLM now turns it on by default.
Move to disaggregated serving, with prefill and decode on separate GPU pools, only if long prompts still stall streams after that. As of October 2026 vLLM, SGLang and TensorRT-LLM all support the split, and vLLM still labels its own version experimental. The split moves the cache between pools on every request and gives you a second fleet to size and run. Do not take that on for a product whose prompts are short.
Keep speculative decoding for a route that waits on latency and has compute to spare. A small draft model guesses the next few tokens, and the large model checks all the guesses in one pass. Every accepted guess saves a decode step, but the checking is extra work on a busy server. Measure it at your peak concurrency before you keep it. The EAGLE-3 paper reports speedups of up to 6.5 times, and in October 2024 the vLLM team measured a 1.4 times slowdown on Llama 3 70B at a high request rate.
Quantize last, and only behind an eval. Fewer bits per weight means less memory to read on each decode step, so the stream speeds up and more conversations fit. Current datacenter GPUs add 4-bit floating-point formats to the older 8-bit and 4-bit integer ones. NVIDIA's Blackwell supports NVFP4, and AMD lists the Open Compute Project's MXFP4 for the Instinct MI350X. The risk is a quality loss on your task that a general benchmark does not show. Run an LLM evaluation on your own cases at each bit width before you move traffic.
When should the model run on the user's device?
Put a model on the user's device when the data is not allowed to leave it, and budget for the tuning it takes. On AnyTeam, a meeting transcription app in production, cloud transcription ruled the product out for sales teams with data-residency obligations. The speech model therefore runs inside the desktop app, and there is no upload path for audio. It is a compact, quantized model running on the CPU, with decoding biased toward sales and product vocabulary. Its compute budget is whatever a video call has not already taken.
Nine rounds of tuning took the model from working on a quiet machine to running comfortably faster than real time on ordinary laptops during a live call. We measured word error rate falling from roughly 18% to 10% over the same rounds, with the largest gains on filler words and industry terms. Expect rounds like these whenever a model has to share a laptop with a video call.
Where does caching cut LLM inference cost?
Put the stable parts of the prompt first. Provider prompt caching reuses only a prefix that repeats exactly, such as a long system prompt or a tool list, and skips prefill on it. If one token early in the prompt changes, everything after it misses. As of October 2026 OpenAI applies it automatically above 1,024 tokens on GPT-5.6 and later, and Anthropic bills a cache read at a tenth of the base input price, lower on some of its newest models. It saves nothing on decode.
Build the cache in the application before tuning the provider's, because a call that never reaches the provider saves prefill and decode together. On Prospex, our own prospect research engine, cache-first execution reuses earlier work when a profile is rerun, and we estimate it cut repeated external API calls by 50 to 60 percent. The cache hit rate is the cost metric we watch on it.
Common questions
What is LLM inference?#
LLM inference is running a trained language model on a new prompt to produce an answer, without changing the model's weights. It happens in two phases. Prefill reads the whole prompt in one parallel pass and sets the wait for the first token, and decode then writes the answer one token at a time. Every API call to a hosted model is an inference request.
What is LLM inference optimization?#
LLM inference optimization is the work of making a model answer faster or at lower cost while the answers stay good enough for the product. The common methods are shorter prompts, prompt caching, batching, speculative decoding, quantization and separate GPU pools for prefill and decode. Time every stage of the request first, because on Bigcircle's two voice agents the stage worth cutting sat outside the model call.
What is the difference between prefill and decode?#
Prefill reads the whole prompt in one parallel pass and usually sets the time to first token. Decode generates one token at a time and usually sets how smooth the stream feels and how long the full answer takes. Long prompts stress prefill. Long answers and growing conversations stress decode, because each new token adds cache that the next step has to read.
Why does the KV cache dominate LLM serving memory?#
The cache holds attention state for every token already processed, on every layer, for every active request. For Llama 3 70B in 16-bit precision that is 327,680 bytes per token, so one 32,768-token conversation holds about 10.7 GB. The weights stay fixed while the cache grows with every user and turn, and when it no longer fits the server queues, evicts or recomputes work.
Does prompt caching make generation free?#
No. Prompt caching reuses an exactly repeated prefix, such as a system prompt or a tool list, so the prefill on those tokens is skipped. It does not skip the token-by-token decode, and it misses when anything early in the prompt changes. Bigcircle also caches at the application layer, where a call that never reaches the provider saves both phases.
When should prefill and decode run on separate GPUs?#
Only when long prompts still stall other users' streams after chunked prefill is on, and the gain outweighs moving the cache between pools on every request. Chunked prefill addresses the same stall inside one engine without a second fleet to run, so try it first. As of October 2026 vLLM, SGLang and TensorRT-LLM all support the split, and vLLM still labels it experimental.
Does speculative decoding always make responses faster?#
No. It cuts latency when the server has spare compute, and it can lower throughput when the server is busy, because checking the draft's guesses is extra work. The vLLM team measured a 1.4 times slowdown on Llama 3 70B at a high request rate in October 2024. Keep it for a route that waits on latency, and measure it at your peak concurrency first.
What should we measure before tuning an inference stack?#
Time every stage of the request, not only the model call. On Bigcircle's voice agents those timings showed where the time went: on one the failures were in turn detection and integration rather than in reasoning, and on the other transcription took the extra time. Then track time to first token and time between tokens separately. Record the longest prompts your product actually sends, because an average prompt length hides the requests that set the bill.
Sources
- Li et al., EAGLE-3, 2025. The 6.5 times speedup.
- vLLM, speculative decoding at high load, October 2024, optimization and chunked prefill, and disaggregated prefill.
- SGLang, prefill and decode disaggregation. TensorRT-LLM, disaggregated serving.
- NVIDIA, Introducing NVFP4. Open Compute Project, Microscaling formats (MX) v1.0.
- OpenAI, prompt caching. Anthropic, prompt caching.