Time to First Token vs Time Between Tokens

Time to first token (TTFT) is the delay between sending a request to a language model and receiving the first token of its reply. Time between tokens (TBT) is the gap between each token after that. A reply streams out over seconds, so "how fast is the model?" has two answers, one for each clock. They map onto the prefill and decode phases of inference, have different causes, respond to different fixes, and need separate budgets. Average them into one number and you can no longer tell which one is hurting you.

What sets time to first token?#

TTFT runs from sending the request to receiving the first output token. Most of it is the prefill phase described in prefill vs decode: the model processes the whole prompt in one compute-heavy pass before it can emit anything. On top of prefill sits queueing, the time the request waited for a GPU slot, which under load is often the larger share.

Prefill cost grows with prompt length, so TTFT is the metric that grows as your prompts grow. A short question prefills in a fraction of a second, while a retrieval-heavy prompt of tens of thousands of tokens on the same hardware can take seconds, and users feel the product slowing down as it gets better informed. Prompt caching is the direct fix: a cache hit on a long shared prefix skips most of prefill. Anthropic's documentation says time to first token generally improves for long cached prompts without promising a figure, so measure the gain on your own prompts rather than quoting one.

What sets time between tokens?#

TBT, also called inter-token latency (ITL) or time per output token (TPOT), is the gap between successive tokens once streaming starts. Decode sets it: every token requires reading the model weights and the growing KV cache through the GPU's memory bus, so TBT is a memory-bandwidth number and depends far less on prompt length. It degrades as continuous batching packs more concurrent requests onto the same hardware, the throughput-against-latency trade that Databricks' engineering guide is built around: bigger batches mean more tokens per second for the operator and slower tokens for each user.

The useful reference for TBT is reading speed. Brysbaert's 2019 meta-analysis put adult silent reading of English non-fiction at 238 words a minute, about four words or five tokens a second. A TBT under about 150 ms keeps ahead of most readers. Below reading speed the stream visibly stutters. Far above it, extra speed buys a human reader little, and an operator can trade it for more requests per GPU.

Why does streaming change which metric users feel?#

Take a 500-token answer with a TTFT of 800 ms and a TBT of 40 ms. Total time is 0.8 + 499 x 0.04, about 20.8 seconds. Without streaming, the user watches a spinner for 20.8 seconds. With streaming, text starts moving at 0.8 seconds and they read along while the rest generates. Streaming makes nothing faster. It changes which metric the user feels: TTFT becomes the perceived latency, and TBT only has to stay ahead of reading.

So "cut end-to-end latency" is usually the wrong instruction for a chat product. Cutting TBT from 40 ms to 20 ms nearly halves end-to-end time and a reader barely notices. Cutting TTFT from 3 seconds to 500 ms changes the experience and moves end-to-end time by only 2.5 seconds.

End-to-end time still matters wherever code, not a person, waits for the whole response: an agent's tool loop, a batch job, a voice pipeline that cannot speak until it has a sentence. There output length dominates, and the fastest fix is usually generating fewer tokens.

Which budget do we set, and at which percentile?#

Bigcircle recommends a separate budget for each metric rather than one combined target, because a breach of each points somewhere different. A TTFT breach points at queueing, prompt growth or a cold prompt cache. A TBT breach points at batch pressure, long-context cache reads or preemption. A reasonable starting budget for chat is TTFT under 500 ms at the median and under 2 seconds at P99, and TBT under 40 ms at the median and under 150 ms at P99. Tighten them for inline completion, where the suggestion has to arrive before the next keystroke.

Percentiles matter more than means because LLM serving has long tails. The median describes the typical user. P99 describes the one who hit a queue spike, a 100,000-token prompt or a preempted request, and in a product with thousands of sessions a day that person exists every day. Tail TBT needs particular care. A stream that averages 25 ms per token and freezes for 3 seconds mid-answer feels broken, and its mean barely moves. NVIDIA's NIM benchmarking defines ITL as one average per request, (end-to-end latency minus TTFT) divided by (output tokens minus 1), and that average cannot show the freeze. The Etalon paper argues the standard metrics miss this and proposes a fluidity index. Without adopting it, record every gap between tokens and plot the P99 of the gaps next to the median.

Measure per stage, at the client#

On the insurance voice agent we built, end-to-end response fell from 2.5 to 3 seconds to around 900 milliseconds on noisy bilingual calls in testing. Latency is a property of the whole path (model, network, telephony, synthesis), so reproduce the number on your own path. We instrumented latency per stage, because "the agent felt slow" is otherwise an unfalsifiable report. On the real-estate voice agent that instrumentation showed the Hindi and Hinglish paths spending their extra time in transcription rather than in the model, where replies took under 1.3 seconds against under 1.2 for English in testing. One blended latency figure would have sent the work to the wrong stage.

Measure TTFT and TBT at the client, not only on the server, because the network and any gateway sit between the two. And read tokens-per-second claims with care, because the term names three things: per-user output speed (the inverse of TBT), system-wide output throughput across all requests, and input-plus-output processing rate, which inflates the figure with prefill tokens. A system-level "10,000 tokens per second" says nothing about whether each user gets 8 or 80. Ask which one a benchmark means, at which percentile, and whether averages are weighted by request or by token, since long responses skew the second.