TTFT vs TBT (Latency Metrics)

"How fast is the model?" is the wrong question, because an LLM has no single latency. A response streams out over seconds, so the experience is governed by two distinct clocks: how long until the first token appears, and how quickly tokens follow after that. These map directly onto the prefill and decode phases of inference, have different root causes, respond to different fixes, and need separate budgets. Average them into one number and you can no longer tell which one is hurting you.

TTFT: the wait before anything appears#

Time to first token (TTFT) is the delay from sending the request to receiving the first output token. It is dominated by the prefill phase described in prefill vs decode: the model must process your entire prompt in one compute-heavy pass before it can emit anything. On top of prefill sits queueing, the time the request spent waiting for a GPU slot, which is often the larger share under load.

Because prefill cost scales with prompt length, TTFT is the metric that grows as your prompts grow. A 500-token question might prefill in 100 ms while a 50,000-token RAG context on the same hardware takes several seconds; users experience that as the product getting slower as it gets smarter. This is also the metric prompt caching most directly improves: a cache hit on a long shared prefix skips most of prefill, routinely cutting TTFT by 50 to 80% on long-context workloads. Common targets: under 500 ms for a chatbot to feel responsive, and closer to 200 ms for inline experiences like code completion, where the suggestion must beat the next keystroke.

TBT: the cadence of the stream#

Time between tokens (TBT), also called inter-token latency (ITL) or time per output token (TPOT), measures the gap between successive tokens once streaming starts. It is bound by the decode phase: every token requires streaming the model weights and the growing KV cache through the GPU's memory bus, so TBT is a memory-bandwidth number, largely independent of prompt length. It degrades as continuous batching packs more concurrent requests onto the same hardware, which is exactly the throughput-versus-latency trade Databricks' engineering guide centers on: bigger batches mean more total tokens per second for the operator and slower tokens for each user.

The useful reference point for TBT is human reading speed: about 4 to 5 words per second, roughly 6 to 8 tokens per second. Any TBT under about 120 ms outpaces reading, and the common chat target of 30 to 50 tokens per second (TBT of 20 to 33 ms) leaves comfortable headroom. Below reading speed the stream visibly stutters; far above it, extra speed buys little for a human and you may be better off trading it for batch density.

Why streaming changes perceived latency#

Take a 500-token answer with TTFT of 800 ms and TBT of 40 ms. Total time is 0.8 + 499 x 0.04, about 20.8 seconds. Without streaming, the user stares at a spinner for the full 20.8 seconds and most will abandon. With streaming, they see text moving at 0.8 seconds and read along while the rest generates; the same end-to-end latency feels fine. Streaming does not make anything faster, it changes which metric the user feels: TTFT becomes the perceived latency, and TBT only needs to stay ahead of reading speed.

This is why "optimize end-to-end latency" is usually the wrong instruction for a chat product. Cutting TBT from 40 ms to 20 ms halves end-to-end time but is barely perceptible to a reader, while cutting TTFT from 3 seconds to 500 ms transforms the experience and changes the end-to-end number by only 2.5 seconds.

Separate budgets, and P50 vs P99#

Because the two metrics have different causes, set a budget for each rather than one combined SLO. A reasonable chat budget: TTFT P50 under 500 ms and P99 under 2 seconds; TBT P50 under 40 ms and P99 under 150 ms. A TTFT breach points at queueing, prompt growth, or a cold prompt cache; a TBT breach points at batch pressure, long-context KV cache reads, or preemption. With separate dashboards, the diagnosis is instant; with one blended latency chart, it is an investigation.

Percentiles matter more than means here because LLM serving has fat tails. P50 describes the typical user; P99 describes the user who hit a queue spike, a 100k-token prompt, or a preempted request, and in a product with thousands of daily sessions, P99 is an everyday experience, not a corner case. Tail TBT deserves special attention: a stream that averages 25 ms per token but freezes for 3 seconds mid-answer feels broken, yet the mean barely moves. The Etalon paper formalizes exactly this failure of averaged metrics and proposes measuring stall-free fluidity instead; even without adopting its metric, plotting your TBT P99 alongside P50 catches most of what it warns about.

End-to-end, tokens per second, and metric traps#

End-to-end latency still composes cleanly: TTFT plus TBT times (output tokens minus 1). It remains the metric that matters for non-streamed programmatic calls, agent tool loops, and anything where downstream code waits for the full response; there, output length dominates, and the cheapest optimization is usually generating fewer tokens. Note that NVIDIA's benchmarking docs define ITL backwards from this identity, as (end-to-end latency minus TTFT) divided by (tokens minus 1), which is why different tools report slightly different numbers for the same run.

Tokens per second is the slipperiest term of all, because it names three different things: per-user output speed (the inverse of TBT), system-wide output throughput across all requests, and input-plus-output processing rate, which inflates the number with cheap prefill tokens. A vendor quoting "10,000 tokens per second" on the system level says nothing about whether each user gets 8 or 80. Always ask which one a benchmark means, and whether averages are request-weighted or token-weighted, since long responses skew the latter.

Practical takeaways#

Instrument TTFT and TBT separately, at the client, with P50 and P99 on both; one blended latency number cannot tell a prefill problem from a decode problem. Budget TTFT against patience (around 500 ms for chat) and TBT against reading speed (under about 100 ms), then spend optimization effort where the breach actually is: prompt caching and queue management for TTFT, batch tuning and context discipline for TBT. Stream anything a human watches, keep end-to-end latency as the metric for machine-to-machine calls, and never accept a tokens-per-second claim without asking which tokens, whose, and at which percentile.