Prompt Caching

Most production prompts are mostly identical from call to call. A long system prompt, a tool catalog, few-shot examples, or a 50-page reference document gets re-sent on every request while only the user's latest message changes. Without caching, the model reprocesses that whole prefix from scratch each time, and you pay full input price for the privilege. Prompt caching stores the model's intermediate work on the stable prefix so the next request skips straight to the new part.

What gets cached: the prefill, not the text#

During prefill, the model runs every prompt token through attention and stores per-layer key/value tensors, the KV cache. Prompt caching saves that computed state between requests, not the raw text. Because each token's entries depend on every token before it, the saved state is only valid for an exact, contiguous prefix starting at position zero. Change one character early in the prompt and everything after that point must be recomputed.

That single fact dictates prompt design: stable content first (system prompt, tool definitions, documents, examples), variable content last (the user's question, per-query retrieved chunks). The classic self-inflicted miss is a timestamp at the top of the system prompt: "Current time: 14:32:07" makes every request a unique prefix and silently disables caching for the entire prompt.

The economics, with mid-2026 numbers#

On the Claude API, cache reads cost 0.1x the base input price, a 90% discount on the cached portion. Writing the cache costs a one-time premium: 1.25x base input for the default 5-minute lifetime, or 2x for a 1-hour lifetime. Work a real example: an agent with a 50,000-token stable prefix on a model priced at $5 per million input tokens pays $0.25 per call for that prefix uncached. With caching, the first call writes for about $0.31, and each later call reads for $0.025. Over a 20-turn session the prefix costs $0.79 instead of $5.00, an 84% saving, and the math only improves with longer sessions.

OpenAI charges no write premium at all: caching is automatic and free, and the cached-token discount varies by model, roughly 90% on recent GPT-5 series models and about 50% on older ones like GPT-4o. Gemini's cached tokens also run about 10% of standard input price on current models (Gemini 2.5 Pro: $0.125 versus $1.25 per million), with explicit caches additionally billing storage per million tokens per hour, about $1 on Flash tiers and $4.50 on Pro tiers.

The latency win is just as real as the cost win, because a cache hit skips most of prefill, the slow part of long prompts. Anthropic reports up to 85% latency reduction on long prompts, with a 100,000-token cached document cutting time-to-first-token from 11.5 seconds to 2.4. OpenAI cites up to 80% TTFT reduction on long prompts.

Implicit or explicit: the three providers differ#

OpenAI caching is fully implicit. Any prompt of 1,024 tokens or more is cached automatically, hits register in 128-token increments, and there is nothing to configure. The one knob is prompt_cache_key, which pins requests sharing a prefix to the same inference machines; in one case study in OpenAI's cookbook, adding it lifted hit rates from 60% to 87%. Each prefix-plus-key combination sustains roughly 15 requests per minute before traffic spills to other machines and hit rates dip.

Gemini offers both modes. Implicit caching is on by default for Gemini 2.5 and newer (minimum 2,048 tokens on 2.5 Pro and Flash) and applies the discount whenever a hit happens, with no guarantee that it will. Explicit caching has you create a cached-content object once and reference it per request, guaranteeing the discount in exchange for managing the cache and paying storage by the token-hour.

Anthropic is explicit only: you mark cache breakpoints with cache_control on the blocks you want cached, with a minimum of 1,024 cacheable tokens on most current models (4,096 on some smaller ones). Below the minimum the request silently runs uncached. The tradeoff across all three: implicit is zero effort with no guarantee, explicit is predictable savings with a little API surface to manage.

Structuring prompts to maximize hits#

Order content by how often it changes: tool definitions, then system prompt, then reference documents, then conversation history, then the current query. On the Claude API this matches the invalidation hierarchy, where a change to tools invalidates the system and message caches behind it. Keep serialization byte-stable too: reordering tool definitions or letting JSON key order drift between requests is an invisible cache killer. Conversation history is naturally cache-friendly because it is append-only, so each turn of an agent loop reads the existing prefix cheaply and extends the cache by one turn. And check the usage fields (cache_read_input_tokens on Anthropic, cached_tokens on OpenAI) instead of assuming hits are happening; most caching surprises are discovered in billing, not in code review.

TTL and the fine print#

Caches are ephemeral. Anthropic's default lifetime is 5 minutes, refreshed at no extra cost every time the cached content is read, so steady traffic keeps a prefix warm indefinitely while idle ones evaporate; the 1-hour tier (2x write) exists for traffic with gaps. OpenAI evicts after 5 to 10 minutes of inactivity and always within about an hour, with extended retention up to 24 hours available on recent GPT-5 models. Gemini explicit caches live for the TTL you set, defaulting to 1 hour, with storage billed for the duration. The caveats follow directly: bursty or once-a-day traffic may never recoup an Anthropic write premium, a Gemini explicit cache held all day for occasional queries can cost more in storage than it saves in reads, and any deploy that edits the system prompt cold-starts every cache at once.

Practical takeaways#

Treat caching as free money for any workload that reuses context: chatbots with long system prompts, RAG over a stable corpus, agents replaying growing histories. Put stable content first and keep it byte-identical, push anything per-request to the end, and never embed fine-grained timestamps early in the prompt. Pick the TTL to match your traffic shape, and measure hit rates in the usage metadata before trusting projected savings. A well-structured prompt routinely cuts input spend 70 to 90% and time-to-first-token by similar margins, which makes prompt structure one of the highest-leverage performance changes you can ship without touching a model.