Semantic Caching

Semantic caching stores a model's response and returns it when a new request is close in meaning to an earlier one. The match is embedding similarity over a threshold, not an exact string, and the store lives in the application or in an AI gateway. A hit never reaches the model.

Prompt caching is a different cache, and the provider keeps computed KV state for an exact prompt prefix, so one changed character early in the prompt misses. Semantic caching can still hit after a paraphrase, so it saves a whole generation. A bad hit returns someone else's answer with no error.

We have not shipped a semantic cache for a client. On Prospex, our own research engine, we cache at the application layer by exact reuse of earlier extraction and retrieval. By our estimate that cut repeated external API calls by 50 percent to 60 percent. Exact reuse and similarity search are both result caches that fail in different ways, and this page is about the second.

What is semantic caching?#

Semantic caching is an application- or gateway-level store of prior prompts and responses. On a new request it embeds the prompt, searches stored vectors, and returns the stored response when the nearest neighbour clears a similarity threshold. Below the threshold the request goes to the model, and the new pair is stored for later.

Users rarely type the same question twice, so an exact-key cache misses paraphrases that should share an answer. "What is the refund policy" and "How do I get my money back" can share a public FAQ reply. They must not share an answer when the wording looks similar and the records behind it belong to different people.

GPTCache is an open-source library built for this store. Its own quick start, checked 9 November 2026, defaults cache.init() to exact-match evaluation. Semantic matching is a later step: you plug in an embedding function, a vector store, and a distance evaluator.

LiteLLM's proxy, also checked that day, offers semantic backends on Redis, Qdrant and Valkey. It embeds the prompt so a nearby cached response can serve instead of the model.

How semantic caching works#

Walk one request through the store. The cache embeds the new prompt with the same embedding model used at write time, because a mix of models puts the same sentence in two places. Vector search then returns the nearest stored prompts, and cosine similarity, the angle between those vectors, is the usual score. Near 1 means they point the same way, and the embeddings page is the place that score is explained.

A cache hit is a score at or above the threshold. The stored text is returned, the model is skipped, and you pay for the embedding lookup instead of a generation. On a miss the request goes to the model as usual. After the answer comes back, the cache stores the new prompt, its embedding, and the response, so a later neighbour can find it.

The threshold is the whole policy, and LiteLLM's config examples use 0.8 as their starting point rather than a number to copy. Our embeddings page already says two paraphrases might score 0.9 on one model and a related pair 0.6, and that the values shift between models.

A cutoff taken from someone else's post will not hold on your traffic. Set it from labelled pairs: questions that should share an answer, and questions that look close and must not.

The lookup is not free. LiteLLM embeds the prompt inline before the request can reach the model, and as of 9 November 2026 it gives up after 5 seconds, records similarity 0.0, and continues as a miss. A hanging embedding endpoint therefore adds a wait to every call rather than stalling it. Measure that wait with the rest of cost and latency on the path, next to model routing, because a cache that misses still spent the lookup.

Semantic caching vs prompt caching#

Keep both names for the thing they name: prompt caching is provider-side reuse of prefill KV state for a prefix that matches token for token. As of October 2026, OpenAI applies it automatically above 1,024 tokens on GPT-5.6 and later, and a cache read costs a tenth of input, 0.1x, on most of those models. Semantic caching is your store of answers, keyed by meaning. A hit skips prefill and decode together, and a miss still paid for an embedding.

	Prompt caching	Semantic caching
Match	Exact prefix, first differing token breaks it	Embedding similarity at or above a threshold
Where it runs	Model provider	Application or AI gateway
What a hit skips	Most of prefill	The model call
Wrong-answer risk	None from the cache itself	A neighbour that was close and not the same request
Agent fit	A growing transcript is a friendly prefix	Tool-calling turns look like their previous turn

A third cache sits next to both: exact reuse of your own results, keyed on the same inputs, which is the store Prospex runs. A rerun of the same company reuses extraction and retrieval instead of calling out again, and we watch the cache hit rate as its cost metric.

It cannot hit a paraphrase, and it cannot serve one company's profile as another company's, while semantic caching trades that safety for coverage of questions that only mean the same thing.

The agent cost guide already splits the two provider and application savings. Put a stable prefix first so prompt caching can discount it, and put a semantic cache only on traffic where a nearby question is allowed to share an answer. Model routing then still decides which model runs on a miss, measured on cost, latency and quality, which is the split our model routing work uses.

When semantic caching returns the wrong answer#

A cache hit is not a proof of a correct answer. Four failures show up once the store is on the request path, and none of them look like an error in the model.

A false hit is a neighbour that cleared the threshold and was a different question. "Cancel my order" and "cancel my subscription" sit close in embedding space on many models. Serving the first answer for the second looks fluent and is wrong. The fix is a tighter threshold plus labelled near-misses in the eval, not a larger store.

Stale data is the slow failure. A cached refund policy from last quarter still matches this morning's wording. Prompt caching goes cold when the prefix changes, while semantic caching keeps serving the old answer until you expire it. A public FAQ can tolerate a short lifetime that an account balance cannot.

Per-user data leaking across users is the failure that should stop a rollout. LiteLLM's semantic cache, as of 9 November 2026, keys the bucket by virtual key, team and organisation by default, so every end user behind that key shares one store. A response generated for one of them, tool calls included, can be served to another who asks a similar question.

Set semantic_cache_scope: end_user there, or the equivalent in your own store. Our memory isolation guide is the same rule in another layer: key facts to the acting user inside the tenant, because a scoped token does not say whose request this is. On a current telecom engagement we scope each agent to the entitlements of the person it acts for, and a semantic cache that ignored that scope would undo it.

Threshold tuning is an eval problem: a looser cutoff raises hit rate and false hits together, and a tighter one wastes generations on questions you have already answered. On the trademark drafting system, every draft passes a legal-integrity harness before an attorney sees it. A routing change on the AI gateway is scored the same way before it takes traffic. Score a semantic cache on held-out paraphrases, including the pairs that must miss, before it serves anyone.

Semantic caching for LLM agents#

Serving a tool-calling turn from a similarity cache is unsafe because a hit returns a stored action, not a stored answer. Each new turn is the last conversation plus a short message, so the next prompt sits next to it in embedding space. If that stored response was a tool call, the cache fires the same tool again.

LiteLLM's proxy, checked 9 November 2026, embeds the whole messages array, system prompt included, and consecutive agent turns sit at about 0.99 similarity. That score clears any threshold you would ship, so the next turn gets the previous turn's stored tool call. A higher cutoff does not reliably separate those turns, because assistant tool_calls are left out of the embedded text.

Keep the store for single-shot, read-only questions: a public FAQ, a policy lookup, a repeated classification with no side effect. Exclude any path that calls a tool, writes to another system, or reads a per-user record.

For the agent transcript, use prompt caching on a stable prefix. For a rerun of the same job, use exact application-layer reuse of the kind Prospex runs. LiteLLM's own guidance, checked the same day, is to put agents on an exact-match cache, where any change to the conversation is a miss.

Skip the semantic store while the questions are unique, the answers go stale inside a day, or a person already reviews every reply. Spend that effort on the eval set and on the isolation keys. A cache that serves the wrong tenant's answer is worse than no cache, because the leak is silent and the invoice looks better.