Semantic caching lets an application answer an incoming LLM query by returning a previously generated response when the new question is semantically close to a cached one — no model call at all. The payoff concentrates where questions repeat: FAQ bots, helpdesk triage, internal knowledge assistants, and any assistant serving a stable knowledge base. One published evaluation of a Redis-backed semantic cache reports hit rates between 61.6% and 68.8% while reducing LLM API calls by up to 68.8%, with positive hit rates exceeding 97% — figures measured on that study’s workload, not a promise for yours. What follows is the decision math: when a semantic layer earns its keep, how to tune it, and how to stop paraphrased lookups from serving wrong answers in production.
What semantic caching actually stores
A semantic cache stores complete LLM responses, not document chunks and not token prefixes. Each entry holds the prompt, its embedding vector, the model’s answer, and metadata such as tenant and model version; Redis documents this pattern as one Hash or JSON document per entry covered by a vector index over the embedding and metadata fields. A lookup is a nearest-neighbour search over the cached prompt embeddings under a distance threshold; a hit returns the stored answer and skips the model entirely, while a miss runs the model and writes the result back.
The distinction matters because two adjacent techniques get conflated with it. Provider prompt caching discounts repeated prefixes but still runs the model end-to-end on every call, so it lowers input cost without reusing a full response. RAG retrieval feeds context into a model that still runs. On the cost gap, the Redis documentation for this use case is blunt: repeated or paraphrased questions that reach an LLM drive per-query cost up by 10–100x compared to a cache lookup, and push P95 latency into multi-second territory while a cache hit returns in tens of milliseconds. Duplication is the qualifying condition — if your latency pain comes from novel queries rather than repeats, engine-level work like speculative decoding in vLLM is the right lever, not a cache.
Choosing your cache layers
Most stacks need two of the three layers below, and the choice follows from which kind of duplication you actually have:
| Layer | Match basis | What it saves | Main failure mode | Best workload |
|---|---|---|---|---|
| Exact-match cache | Byte-identical prompt key | Latency and tokens on exact repeats | Misses every paraphrase | Idempotent jobs, deterministic tool outputs |
| Semantic cache | Embedding distance under a threshold | The full model call on paraphrases | Serving a near-miss answer as correct | FAQ and helpdesk-style assistants |
| Provider prompt caching | Repeated prompt prefixes | Input-token cost on shared prefixes | No latency win, no full-response reuse | Long system prompts, multi-turn sessions |
Exact-match hits are free from a correctness standpoint but blind to phrasing: the Redis docs single out “What’s your return policy?” versus “How do I return an item?” as the near-duplicates that dominate FAQ traffic and that a string key never matches. That blindness is the entire reason the semantic layer exists.
Tuning the similarity threshold
The threshold is where semantic caching succeeds or fails. The Redis guidance names the problem plainly: the core difficulty is threshold tuning — too loose and you serve wrong answers, too tight and the hit rate collapses. Effective semantic caching combines soft similarity matching with hard metadata boundaries — tenant, locale, model version, safety flags — so reuse stays inside well-defined limits, and those boundaries belong inside the vector query itself rather than in post-filtering application code.
As an example starting point for a single-tenant internal assistant, a cosine distance threshold in the neighbourhood of 0.15 with tenant and model-version filters enforced in the search is a reasonable first guess — treat that as a hypothesis to sweep, not a measured recommendation. Sweep it on your own logged queries: at each candidate threshold, measure the hit rate and the wrong-answer rate against fresh model answers, and pick the point where the wrong-answer rate is low enough for your product, not the point that maximizes savings.
Guarding correctness under churn
A semantic cache is not a source of truth; it is a non-authoritative, rebuildable shortcut. Three guards keep it honest under churn. First, TTLs on every entry so stale answers age out as the underlying knowledge base moves, with LRU or LFU eviction bounding memory when the prompt distribution shifts. Second, model version in the metadata key: when you upgrade the model behind the assistant, the filter prevents cross-version reuse and the cache warms up cleanly. Third, continuous evaluation — sample served cache hits and score them against fresh answers in the same harness you use for production tracing and evaluation, so quality regressions surface as metrics rather than user complaints.
On expected savings, keep the framing honest: Redis’s own guidance for this pattern is that FAQ-style workloads can shed 30% or more of token spend without a measurable quality regression, and the arXiv evaluation above reported larger reductions on its particular workload. Your number is a function of your duplication rate, so measure it rather than quoting either figure in a planning document.
A safe rollout procedure
This is the order that keeps wrong answers out of production:
- Quantify duplication before writing any code: embed a week of production queries and measure how many cluster under a candidate distance threshold. No meaningful clustering, no project.
- Run the cache in shadow mode first — log would-be hits and still call the model.
- Score every shadow hit against the fresh model answer using your existing evaluation harness.
- Sweep the threshold offline using those scores, and fix the metadata boundaries you will enforce in the vector query.
- Set TTL and eviction policy, and record that the cache is non-authoritative and rebuildable.
- Enable progressively behind a kill switch, monitoring hit rate and sampled answer quality together.