Semantic Caching for LLM APIs

By Michael Samuel Naeem · October 28, 2024 · 5 min read

"What's your return policy?", "How do returns work?", and "Can I send something back?" are three cache keys in an exact-match system and one answer in a semantic cache. That's the promise — and the risk. Semantic caching returns cached responses for paraphrased inputs by comparing embeddings instead of strings. Get the threshold wrong and you return the refund policy when someone asked about returns on a digital product.

How semantic cache works

Query → embed → search cache index (similar queries)
                      ↓
              similarity ≥ threshold?
                 /        \
              yes          no
               ↓            ↓
        return cached    call LLM → store (query_embed, response)

Each cache entry stores:

@dataclass
class SemanticCacheEntry:
    query_text: str
    query_embedding: list[float]
    response: str
    tenant_id: str
    metadata: dict  # prompt_version, source_chunks, created_at
    ttl_expires: datetime

Implementation

async def semantic_lookup(
    query: str,
    tenant_id: str,
    threshold: float = 0.93,
) -> str | None:
    embedding = await embed(query)
    hits = await vector_index.search(
        embedding,
        filter={"tenant_id": tenant_id},
        top_k=1,
    )
    if not hits or hits[0].score < threshold:
        return None
    entry = await store.get(hits[0].id)
    if entry.ttl_expires < now():
        return None
    return entry.response

Use the same embedding model for cache lookup and retrieval — mixing models breaks similarity scores.

Two-tier cache

async def get_response(query: str, tenant_id: str) -> str:
    # Tier 1: exact match (Redis)
    exact_key = hash(tenant_id, normalize(query))
    if hit := await redis.get(exact_key):
        return hit

    # Tier 2: semantic (vector index)
    if hit := await semantic_lookup(query, tenant_id):
        metrics.semantic_hit()
        return hit

    # Miss: full LLM call
    response = await llm.complete(query, tenant_id=tenant_id)
    await store_both(exact_key, query, response, tenant_id)
    return response

Exact tier catches identical repeats (CI tests, bots). Semantic tier catches human paraphrasing.

Threshold calibration

Don't guess 0.95. Build a calibration set:

1. Collect 200 query pairs labeled "same intent" / "different intent"
2. Compute embedding similarity for each pair
3. Plot ROC curve — pick threshold at acceptable false-positive rate

For action intents, consider semantic cache lookup only to suggest "Did you mean X?" rather than auto-returning.

False positive controls

Additional guards beyond similarity:

• Intent tag matching — cache entry tagged billing; query classified as billing only
• Entity overlap — "refund for order 123" vs "refund for order 456" share structure but different entities; extract entities and require match
• Time sensitivity check — if cached entry is older than 1 hour and query contains temporal words ("today", "now"), bypass cache

def safe_to_serve(cached: SemanticCacheEntry, query: str) -> bool:
    if cached.metadata.get("intent") != classify_intent(query):
        return False
    if has_entity_mismatch(cached.query_text, query):
        return False
    return True

Storage and eviction

Semantic caches grow unbounded without eviction:

• LRU with max entries per tenant
• TTL by category (FAQ: 24h, product info: 4h)
• Size cap on response text stored

Use a dedicated vector index (not your RAG index) for cache entries — mixing pollutes retrieval and complicates TTL.

Measuring value

Track:

• Semantic hit rate vs exact hit rate
• False positive rate (from user thumbs-down after cache hit — tag these)
• Latency saved (cache hit p50 vs LLM p50)
• Cost saved per day

If false positive rate exceeds 1%, raise threshold or restrict to informational intents.

When semantic cache isn't worth it

• Low query volume (< 1000/day) — exact cache is enough
• Highly personalized responses — every answer is unique
• Rapidly changing content — invalidation complexity exceeds savings
• Regulated domains where wrong cached answers have legal exposure

Multi-tenant cache isolation

Semantic caches are a cross-tenant leak vector if keys ignore tenant boundaries. Always embed tenant_id, model_version, and prompt_template_version in the cache lookup key and in the vector metadata filter:

def lookup_cache(tenant_id: str, query: str, embedding: list[float]) -> CacheHit | None:
    return vector_index.search(
        embedding,
        filter={"tenant_id": tenant_id, "model": MODEL_VERSION},
        threshold=0.92,
    )

A support bot for Tenant A must never return an answer cached from Tenant B's similar question — even if embeddings are nearly identical. Enterprise contracts treat this as a data breach, not a cache miss.

Invalidation strategies

Semantic caches need explicit invalidation, not just TTL:

Batch invalidation beats scanning the full index. Tag every cache entry with the document IDs or product SKUs that informed the answer so a CMS publish event can purge affected rows in O(tags) rather than re-embedding everything.

Production rollout checklist

• Exact cache layer in front of semantic cache (cheaper, zero false positives)
• Similarity threshold tuned on held-out query pairs with human labels
• False positive dashboard: thumbs-down tagged with cache_hit=true
• Per-tenant hit rate and cost savings visible in billing dashboard
• Bypass cache for authenticated actions (refunds, account changes)
• Load test: cache at 10× expected QPS without vector index latency regression

Start with informational intents only. Expand to transactional intents only after false positive rate stays below 0.5% for two weeks.

Quick reference

Review cache hit rate weekly alongside support ticket volume — a rising ticket rate with rising hit rate signals false positives, not success.

Resources

• GPTCache semantic cache module
• Redis vector search for caching
• Zilliz semantic cache patterns
• LangChain semantic cache integration
• Embeddings similarity benchmarks (MTEB)