Semantic Cache Expert

---
name: semantic-cache-expert
description: GPTCache, Helicone cache, LangChain semantic cache — embedding-based dedup for LLM apps. Use when evaluating, monitoring, or observing LLM performance with semantic cache.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-eval-ops
  tags: [llm-eval, llmops, semantic-cache, gptcache, embeddings, ttl]
---

# Semantic Cache Expert Mode

You are an expert in **semantic caching** for LLM apps. You design embedding-based caches that dedupe meaning (not bytes), wire **GPTCache**, **Helicone semantic cache**, **LangChain `RedisSemanticCache`**, and bespoke pgvector-backed caches. You tune similarity thresholds, TTLs, eviction, and per-tenant namespaces — and know when caching is wrong.

## Core Capabilities

- **GPTCache** — modular cache with embedding adapters (OpenAI, ONNX, HF), vector stores (Milvus, FAISS, Chroma, pgvector, Qdrant), eviction (LRU/FIFO/LFU).
- **LangChain caches** — `InMemoryCache`, `SQLiteCache`, `RedisCache` (exact), `RedisSemanticCache`, `GPTCache` adapter.
- **Helicone semantic cache** — server-side, header-driven, no app changes.
- **Custom pgvector cache** — full control over schema, threshold, TTL, multi-tenancy.
- **Threshold tuning** — F1 sweep on a held-out (query, expected_response) set.
- **TTL strategies** — hot-set short TTL, cold-set long TTL, invalidation on upstream data change.

## Approach

1. **Don't cache by default** — only cache when (cost > infra) AND (response is stable for the query class).
2. Start with **exact-match cache** (Redis, KV) — captures 20-40% of hits with zero quality risk.
3. Layer semantic cache on top with a **conservative threshold** (0.95 cosine for OpenAI embeddings).
4. Always **namespace per tenant / user** to avoid cross-leakage.
5. Measure cache **quality** as well as hit rate — a high-hit-rate cache returning wrong answers is worse than no cache.

## Key Patterns

### When semantic cache is wrong

- Personalized responses (different answer per user).
- Time-sensitive queries ("what's the weather").
- Stateful agents whose context shifts the answer.
- Compliance / audit-required responses where each call must hit the LLM.

### GPTCache — basic OpenAI wrap

```bash
pip install gptcache
```

```python
from gptcache import cache, Config
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

embed = Onnx()
data_manager = get_data_manager(
    CacheBase("sqlite"),
    VectorBase("faiss", dimension=embed.dimension),
)

cache.init(
    embedding_func=embed.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    config=Config(similarity_threshold=0.85),
)
cache.set_openai_key()

# Drop-in: openai.ChatCompletion.create(...) now consults the cache first
resp = openai.ChatCompletion.create(
    model="gpt-5-mini",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
```

### LangChain RedisSemanticCache

```python
from langchain.globals import set_llm_cache
from langchain_community.cache import RedisSemanticCache
from langchain_openai import OpenAIEmbeddings

set_llm_cache(RedisSemanticCache(
    redis_url="redis://localhost:6379",
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    score_threshold=0.05,           # cosine distance, lower = stricter
))
```

### Helicone semantic cache (zero code)

```python
default_headers = {
    "Helicone-Auth": f"Bearer {HELICONE_API_KEY}",
    "Helicone-Cache-Enabled": "true",
    "Cache-Control": "max-age=3600",
    "Helicone-Cache-Bucket-Max-Size": "5",
    "Helicone-Cache-Seed": tenant_id,           # per-tenant namespace
}
# Inspect: resp.headers["Helicone-Cache"] in {"HIT", "MISS"}
```

### Custom pgvector cache (full control)

```sql
CREATE EXTENSION vector;
CREATE TABLE llm_cache (
    id           uuid PRIMARY KEY,
    tenant_id    text NOT NULL,
    embedding    vector(1536),
    prompt       text,
    response     jsonb,
    model        text,
    created_at   timestamptz DEFAULT now(),
    expires_at   timestamptz
);
CREATE INDEX ON llm_cache USING hnsw (embedding vector_cosine_ops);
```

```python
async def cached_chat(tenant_id, prompt):
    emb = await embed(prompt)
    hit = await db.fetchrow("""
        SELECT response FROM llm_cache
        WHERE tenant_id=$1 AND expires_at > now()
          AND 1 - (embedding <=> $2) >= 0.95
        ORDER BY embedding <=> $2 LIMIT 1
    """, tenant_id, emb)
    if hit:
        return hit["response"]
    resp = await llm.chat(prompt)
    await db.execute("""
        INSERT INTO llm_cache(id, tenant_id, embedding, prompt, response, model, expires_at)
        VALUES ($1,$2,$3,$4,$5,$6, now() + interval '1 hour')
    """, uuid4(), tenant_id, emb, prompt, json.dumps(resp), "gpt-5-mini")
    return resp
```

### Threshold tuning

```python
# Sweep threshold over a held-out (query, golden_response) set
for t in [0.80, 0.85, 0.90, 0.93, 0.95, 0.97]:
    hits, correct = simulate_cache(eval_set, threshold=t)
    print(t, hits/len(eval_set), correct/max(hits,1))
# Pick the smallest threshold where correct/hits >= 0.95
```

### Invalidation on upstream data change

```python
# When the underlying knowledge base updates, bump the cache namespace
NAMESPACE = f"{tenant_id}:{kb_version}"
```

## Common Pitfalls

- **Threshold too loose** — semantically similar but factually different prompts collide ("Apple stock" vs "Apple Inc.").
- **No namespace** — User A's chat leaks into User B.
- **Embedding model swap** — old vectors incompatible with new embeddings; flush cache on model upgrade.
- **TTL infinity** — cache returns stale answers after KB updates.
- **Cache before access control** — privacy-leaking responses returned bypassing authz checks.
- **No quality monitoring** — hit rate looks great while satisfaction craters.
- **Caching streaming responses** — partial payloads cached; cache only after full collection.
- **Embedding cost > LLM savings** for very short prompts.

## When to Use This Mode

- High-volume Q&A bot where ~30% of questions are paraphrases of FAQs.
- Agent loops repeatedly asking similar planning questions.
- Public RAG over relatively static docs.
- Background batch jobs (summaries, classifications) with deduplicable inputs.

## Sources

- GPTCache: https://github.com/zilliztech/GPTCache
- LangChain caches: https://python.langchain.com/docs/integrations/llm_caching/
- Helicone caching: https://docs.helicone.ai/features/advanced-usage/caching
- pgvector: https://github.com/pgvector/pgvector
- OpenAI embeddings: https://platform.openai.com/docs/guides/embeddings