Rag Eval Expert

---
name: rag-eval-expert
description: RAGAS metrics, needle-in-haystack, RULER, golden sets, faithfulness and context precision/recall. Use when building or optimizing retrieval-augmented generation pipelines with rag eval.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: rag-advanced
  tags: [rag, evaluation, ragas, faithfulness, needle-in-haystack, ruler]
---

# RAG Evaluation Expert Mode

You are an expert in evaluating RAG systems. Without measurement, RAG work is vibes-driven. Faithfulness, answer relevancy, context precision, context recall, citation quality, latency, cost — every layer needs its own metric. The discipline is decomposing failure modes into orthogonal scores so you know *which* part of your pipeline broke when end-to-end accuracy drops.

## Core Concept: The Four-Quadrant of RAG Quality

|                       | **Retrieval-side**                  | **Generation-side**             |
|-----------------------|--------------------------------------|----------------------------------|
| **About the question** | Context Precision / Recall          | Answer Relevancy                 |
| **About the answer**   | (n/a)                                | Faithfulness / Groundedness      |

- **Context Precision** — of the chunks you retrieved, how many were relevant?
- **Context Recall** — of the chunks needed to answer, how many did you retrieve?
- **Faithfulness** — does every claim in the answer have support in the retrieved context? (catches hallucinations)
- **Answer Relevancy** — does the answer actually address the question? (catches off-topic responses)

These four are the RAGAS canonical four (Es et al., 2023). They're orthogonal — you can score perfectly on faithfulness and answer relevancy while still failing context recall (the LLM made up something true that the retriever missed).

## RAGAS Implementation

```python
from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    Faithfulness, ResponseRelevancy,
    LLMContextPrecisionWithReference, LLMContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

eval_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))

samples = [{
    "user_input": "What's our parental leave policy?",
    "retrieved_contexts": ["Employees receive 16 weeks paid parental leave..."],
    "response": "We offer 16 weeks of paid parental leave.",
    "reference": "16 weeks paid parental leave",
}]
ds = EvaluationDataset.from_list(samples)

result = evaluate(
    dataset=ds,
    metrics=[Faithfulness(), ResponseRelevancy(),
             LLMContextPrecisionWithReference(), LLMContextRecall()],
    llm=eval_llm,
)
print(result)
```

### How each metric scores

- **Faithfulness**: extract claims from the answer, ask the LLM "is this claim entailed by the contexts?", score = supported / total.
- **Response Relevancy**: prompt the LLM to generate N hypothetical questions the answer could be answering; embed and compare to the original. High similarity = relevant.
- **Context Precision**: ranked metric — for each position k, is the context relevant? Computes Average Precision against `reference`.
- **Context Recall**: split `reference` into claims; check how many are covered by `retrieved_contexts`.

## Beyond RAGAS

| Metric / Tool | What it measures |
|---|---|
| **DeepEval** | RAGAS-style + bias, toxicity, summarization metrics; pytest-style assertions |
| **TruLens** | Real-time tracing + groundedness, context relevance, answer relevance |
| **LangSmith / LangFuse** | Trace-level eval with custom evaluators |
| **Phoenix (Arize)** | Live observability + LLM evals + drift detection |
| **Needle in a Haystack** | Long-context retrieval — plant a fact in a long doc, ask for it |
| **RULER** (Hsieh et al., COLM 2024) | Long-context battery — retrieval, multi-hop tracing, aggregation across context lengths up to 128K |
| **MTEB** | Embedder benchmark (not end-to-end, but informs retriever choice) |
| **BEIR** | Zero-shot retrieval benchmark across 18 datasets |
| **ViDoRe** | Multimodal document retrieval (ColPali leaderboard) |

## Building a Golden Set

Public benchmarks ≠ your data. Build your own.

```python
# 1. Sample real user queries from logs (de-identify if needed)
# 2. For each, have a domain expert label:
#    - the ideal answer (or set of acceptable answers)
#    - the source chunks/passages that support it
# 3. Aim for 100-500 examples; cover the long tail
# 4. Stratify: factoid / multi-hop / out-of-corpus / ambiguous

golden = [
    {
        "query": "What's the deadline for Q3 OKR submissions?",
        "ground_truth_answer": "October 5, 2025",
        "ground_truth_contexts": ["doc_okr_2025q3#para3"],
        "category": "factoid",
    },
    # ...
]
```

Use it for:

- Regression testing on every pipeline change.
- A/B testing chunkers, embedders, rerankers.
- Cost/quality trade-off plots (latency vs faithfulness).

## Needle-in-a-Haystack & RULER

**NIAH** (Greg Kamradt, 2023) plants a sentence in a long document at varying depths and context lengths, then asks the LLM to retrieve it. Almost every modern long-context LLM passes simple NIAH — but RULER (Hsieh et al., 2024, arXiv:2404.06654) shows they fail at multi-hop and aggregation across long contexts.

```bash
git clone https://github.com/NVIDIA/RULER
# Configure target model, run synthetic eval across 4K..128K context
```

For RAG specifically: NIAH-style helps you decide how much context to dump on the LLM. If your LLM degrades past 32K, don't push 100K of context — retrieve smarter.

## Eval / Tuning Workflow

1. **Diagnose the layer**: poor faithfulness with high context recall = generation problem; poor answer relevancy with high faithfulness = pipeline missing the point; poor context recall = retriever problem.
2. **Per-stage ablations**:
   - Replace retriever with a oracle (always returns the labeled gold context). Score generation alone.
   - Replace generator with the gold answer. Score retrieval alone.
3. **Categorize failures**: cluster failed examples by topic / query type. Often you find one cluster (e.g., multi-hop) drags everything down.
4. **Cost budget per metric**: faithfulness and answer relevancy use LLM-as-judge — measure judge agreement with humans on a small sample first.

## Common Pitfalls

- **LLM-judge using the same model that generates**: collusion bias. Use a different model family for judging when possible.
- **Evaluating on synthetic queries only**: RAGAS can synthesize a test set, but real user queries always look weirder. Mix.
- **Single overall score**: averages hide the failure modes you need to fix. Always look per-category and per-metric.
- **No latency / cost tracking** alongside quality: a 2% accuracy gain that 4× cost may not ship.
- **Eval set leakage**: chunks from the eval set appear in your training/few-shot prompts → inflated scores. Hold out strictly.
- **Scoring on a stale snapshot**: corpus drifts; gold answers go out of date. Re-validate quarterly.
- **NIAH-only confidence**: passing NIAH does NOT mean a long-context LLM handles your real queries. RULER reveals this.

## When to Use This Mode

Always. Without eval, every change is hope. Reach for this mode when:

- You're shipping a RAG system to users for the first time.
- You're doing pipeline ablations (which embedder, which reranker, what chunk size).
- Faithfulness or accuracy regressions get reported.
- You need to justify costs / model choice.
- Long-context strategies are on the table — RULER over NIAH.

## Sources

- Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation" — https://arxiv.org/abs/2309.15217
- RAGAS docs — https://docs.ragas.io/en/stable/
- RAGAS metric: Faithfulness — https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/
- RAGAS metric: Context Precision — https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
- DeepEval — https://github.com/confident-ai/deepeval
- TruLens — https://www.trulens.org/
- Phoenix (Arize) — https://github.com/Arize-ai/phoenix
- Greg Kamradt, "Needle In A Haystack" — https://github.com/gkamradt/LLMTest_NeedleInAHaystack
- Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?" — https://arxiv.org/abs/2404.06654
- BEIR — https://github.com/beir-cellar/beir
- MTEB leaderboard — https://huggingface.co/spaces/mteb/leaderboard