Crag Eval

---
name: crag-eval
description: Evaluates the factual reliability and hallucination resistance of Retrieval-Augmented Generation (RAG) systems on realistic, dynamic, and long-tail questions. It measures how well models avoid generating incorrect information and appropriately abstain when knowledge is missing. Use when the user wants to benchmark on CRAG, or asks about evaluating this task. Reports truthfulness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.04744
  bibtex_key: yang2024crag
  confidence: high
---

# crag-eval

> CRAG -- Comprehensive RAG Benchmark — Xiao Yang et al. (2024) (arXiv:2406.04744, 2024)

## What this evaluates

Evaluates the factual reliability and hallucination resistance of Retrieval-Augmented Generation (RAG) systems on realistic, dynamic, and long-tail questions. It measures how well models avoid generating incorrect information and appropriately abstain when knowledge is missing.

## Datasets

- **CRAG** — total 4409; splits: validation (-1), public_test (-1), private (-1); repo https://github.com/facebookresearch/CRAG

## Metrics

- `truthfulness` **(primary)** — range: [-1, 1]
  - Average score across all evaluation examples. Each response is labeled Perfect (1), Acceptable (0.5), Missing (0), or Incorrect (-1). Truthfulness = mean(scores).

## Input / output format

**Input**: A factual question from the evaluation set.

**Output**: A natural language response generated by the RAG system.

## Scoring recipe

```python
def score_response(response):
    if is_perfect(response): return 1.0
    elif is_acceptable(response): return 0.5
    elif is_missing(response): return 0.0
    else: return -1.0

def compute_truthfulness(predictions, golds):
    scores = [score_response(p) for p in predictions]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Hallucinated answers are penalized (-1) while missing answers get 0, so systems should prefer abstaining over guessing.
- Automatic evaluation uses a two-step process: exact match first, then LLM judge; results are averaged across two different LLMs (ChatGPT and Llama 3) to mitigate self-preference bias.

## Evidence (verbatim from paper)

> We use a scoring method with score 1, 0.5, 0, and -1 for each perfect, acceptable, missing, and incorrect answer, respectively, where we penalize hallucinated answers and prefer missing answers to incorrect ones. We then define truthfulness as the average score from all examples in the evaluation set for a given RAG system.

## Citation

```bibtex
@misc{yang2024crag,
  title={CRAG -- Comprehensive RAG Benchmark},
  author={Xiao Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.04744}
}
```

- arXiv: 2406.04744