Dragon Rag Eval

---
name: dragon-rag-eval
description: Evaluates retrieval and end-to-end performance of RAG systems on a dynamic, daily-updating news corpus. It probes a model's ability to accurately retrieve relevant document chunks and generate factually consistent responses to knowledge-graph-derived queries. Use when the user wants to benchmark on Public Texts, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.05713
  bibtex_key: chernogorskii2025dragon
  confidence: medium
---

# dragon-rag-eval

> DRAGON: Dynamic RAG Benchmark On News — Chernogorskii et al. (2025) (arXiv:2507.05713, 2025)

## What this evaluates

Evaluates retrieval and end-to-end performance of RAG systems on a dynamic, daily-updating news corpus. It probes a model's ability to accurately retrieve relevant document chunks and generate factually consistent responses to knowledge-graph-derived queries.

## Datasets

- **Public Texts** — total ?; splits: test (-1); repo https://github.com/RussianNLP/DRAGON

## Metrics

- `Hit Rate` — range: [0, 1]
  - Fraction of queries where the ground-truth relevant document appears in the top-5 retrieved chunks.
- `Recall` — range: [0, 1]
  - Fraction of ground-truth relevant documents successfully retrieved among the top-5 chunks.
- `NDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 5, measuring the ranking quality of retrieved chunks.
- `ROUGE-L` **(primary)** — range: [0, 1]
  - Longest common subsequence F1 score between the generated response and the reference answer.
- `Substring Matching` — range: [0, 1]
  - Exact substring match score between generated and reference text.
- `Judge Score` — range: [0, 1]
  - LLM-as-a-judge evaluation score for response quality and factual alignment (see Appx. H.4).

## Input / output format

**Input**: User query paired with the top-5 retrieved document chunks, formatted into a system prompt template.

**Output**: Generated natural language response to the query.

## Scoring recipe

```python
def score(predictions, golds, retrieved_docs):
    hit_rate = sum(1 for q in queries if golds[q] in retrieved_docs[:5]) / len(queries)
    recall = sum(len(set(golds[q]) & set(retrieved_docs[:5])) / len(golds[q]) for q in queries) / len(queries)
    ndcg = compute_ndcg_at_k(retrieved_docs, golds, k=5)
    rouge_l = compute_rouge_l(predictions, golds)
    sm = compute_substring_match(predictions, golds)
    judge = compute_llm_judge_score(predictions, golds)
    return {'Hit Rate': hit_rate, 'Recall': recall, 'NDCG': ndcg, 'ROUGE-L': rouge_l, 'Substring Matching': sm, 'Judge Score': judge}
```

## Common pitfalls

- Retrieval is strictly limited to the top-5 chunks; using more or fewer will invalidate Hit Rate/Recall/NDCG.
- Prompt context must be truncated if it exceeds the model's max context length, which can drop retrieved chunks and hurt retrieval metrics.
- Chunking must use exactly 500-character length with 100-character overlap; different splitting alters vectorization and retrieval alignment.

## Evidence (verbatim from paper)

> The performance of retrieval is measured with the 3 main metrics: Hit Rate, Recall, and NDCG. End-to-end RAG-system evaluation is performed via ROUGE-L, Substring Matching (SM), and Judge Score. See Appx.[H.4] for their description.

## Citation

```bibtex
@misc{chernogorskii2025dragon,
  title={DRAGON: Dynamic RAG Benchmark On News},
  author={Chernogorskii et al. (2025)},
  year={2025},
  note={arXiv:2507.05713}
}
```

- arXiv: 2507.05713