Rag Reasoning Eval

---
name: rag-reasoning-eval
description: Evaluates retrieval-augmented reasoning systems on their ability to iteratively refine answers using a critique language model. It probes robustness to noisy retrieval, out-of-distribution generalization, and the effectiveness of contrastive critique synthesis over standard self-refinement baselines. Use when the user wants to benchmark on PopQA, TriviaQA, NaturalQuestions, 2WikiMultihopQA, ASQA, HotpotQA, SQuAD, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.14858
  bibtex_key: [REDACTED]
  confidence: high
---

# rag-reasoning-eval

> Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization — Wei et al. (2025) (arXiv:2504.14858, 2025)

## What this evaluates

Evaluates retrieval-augmented reasoning systems on their ability to iteratively refine answers using a critique language model. It probes robustness to noisy retrieval, out-of-distribution generalization, and the effectiveness of contrastive critique synthesis over standard self-refinement baselines.

## Datasets

- **PopQA** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **NaturalQuestions** — total ?; splits: test (-1)
- **2WikiMultihopQA** — total ?; splits: test (-1)
- **ASQA** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **SQuAD** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Binary score (1 if the final generated answer exactly matches the ground-truth answer, 0 otherwise), averaged over all instances in the dataset.
- `str-em` — range: [0, 1]
  - Official string exact-match metric used for ASQA. Scores 1 if the prediction matches the reference string exactly, 0 otherwise.

## Input / output format

**Input**: Question text and top-5 retrieved passages.

**Output**: Final generated answer string.

## Scoring recipe

```python
def compute_metrics(predictions, golds, dataset_name):
    scores = []
    for pred, gold in zip(predictions, golds):
        if dataset_name == 'ASQA':
            scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
        else:
            scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
    return sum(scores) / len(scores) if scores else 0.0
```

## Common pitfalls

- ASQA uses string exact match (str-em) instead of standard accuracy, which can penalize valid paraphrases or multi-sentence answers.
- The 'fixed' variant uses exactly 1 refinement iteration, while 'auto' dynamically terminates, making direct comparison of iteration counts misleading.
- Retrieval is strictly limited to Top-5 passages for all experiments, which may not reflect full RAG pipeline performance or optimal retrieval settings.

## Evidence (verbatim from paper)

> Following previous work[[51]], we adopt the official correctness metric (str-em) for ASQA[[45]], and use accuracy for the other tasks, which measures whether the final generations of the model align with the ground-truth[[41], [52]].

## Citation

```bibtex
@misc{wei2025ragreasonalignment,
  title={Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2504.14858}
}
```

- arXiv: 2504.14858