Rowen Hallucination Eval

---
name: rowen-hallucination-eval
description: This evaluation probes an LLM's ability to generate factually correct responses and mitigate hallucinations under adaptive retrieval-augmented generation. It measures factual accuracy via LLM-based scoring on open-ended questions and exact-match accuracy on multi-step reasoning yes/no questions. Use when the user wants to benchmark on TruthfulQA, StrategyQA, or asks about evaluating this task. Reports FactScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.10612
  bibtex_key: ding2024rowen
  confidence: high
---

# rowen-hallucination-eval

> Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMs — Hanxing Ding et al. (arXiv:2402.10612, 2024)

## What this evaluates

This evaluation probes an LLM's ability to generate factually correct responses and mitigate hallucinations under adaptive retrieval-augmented generation. It measures factual accuracy via LLM-based scoring on open-ended questions and exact-match accuracy on multi-step reasoning yes/no questions.

## Datasets

- **TruthfulQA** — total 817; splits: test (-1)
- **StrategyQA** — total 500; splits: test (-1)

## Metrics

- `FactScore` **(primary)** — range: [0, 100]
  - Compares generated responses to reference answers using prompt-based instructions to assess factual correctness. Higher scores indicate better factual alignment.
- `BLEU` — range: [0, 100]
  - Measures n-gram precision overlap between generated and reference text, penalizing for verbosity.
- `Rouge-L` — range: [0, 100]
  - Measures the longest common subsequence recall/precision between generated and reference text.
- `Accuracy` — range: [0, 1]
  - Exact-match accuracy of generated yes/no answers compared to gold-standard answers.

## Input / output format

**Input**: Question prompts from TruthfulQA and StrategyQA datasets.

**Output**: Generated textual responses (including reasoning process and final answer for StrategyQA).

## Scoring recipe

```python
# TruthfulQA
factscore = llm_as_judge(predictions, references, prompt=APPENDIX_C_TABLE9)
bleu = compute_bleu(predictions, references)
rouge_l = compute_rouge_l(predictions, references)
# StrategyQA
accuracy = sum(1 for p, g in zip(predictions, references) if p.strip().lower() == g.strip().lower()) / len(predictions)
```

## Common pitfalls

- FactScore relies on a specific LLM-as-a-judge prompt template (Appendix Table 9), so results may vary if different prompts or models are used.
- StrategyQA evaluation uses a randomly sampled subset of 500 examples rather than the full benchmark, which limits direct comparison with full-dataset results.
- BLEU and Rouge-L measure lexical overlap only and do not capture factual correctness or reasoning quality.

## Evidence (verbatim from paper)

> We report the FactScore*Min et al. ([2023](#bib.bib25 ""))*, which is widely used in previous work*Dhuliawala et al. ([2023](#bib.bib5 "")); Kang et al. ([2023](#bib.bib16 ""))*, by comparing generated responses to reference answers using prompt-based instructions (Appendix[C](#A3 "Appendix C Prompt Instruction Examples ‣ Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models") Table[9](#A3.T9 "Table 9 ‣ Appendix C Prompt Instruction Examples ‣ Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models")). Notably, our experiments emphasize evaluating the performance on the FactScore.

## Citation

```bibtex
@misc{ding2024rowen,
  title={Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMs},
  author={Hanxing Ding et al.},
  year={2024},
  note={arXiv:2402.10612}
}
```

- arXiv: 2402.10612