Crumqs Eval

---
name: crumqs-eval
description: Evaluates RAG systems on synthetic unanswerable and multi-hop queries to measure their refusal behavior, hallucination rates, and susceptibility to reasoning shortcuts when contexts are disjointed or out-of-distribution. Use when the user wants to benchmark on CRUMQs, UAEval4RAG, MultiHop-RAG, or asks about evaluating this task. Reports cheatability ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.11956
  bibtex_key: liu2025crumqs
  confidence: high
---

# crumqs-eval

> Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries — Liu et al. (2025) (arXiv:2510.11956, 2025)

## What this evaluates

Evaluates RAG systems on synthetic unanswerable and multi-hop queries to measure their refusal behavior, hallucination rates, and susceptibility to reasoning shortcuts when contexts are disjointed or out-of-distribution.

## Datasets

- **CRUMQs** — total 3048; splits: test (3048)
- **UAEval4RAG** — total 7559; splits: test (7559)
- **MultiHop-RAG** — total 2556; splits: test (2556)

## Metrics

- `acceptable ratio` — range: [0, 1]
  - Proportion of model responses deemed acceptable (i.e., correctly refusing or stating unanswerability) for unanswerable queries.
- `unanswered ratio` — range: [0, 1]
  - Proportion of model responses that fail to provide an answer or clarification for unanswerable queries.
- `ask-for-clarification ratio` — range: [0, 1]
  - Proportion of model responses that request additional information for unanswerable queries.
- `accuracy` — range: [0, 1]
  - Semantic equivalence between target and predicted answers, judged via LLM (Gemini-2.0-Flash).
- `F1 score` — range: [0, 1]
  - Average F1 for answer prediction and paragraph-level support identification across models and datasets.
- `cheatability ratio` **(primary)** — range: [0, 1]
  - Ratio of F1 scores in the DiRe probe (disjointed contexts) vs. non-probe (oracle) settings, representing the percentage of performance attributable to disconnected reasoning.

## Input / output format

**Input**: Query alone (for unanswerability evaluation) or query paired with retrieved/disjointed document chunks (for cheatability evaluation).

**Output**: Generated answer text, optionally accompanied by paragraph-level support identification.

## Scoring recipe

```python
# Unanswerability metrics
acceptable = sum(1 for pred in preds if is_acceptable_refusal(pred)) / len(preds)
unanswered = sum(1 for pred in preds if is_unanswered(pred)) / len(preds)
clarification = sum(1 for pred in preds if asks_for_clarification(pred)) / len(preds)
accuracy = llm_judge_semantic_equivalence(gold, preds)

# Cheatability metric
f1_non_probe = compute_f1(gold_answers, gold_supports, oracle_preds)
f1_probe = compute_f1(gold_answers, gold_supports, disjointed_preds)
cheatability_ratio = f1_probe / f1_non_probe
```

## Common pitfalls

- Unanswerability evaluation explicitly assumes no access to external documents, so models must rely on internal knowledge or explicit refusal.
- Cheatability relies on the DiRe probe transformation; a high ratio indicates shortcut reliance rather than robust multi-hop reasoning.
- Accuracy for unanswerable queries uses LLM-as-a-judge for semantic equivalence, which may introduce bias compared to exact-match metrics.

## Evidence (verbatim from paper)

> For unanswerability evaluation, we adopt the metrics of acceptable ratio, unanswered ratio, and ask-for-clarification ratio from [[29]]. We additionally score accuracy by running LLM judgments of semantic equivalence between target and predicted answers (Gemini-2.0-Flash prompted as in [[25]]). For cheatability evaluation, we compute the average F1 score for each modeldatasettask setting as in [[39], [38]]. The cheatability of each dataset is then measured as the ratio of F1 scores in the probe vs. non-probe settings, which represents the percentage of model performance attributable to disconnected reasoning.

## Citation

```bibtex
@misc{liu2025crumqs,
  title={Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.11956}
}
```

- arXiv: 2510.11956