Mtrag Un Eval

---
name: mtrag-un-eval
description: Evaluates multi-turn RAG systems on handling unanswerable, underspecified, and non-standalone questions. It probes both retrieval ranking quality and generation quality, including the model's ability to correctly refuse or request clarification when context is insufficient. Use when the user wants to benchmark on MTRAG-UN, or asks about evaluating this task. Reports RB_llm.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.23184
  bibtex_key: rosenthal2026mtragun
  confidence: high
---

# mtrag-un-eval

> MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations — Rosenthal et al. (2026) (arXiv:2602.23184, 2026)

## What this evaluates

Evaluates multi-turn RAG systems on handling unanswerable, underspecified, and non-standalone questions. It probes both retrieval ranking quality and generation quality, including the model's ability to correctly refuse or request clarification when context is insufficient.

## Datasets

- **MTRAG-UN** — total 666; splits: test (-1); repo https://github.com/IBM/mt-rag-benchmark

## Metrics

- `RB_llm` **(primary)** — range: [0, 1]
  - LLM-as-a-judge metric where a reference LLM (GPT-OSS-120B) scores the generated response against the ground truth answer and provided context. Scores are conditioned on answerability.
- `nDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 5, measuring the quality of the ranked list of retrieved documents relative to an ideal ranking.

## Input / output format

**Input**: Retrieval: conversational history + current turn query. Generation: retrieved passages (top 5 or up to 10) + user question + system prompt instructing <150 word response, IDK fallback, and clarification handling.

**Output**: Retrieval: ranked list of document IDs/text. Generation: natural language response (<150 words), optionally containing "I do not have specific information" or a clarification request.

## Scoring recipe

```python
def score_rb_llm(predictions, gold_answers, contexts, judge_model):
    scores = []
    for pred, gold, ctx in zip(predictions, gold_answers, contexts):
        # Judge model evaluates pred against gold and ctx
        score = judge_model.evaluate(response=pred, reference=gold, context=ctx)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Models frequently fail to recognize unanswerable questions, instead generating plausible but hallucinated answers.
- Query rewrite significantly impacts non-standalone question performance, but retrieval drops sharply in Banking/Telco domains due to dense hyperlinks and cross-page dependencies.
- LLM judges (GPT-OSS-120B) are used as proxies for human judgment, but their correlation and potential biases should be considered when interpreting generation scores.

## Evidence (verbatim from paper)

> We adopt the evaluation metrics of Katsis et al. (2025): (1) reference-based $	extrm{RB}_{	extrm{llm}}$ and $	extrm{RB}_{	extrm{alg}}$, (2) the IDK ("I Don’t Know") judge, and (3) faithfulness judge from RAGAS RL${}_{	extrm{F}}$. All evaluation metrics are conditioned to account for answerability.

## Citation

```bibtex
@misc{rosenthal2026mtragun,
  title={MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations},
  author={Rosenthal et al. (2026)},
  year={2026},
  note={arXiv:2602.23184}
}
```

- arXiv: 2602.23184