Mmu Rag Arena Eval

---
name: mmu-rag-arena-eval
description: Evaluates RAG systems in a live, user-centric arena setting by routing queries to appropriate retrieval pipelines and measuring response quality through direct human feedback. It probes capabilities like retrieval grounding, synthesis coherence, response relevance, and appropriate verbosity in realistic deep-research scenarios. Use when the user wants to benchmark on MMU-RAG Competition / RAG Arena, or asks about evaluating this task. Reports Preference Ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20735
  bibtex_key: ran2026rmitadm
  confidence: high
---

# mmu-rag-arena-eval

> RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition — Ran et al. (2026) (arXiv:2602.20735, 2026)

## What this evaluates

Evaluates RAG systems in a live, user-centric arena setting by routing queries to appropriate retrieval pipelines and measuring response quality through direct human feedback. It probes capabilities like retrieval grounding, synthesis coherence, response relevance, and appropriate verbosity in realistic deep-research scenarios.

## Datasets

- **MMU-RAG Competition / RAG Arena** — total ?; splits: test (-1); repo https://github.com/rmit-ir/NeurIPS-MMU-RAG

## Metrics

- `Preference Ratio` **(primary)** — range: [0, 1]
  - Computed as the ratio of 👍 (thumbs up) to 👎 (thumbs down) responses submitted by participants during the evaluation session.

## Input / output format

**Input**: User-submitted queries in a (semi-)realistic deep-research scenario, interacting with the system via a web interface.

**Output**: System-generated responses, followed by participant binary feedback (👍/👎), open-ended comments, and verbal reflections.

## Scoring recipe

```python
def calculate_preference_ratio(upvotes, downvotes):
    total = upvotes + downvotes
    if total == 0:
        return 1.0
    return upvotes / total
```

## Common pitfalls

- Relying solely on aggregate quantitative metrics may miss nuanced trade-offs in verbosity, grounding, and response structure that only emerge through live user interaction.
- Synthetic LLM-generated queries and LLM-as-a-Judge evaluations can introduce distributional mismatch and evaluation biases, failing to capture realistic user failure modes.

## Evidence (verbatim from paper)

> We analyzed feedback for each query using two complementary approaches: (i) Preference Ratio, computed as the ratio of 👍 to 👎 responses, and (ii) manual inspection of open-ended comments to identify qualitative themes and contextual nuances.

## Citation

```bibtex
@misc{ran2026rmitadm,
  title={RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition},
  author={Ran et al. (2026)},
  year={2026},
  note={arXiv:2602.20735}
}
```

- arXiv: 2602.20735