Must Rag Eval

---
name: must-rag-eval
description: This evaluation probes a model's ability to answer music-specific factual and contextual questions using retrieval-augmented generation. It measures accuracy on both in-domain artist metadata and out-of-domain music knowledge across multiple-choice formats. Use when the user wants to benchmark on ArtistMus, TrustMus, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.23334
  bibtex_key: kwon2025must_rag
  confidence: high
---

# must-rag-eval

> MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation — Kwon et al. (2025) (arXiv:2507.23334, 2025)

## What this evaluates

This evaluation probes a model's ability to answer music-specific factual and contextual questions using retrieval-augmented generation. It measures accuracy on both in-domain artist metadata and out-of-domain music knowledge across multiple-choice formats.

## Datasets

- **ArtistMus** — total ?; splits: Seen (-1), Unseen (-1)
- **TrustMus** — total 400; splits: Ppl (100), IT (100), GFT (100), CH (100)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. A response is marked correct if it matches the ground truth answer and adheres to the expected output format.

## Input / output format

**Input**: A multiple-choice question, optionally accompanied by retrieved context passages. For zero-shot baselines, only the question is provided.

**Output**: The model must generate the correct answer choice, strictly adhering to the expected format without extra conversational text.

## Scoring recipe

```python
correct = 0
for q, gold, pred in dataset:
    if pred.strip() == gold.strip() and format_valid(pred):
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- Responses that contain the correct answer but include extra conversational filler or deviate from the strict expected format are automatically scored as incorrect.
- The 'Seen' vs 'Unseen' split is defined by artist presence in the training set, not question difficulty, which can skew baseline comparisons if not accounted for.

## Evidence (verbatim from paper)

> All evaluations use a multiple-choice QA format. Following [[11]], we consider a response incorrect if it deviates from the expected format.

## Citation

```bibtex
@misc{kwon2025must_rag,
  title={MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation},
  author={Kwon et al. (2025)},
  year={2025},
  note={arXiv:2507.23334}
}
```

- arXiv: 2507.23334