Mucue Eval

---
name: mucue-eval
description: Evaluates a model's ability to understand music across a spectrum of tasks, ranging from low-level acoustic perception (e.g., pitch, chord, rhythm) to high-level cognitive reasoning (e.g., genre, mood, structure, lyrical comprehension). It probes whether foundation models can process long-context audio and lyrics jointly to answer standardized multiple-choice questions. Use when the user wants to benchmark on MuCUE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01178
  bibtex_key: jiang2025advancing
  confidence: high
---

# mucue-eval

> Advancing the Foundation Model for Music Understanding — Jiang et al. (2025) (arXiv:2508.01178, 2025)

## What this evaluates

Evaluates a model's ability to understand music across a spectrum of tasks, ranging from low-level acoustic perception (e.g., pitch, chord, rhythm) to high-level cognitive reasoning (e.g., genre, mood, structure, lyrical comprehension). It probes whether foundation models can process long-context audio and lyrics jointly to answer standardized multiple-choice questions.

## Datasets

- **MuCUE** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy calculated as the number of correctly answered multiple-choice questions divided by the total number of questions. Final scores are averaged across all 26 sub-tasks in the benchmark.

## Input / output format

**Input**: A music audio clip (up to 390 seconds) optionally paired with lyrical text, followed by a multiple-choice question related to the audio's properties (e.g., key, tempo, genre, mood, structure, or lyrics).

**Output**: A single letter or option identifier corresponding to the correct answer from the provided multiple-choice options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    total = len(predictions)
    for pred, gold in zip(predictions, gold_labels):
        if pred.strip().upper() == gold.strip().upper():
            correct += 1
    return correct / total if total > 0 else 0.0
```

## Common pitfalls

- The benchmark aggregates scores across 26 highly heterogeneous tasks (from acoustic feature detection to abstract summarization), which can mask severe performance drops on specific sub-tasks.
- Multiple-choice formatting is not universally supported by all audio-LLMs; excluding models that fail to parse MCQs may introduce selection bias in reported comparisons.
- Contamination risk is acknowledged but not quantified, as the paper only states that held-out data was used for some large datasets without providing exact split ratios or contamination audits.

## Evidence (verbatim from paper)

> Achieving an average score of 65.7, our model establishes a new state-of-the-art, outperforming the next-best model, Qwen2.5-Omni, by a significant margin of over 15 points in average accuracy. This substantial improvement across a diverse set of 26 tasks underscores the efficacy of our unified architecture and targeted training strategy.

## Citation

```bibtex
@misc{jiang2025advancing,
  title={Advancing the Foundation Model for Music Understanding},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2508.01178}
}
```

- arXiv: 2508.01178