Muchomusic Eval

---
name: muchomusic-eval
description: Evaluates multimodal audio-language models' ability to understand music through factual knowledge and reasoning tasks. It probes whether models can ground their answers in audio content rather than relying on language priors or hallucinating musical elements. Use when the user wants to benchmark on MuChoMusic, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.01337
  bibtex_key: weck2024muchomusic
  confidence: high
---

# muchomusic-eval

> MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models — Weck et al. (2024) (arXiv:2408.01337, 2024)

## What this evaluates

Evaluates multimodal audio-language models' ability to understand music through factual knowledge and reasoning tasks. It probes whether models can ground their answers in audio content rather than relying on language priors or hallucinating musical elements.

## Datasets

- **MuChoMusic** — total 1187; splits: test (1187)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions where the model correctly selects the ground-truth answer option from the provided multiple-choice set.
- `instruction following rate (IFR)` — range: percent
  - Percentage of questions where the model's output matches the expected answer format, regardless of correctness.

## Input / output format

**Input**: A music audio clip paired with a text question and a set of multiple-choice answer options (typically 4, but can be reduced to 2 or 3 for ablation).

**Output**: A single selected answer option from the provided choices. Models must output the answer in a specific format to be counted as instruction-following.

## Scoring recipe

```python
def compute_metrics(predictions, golds, format_checker):
    total = len(golds)
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = (correct / total) * 100
    follows_format = sum(1 for p in predictions if format_checker(p))
    ifr = (follows_format / total) * 100
    return {'accuracy': accuracy, 'if': ifr}
```

## Common pitfalls

- Models often fail to output answers in the correct format, which artificially lowers accuracy scores.
- Models exhibit strong language bias, answering based on text priors rather than audio content, leading to poor audio-grounded reasoning.
- Auditory and language hallucinations cause models to ignore provided options and generate uninformative or incorrect responses.

## Evidence (verbatim from paper)

> We report results for all models in Table 3, showing the overall accuracy score alongside detailed scores on knowledge and reasoning questions, and the instruction following rate (IFR).

## Citation

```bibtex
@misc{weck2024muchomusic,
  title={MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models},
  author={Weck et al. (2024)},
  year={2024},
  note={arXiv:2408.01337}
}
```

- arXiv: 2408.01337