Musebench Eval

---
name: musebench-eval
description: Evaluates multimodal language models' ability to perform fine-grained, interactive reasoning over symbolic music scores and expressive performance audio. It probes capabilities in score–audio alignment, performance error detection, and expressive deviation analysis across text, audio, and image modalities. Use when the user wants to benchmark on MuseBench, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.11968
  bibtex_key: zhao2026museagent1
  confidence: high
---

# musebench-eval

> MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio — Qihao Zhao et al. (arXiv:2601.11968, 2026)

## What this evaluates

Evaluates multimodal language models' ability to perform fine-grained, interactive reasoning over symbolic music scores and expressive performance audio. It probes capabilities in score–audio alignment, performance error detection, and expressive deviation analysis across text, audio, and image modalities.

## Datasets

- **MuseBench** — total ?; splits: test (-1)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly answered questions. Computed as (number of correct predictions / total number of instances) × 100.
- `Levenshtein Distance` — range: other
  - Average edit distance (insertions, deletions, substitutions) between predicted and gold ABC notation strings per instance. Lower values indicate better structural conversion accuracy.
- `LSA` — range: percent
  - Latent Semantic Analysis similarity score measuring semantic overlap between predicted and reference text.
- `ROUGE-1` — range: percent
  - Unigram recall/precision/f1 overlap between predicted and reference text.
- `ROUGE-L` — range: percent
  - Longest common subsequence overlap between predicted and reference text.
- `METEOR` — range: percent
  - Metric for evaluating the alignment between hypothesis and reference translations, considering synonymy and stemming.
- `Avg` — range: percent
  - Arithmetic mean of LSA, ROUGE-1, ROUGE-L, and METEOR scores.

## Input / output format

**Input**: Multimodal inputs comprising text-based questions, performance audio recordings, and images of music scores.

**Output**: Model-generated text answers, transcribed ABC notation, or descriptive analysis text.

## Scoring recipe

```python
def compute_accuracy(preds, golds):
    return sum(p == g for p, g in zip(preds, golds)) / len(golds) * 100

def compute_levenshtein(preds, golds):
    return sum(levenshtein(p, g) for p, g in zip(preds, golds)) / len(golds)

def compute_semantic(preds, golds):
    lsa = sum(lsa_sim(p, g) for p, g in zip(preds, golds)) / len(golds)
    r1 = sum(rouge1(p, g) for p, g in zip(preds, golds)) / len(golds)
    rl = sum(rougel(p, g) for p, g in zip(preds, golds)) / len(golds)
    met = sum(meteor(p, g) for p, g in zip(preds, golds)) / len(golds)
    return lsa, r1, rl, met, (lsa + r1 + rl + met) / 4
```

## Common pitfalls

- Random baseline accuracy varies by modality (25% for text/image, 50% for audio), making cross-modality accuracy comparisons misleading without normalization.
- Levenshtein Distance reports a dataset-level average where lower is better, contradicting the higher-is-better convention of all other reported metrics.
- Open-set semantic metrics (LSA, ROUGE, METEOR) are averaged without specifying the exact tokenizer, library, or alignment strategy used.

## Evidence (verbatim from paper)

> Following the evaluation protocol of NotaGPT, we consider two settings: (i) closed-set conversion of sheet music into ABC notation, evaluated using Levenshtein Distance, and (ii) open-set visual music analysis, assessed with semantic metrics including LSA, ROUGE, and METEOR.

## Citation

```bibtex
@misc{zhao2026museagent1,
  title={MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio},
  author={Qihao Zhao et al.},
  year={2026},
  note={arXiv:2601.11968}
}
```

- arXiv: 2601.11968