Ssmr Bench Eval

---
name: ssmr-bench-eval
description: Evaluates musical reasoning capabilities across rhythm, chords, intervals, and scales using sheet music problems. It tests both textual and visual (staff notation) modalities to measure how well models recognize musical elements and perform logical deductions. Use when the user wants to benchmark on Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.04059
  bibtex_key: wang2025towards
  confidence: high
---

# ssmr-bench-eval

> Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning — Wang et al. (2025) (arXiv:2509.04059, 2025)

## What this evaluates

Evaluates musical reasoning capabilities across rhythm, chords, intervals, and scales using sheet music problems. It tests both textual and visual (staff notation) modalities to measure how well models recognize musical elements and perform logical deductions.

## Datasets

- **Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench)** — total 3200; splits: test (3200)

## Metrics

- `accuracy` **(primary)** — range: [0, 100] percent
  - Calculated as the percentage of correctly answered multiple-choice questions out of the total number of questions. Overall accuracy is the average across all sub-tasks (Rhythm, Chord, Interval, Scale).

## Input / output format

**Input**: Multiple-choice questions about sheet music properties, provided either as textual descriptions or as images of musical staff notation. Model options are shuffled before inference.

**Output**: Free-text model response containing the selected option or reasoning, which is subsequently parsed to extract the final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().upper() == g.strip().upper())
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Options are shuffled to prevent models from exploiting positional bias or contamination.
- Answers must be parsed using the Math-Verify tool rather than simple string matching.
- Visual modality performance is heavily bottlenecked by the model's ability to first recognize musical symbols before reasoning.

## Evidence (verbatim from paper)

> Specifically, GPT-5 attains the highest overall accuracy at 97.00%, followed by Gemini-2.5-Pro at 96.10%, DeepSeek-R1 at 93.63%, and GPT-4.1 at 79.81%.

## Citation

```bibtex
@misc{wang2025towards,
  title={Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.04059}
}
```

- arXiv: 2509.04059