Syntheory Eval

---
name: syntheory-eval
description: Probes whether music foundation models encode discrete and continuous Western music theory concepts by training linear or MLP classifiers on their internal audio embeddings. Use when the user wants to benchmark on SynTheory, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.00872
  bibtex_key: wei2024do
  confidence: high
---

# syntheory-eval

> Do Music Generation Models Encode Music Theory? — Wei et al. (2024) (arXiv:2410.00872, 2024)

## What this evaluates

Probes whether music foundation models encode discrete and continuous Western music theory concepts by training linear or MLP classifiers on their internal audio embeddings.

## Datasets

- **SynTheory** — total ?; splits: train (-1), test (-1), val (-1); repo https://github.com/brown-palm/syntheory

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of total predictions for classification tasks (notes, intervals, scales, chords, chord progressions, time signatures).
- `R^2` — range: other
  - Coefficient of determination for the continuous tempo regression task, calculated as 1 - (SS_res / SS_tot) where SS is sum of squared errors.

## Input / output format

**Input**: Mean-pooled audio embeddings (dimension varies by model and layer) extracted from 4-second mono audio clips.

**Output**: Predicted class label (for classification) or predicted BPM value (for regression).

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type == "tempo":
        ss_res = sum((y_true - y_pred)**2 for y_true, y_pred in zip(gold, predictions))
        ss_tot = sum((y_true - mean(gold))**2 for y_true in gold)
        return 1 - (ss_res / ss_tot)
    else:
        correct = sum(1 for y_true, y_pred in zip(gold, predictions) if y_true == y_pred)
        return correct / len(gold)
```

## Common pitfalls

- Mean-pooling across time discards temporal dynamics, which may unfairly penalize models on time-varying concepts like chord progressions.
- The tempo split stratifies by BPM range (middle 70% train, extremes test/val) to test generalization, which does not reflect a uniform BPM distribution in real-world music.
- Layer selection is performed independently per concept and model using the validation set, introducing potential selection bias if not properly cross-validated.

## Evidence (verbatim from paper)

> For the classification tasks, we measure the accuracy of our trained probes on the following SynTheory tasks: Notes (12): C, C#, D, D#, E, F, F#, G, G#, A, A#, and B - Intervals (12)... Time Signatures (8)... These tasks are trained on a 70% train, 15% test, and 15% validation split... For the Tempo dataset, we train a regression probe, over the 161 tempo values... We use MSE loss and report the R^2 score.

## Citation

```bibtex
@misc{wei2024do,
  title={Do Music Generation Models Encode Music Theory?},
  author={Wei et al. (2024)},
  year={2024},
  note={arXiv:2410.00872}
}
```

- arXiv: 2410.00872