Music Audio Representation Eval

---
name: music-audio-representation-eval
description: Evaluates the quality of pre-trained audio embeddings for downstream music understanding tasks including tagging, genre classification, mood prediction, pitch/instrument detection, key classification, and emotion recognition. It tests whether frozen embeddings can be effectively probed with simple MLP classifiers to achieve competitive performance without fine-tuning the backbone model. Use when the user wants to benchmark on MSDS, MSD50, MSD100, MSD500, AMM, MuMu, MTT, NSynthP, NSynthI, GTZAN, Emo, GSKey, Jam-50, Jam-All, Jam-MT, or asks about evaluating this task. Reports weighted accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.03799
  bibtex_key: [REDACTED]
  confidence: high
---

# music-audio-representation-eval

> Supervised and Unsupervised Learning of Audio Representations for Music Understanding — McCallum et al. (2022) (arXiv:2210.03799, 2022)

## What this evaluates

Evaluates the quality of pre-trained audio embeddings for downstream music understanding tasks including tagging, genre classification, mood prediction, pitch/instrument detection, key classification, and emotion recognition. It tests whether frozen embeddings can be effectively probed with simple MLP classifiers to achieve competitive performance without fine-tuning the backbone model.

## Datasets

- **MSDS** — total 242000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
- **MSD50** — total 36000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
- **MSD100** — total 115000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
- **MSD500** — total 156000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
- **AMM** — total 67000; splits: artist-based (-1); repo https://github.com/fdlm/listening-moods
- **MuMu** — total 147000; splits: official (-1); repo https://zenodo.org/record/1236906#.YoPIAhNBx0s
- **MTT** — total 26000; splits: published (-1); repo https://github.com/jongpillee/music_dataset_split
- **NSynthP** — total 306000; splits: published (-1)
- **NSynthI** — total 306000; splits: published (-1)
- **GTZAN** — total 930; splits: fault-filtered (-1)
- **Emo** — total 744; splits: artist-based (-1)
- **GSKey** — total 2100; splits: train/val/test (-1); repo https://github.com/GiantSteps/giantsteps-mtg-key-dataset
- **Jam-50** — total 54000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset
- **Jam-All** — total 56000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset
- **Jam-MT** — total 18000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset

## Metrics

- `weighted accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy weighted by class frequency in the test set, commonly used in music key detection benchmarks.
- `coefficient of determination (R²)` — range: other
  - R² = 1 - (SS_res / SS_tot), where SS_res is the sum of squared residuals between predictions and ground truth, and SS_tot is the total sum of squares. Used for continuous emotion valence and arousal regression.

## Input / output format

**Input**: Timeline-averaged global average pooled activations from the final convolutional layer of the SF-NFNet-F0 architecture, sampled at 0.5 Hz from 3-second audio snippets. These fixed embeddings are fed into a simple multi-layer perceptron (MLP) classifier.

**Output**: Discrete class labels for classification tasks (tagging, genre, mood, pitch, instrument, key), or continuous scalar values for regression tasks (emotion valence and arousal).

## Scoring recipe

```python
def compute_metric(task, preds, golds):
    if task == 'key_classification':
        weights = get_class_frequencies(golds)
        correct = sum(w * (p == g) for w, p, g in zip(weights, preds, golds))
        return correct / sum(weights)
    elif task == 'emotion_regression':
        ss_res = sum((p - g)**2 for p, g in zip(preds, golds))
        ss_tot = sum((g - mean(golds))**2 for g in golds)
        return 1 - (ss_res / ss_tot)
    else:
        return sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)
```

## Common pitfalls

- Failing to use artist-based splits for datasets like AMM and Emo, which causes severe data leakage and inflated performance due to artist correlation in the labels.
- Over-optimizing probe hyperparameters without constraining them to prior SOTA ranges, which confounds embedding quality evaluation with probe tuning capabilities.
- Ignoring dataset-specific windowing strategies (e.g., 1-second vs 4-second feature windows for NSynth), which drastically changes embedding aggregation and downstream probe performance.

## Evidence (verbatim from paper)

> In all cases, we train probes using Adam optimization with a cosine learning rate schedule with 1,000 steps of warmup followed by a decay to zero over the remainder of the steps. We optimize the learning rate, number of training steps, and l2-regularization of each probe to achieve best performance / prevent overfitting. For evaluation we compute a weighted accuracy score common in key classification... This poses a regression problem—we use the coefficient of determination as the evaluation metric for both valence (EmoV) and arousal (EmoA).

## Citation

```bibtex
@misc{mccallum2022music,
  title={Supervised and Unsupervised Learning of Audio Representations for Music Understanding},
  author={McCallum et al. (2022)},
  year={2022},
  note={arXiv:2210.03799}
}
```

- arXiv: 2210.03799