Music Autotagging Eval

---
name: music-autotagging-eval
description: Evaluates audio representation models on music autotagging tasks, measuring how well they predict categorical genre/instrument/mood tags and continuous musical features from audio input. It compares performance across generic tag datasets and expert-annotated continuous features to highlight limitations in current evaluation practices. Use when the user wants to benchmark on MagnaTagATune, MTG-Jamendo, MGPHot-tag, MGPHot-reg, or asks about evaluating this task. Reports MAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.06936
  bibtex_key: ramoneda2025mgphot
  confidence: high
---

# music-autotagging-eval

> Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets — Ramoneda et al. (2025) (arXiv:2509.06936, 2025)

## What this evaluates

Evaluates audio representation models on music autotagging tasks, measuring how well they predict categorical genre/instrument/mood tags and continuous musical features from audio input. It compares performance across generic tag datasets and expert-annotated continuous features to highlight limitations in current evaluation practices.

## Datasets

- **MagnaTagATune** — total ?; splits: test (-1)
- **MTG-Jamendo** — total ?; splits: test (-1)
- **MGPHot-tag** — total ?; splits: test (-1)
- **MGPHot-reg** — total ?; splits: test (-1)

## Metrics

- `MAP` **(primary)** — range: [0, 1]
  - Mean Average Precision: computes the average precision across all tags/classes by averaging precision at each recall level for each class, then averaging across classes.
- `RMSE` — range: other
  - Root Mean-Squared Error: measures the square root of the mean squared difference between predicted and ground-truth continuous values.

## Input / output format

**Input**: Raw audio waveform or precomputed audio embeddings.

**Output**: Categorical tag probabilities/labels for classification tasks; continuous numerical values for regression tasks.

## Scoring recipe

```python
def compute_map(preds, labels):
    ap = []
    for c in range(preds.shape[1]):
        scores = preds[:, c]
        truth = labels[:, c]
        sorted_idx = np.argsort(-scores)
        tp = np.cumsum(truth[sorted_idx])
        fp = np.cumsum(1 - truth[sorted_idx])
        prec = tp / (tp + fp + 1e-8)
        ap.append(np.mean(prec[truth[sorted_idx] == 1]))
    return np.mean(ap)

def compute_rmse(preds, labels):
    return np.sqrt(np.mean((preds - labels) ** 2))

# Average over 5 random seeds
final_map = np.mean([compute_map(p, l) for p, l in runs])
final_rmse = np.mean([compute_rmse(p, l) for p, l in runs])
```

## Common pitfalls

- Using official MTG-Jamendo splits instead of the paper's custom split that includes all available tags per category.
- Assuming category names (e.g., 'Genre', 'Instrument') are directly comparable across datasets, as underlying tag definitions differ substantially.
- Reporting MAE/MSE instead of the primary RMSE metric without noting the paper's explicit choice of RMSE for interpretability.

## Evidence (verbatim from paper)

> Table[3] reports the mean average precision (MAP ↑) for the three tagging tasks and root mean-squared error (RMSE ↓) for the regression task. Each score is the mean of five runs initialized with different seeds.

## Citation

```bibtex
@misc{ramoneda2025mgphot,
  title={Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets},
  author={Ramoneda et al. (2025)},
  year={2025},
  note={arXiv:2509.06936}
}
```

- arXiv: 2509.06936