Music Tagging Eval

---
name: music-tagging-eval
description: Evaluates a model's ability to predict multiple audio tags (e.g., genre, mood, instruments) from short audio segments. It probes long-range temporal dependency modeling and robustness to class imbalance in user-generated music metadata. Use when the user wants to benchmark on MagnaTagATune (MTAT), Million Song Dataset (MSD), or asks about evaluating this task. Reports AUPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.04972
  bibtex_key: won2019toward
  confidence: high
---

# music-tagging-eval

> Toward Interpretable Music Tagging with Self-Attention — Won et al. (2019) (arXiv:1906.04972, 2019)

## What this evaluates

Evaluates a model's ability to predict multiple audio tags (e.g., genre, mood, instruments) from short audio segments. It probes long-range temporal dependency modeling and robustness to class imbalance in user-generated music metadata.

## Datasets

- **MagnaTagATune (MTAT)** — total ?; splits: test (-1)
- **Million Song Dataset (MSD)** — total ?; splits: test (-1)

## Metrics

- `AUPR` **(primary)** — range: [0, 1]
  - Area Under the Precision-Recall curve. Computed by integrating precision over recall thresholds across all predicted probabilities for each tag. Preferred for highly skewed, multi-label datasets.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.

## Input / output format

**Input**: Log mel-spectrograms of audio segments (typically ~4.1s or ~16.4s long).

**Output**: Multi-label binary predictions (probabilities) for a fixed vocabulary of music tags (e.g., genre, mood, instrument).

## Scoring recipe

```python
def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

# Macro-average across all tags
aupr = np.mean([compute_aupr(y_true[:, i], y_pred[:, i]) for i in range(num_tags)])
auroc = np.mean([compute_auroc(y_true[:, i], y_pred[:, i]) for i in range(num_tags)])
```

## Common pitfalls

- AUROC can be misleading on these datasets due to high popularity bias and class skewness; AUPR is explicitly recommended instead.
- Attention heatmaps indicate where the model focuses but do not explain classification reasoning, potentially highlighting irrelevant loud segments for quiet tags.
- Model performance drops when input sequence length increases from ~4.1s to ~16.4s without adjusting architecture depth or receptive field.

## Evidence (verbatim from paper)

> Following previous research [[28]], we report the Area Under Precision Recall curve (AUPR) along with conventional Area Under Receiver Operating Characteristic curve (AUROC). AUPR is known to be more informative to evaluate the algorithm’s performance when it deals with highly skewed datasets [[7]]. Since we are using user-generated tags (MTAT and MSD), there is popularity biased skewness in their distributions.

## Citation

```bibtex
@misc{won2019toward,
  title={Toward Interpretable Music Tagging with Self-Attention},
  author={Won et al. (2019)},
  year={2019},
  note={arXiv:1906.04972}
}
```

- arXiv: 1906.04972