World Music Corpora Eval

---
name: world-music-corpora-eval
description: Evaluates audio foundation models' cross-cultural generalization across diverse musical traditions (Western, Greek, Turkish, Indian) using multi-label tagging and few-shot learning. Probes whether pre-trained representations capture cultural musical knowledge without extensive adaptation. Use when the user wants to benchmark on Turkish-makam, Hindustani, Carnatic, MagnaTagATune, FMA-medium, Lyra, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.17055
  bibtex_key: papaioannou2025universal
  confidence: high
---

# world-music-corpora-eval

> Universal Music Representations? Evaluating Foundation Models on World Music Corpora — Papaioannou et al. (2025) (arXiv:2506.17055, 2025)

## What this evaluates

Evaluates audio foundation models' cross-cultural generalization across diverse musical traditions (Western, Greek, Turkish, Indian) using multi-label tagging and few-shot learning. Probes whether pre-trained representations capture cultural musical knowledge without extensive adaptation.

## Datasets

- **Turkish-makam** — total ?; splits: train (-1), val (-1), test (-1)
- **Hindustani** — total ?; splits: train (-1), val (-1), test (-1)
- **Carnatic** — total ?; splits: train (-1), val (-1), test (-1)
- **MagnaTagATune** — total ?; splits: train (-1), val (-1), test (-1)
- **FMA-medium** — total ?; splits: train (-1), val (-1), test (-1)
- **Lyra** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the model's ability to distinguish between positive and negative labels across all classification thresholds, averaged across all multi-label tags.
- `mAP` — range: [0, 1]
  - Mean Average Precision. Computes the average precision for each label and averages them across all tags. Suitable for multi-label classification where label distributions are imbalanced.
- `Macro-F1` — range: [0, 1]
  - Harmonic mean of precision and recall calculated per class, then averaged equally across all classes. Gives equal weight to rare and frequent tags.
- `Micro-F1` — range: [0, 1]
  - Harmonic mean of global precision and recall calculated across all instances and classes. Accounts for class imbalance by weighting metrics by support size.

## Input / output format

**Input**: Mono audio clips processed in model-specific windows (10–30 seconds) and resampled to model-specific rates (16–48 kHz), paired with multi-label genre/tradition tags.

**Output**: Per-class prediction scores or probabilities for multi-label classification; for few-shot learning, discrete class predictions based on cosine distance to support set prototypes.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, f1_score

def compute_metrics(y_true, y_scores, task='probing'):
    if task in ['probing', 'sft']:
        auc = roc_auc_score(y_true, y_scores, average='macro')
        map_score = average_precision_score(y_true, y_scores, average='macro')
        return {'ROC-AUC': auc, 'mAP': map_score}
    else: # ML-FSL
        y_pred = (y_scores >= 0.5).astype(int)
        macro_f1 = f1_score(y_true, y_pred, average='macro')
        micro_f1 = f1_score(y_true, y_pred, average='micro')
        return {'Macro-F1': macro_f1, 'Micro-F1': micro_f1}
```

## Common pitfalls

- Data leakage if ML-FSL evaluation samples are not strictly drawn from held-out test sets.
- Model-specific audio preprocessing (window length, sampling rate) must be strictly followed to avoid representation mismatch.
- Standard accuracy is inappropriate; multi-label metrics (ROC-AUC, mAP, F1 variants) must be used due to overlapping genre tags.

## Evidence (verbatim from paper)

> For the Probing and SFT methodologies, we report area under the receiver operating characteristic curve (ROC-AUC) and mean average precision (mAP). These metrics are particularly well-suited for multi-label classification tasks [51] and are consistent with prior work in music tagging [17, 28]. For ML-FSL evaluation, we report macro-F1 (M-F1) and micro-F1 (mF1) scores, which align with the LC-Protonets evaluation framework [29]. F1 score is the harmonic mean of the precision and recall scores. Macro-F1 gives equal weight to all classes, while micro-F1 accounts for class imbalance by calculating metrics globally across all instances.

## Citation

```bibtex
@misc{papaioannou2025universal,
  title={Universal Music Representations? Evaluating Foundation Models on World Music Corpora},
  author={Papaioannou et al. (2025)},
  year={2025},
  note={arXiv:2506.17055}
}
```

- arXiv: 2506.17055