Mir Ref Eval

---
name: mir-ref-eval
description: This framework evaluates the quality, robustness, and downstream extractability of learned music audio representations. It probes how well representations encode task-relevant information (e.g., instruments, pitch, singer identity) and measures their resilience to real-world audio degradations like noise, gain changes, and compression. Use when the user wants to benchmark on TinySOL, Beatport EDM, VocalSet, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.05994
  bibtex_key: plachouras2023mir_ref
  confidence: medium
---

# mir-ref-eval

> mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks — Plachouras et al. (2023) (arXiv:2312.05994, 2023)

## What this evaluates

This framework evaluates the quality, robustness, and downstream extractability of learned music audio representations. It probes how well representations encode task-relevant information (e.g., instruments, pitch, singer identity) and measures their resilience to real-world audio degradations like noise, gain changes, and compression.

## Datasets

- **TinySOL** — total ?; splits: test (-1)
- **Beatport EDM** — total ?; splits: test (-1)
- **VocalSet** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard harmonic mean of precision and recall, typically macro-averaged across classes. Used to measure classification performance on downstream tasks like singer identification.
- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances. Used alongside F1 to track performance drops under audio perturbations.

## Input / output format

**Input**: Raw audio files processed through pre-trained representation models to extract fixed-dimensional embeddings, which are then fed into downstream classifiers (SLP or MLPs of varying sizes).

**Output**: Class predictions from the downstream classifier (e.g., instrument label, musical key, singer ID, pitch class).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    # Macro-averaged F1 score
    f1 = f1_score(gold_labels, predictions, average='macro')
    return {'accuracy': accuracy, 'f1_score': f1}
```

## Common pitfalls

- Assuming representation features are linearly separable; downstream model capacity significantly impacts performance and can mask or reveal representation quality.
- Generalizing robustness across tasks; resilience to audio perturbations (e.g., noise, compression) varies heavily by task and representation training objective.

## Evidence (verbatim from paper)

> For singer identification in VocalSet, most models have a significant performance difference between the linear classifier and larger models, with cases like NeuralFP doubling their F1 score, although others like MERT exhibit no notable performance differences. From our experiments, we found that these representations generally struggle with audio deformations like white noise and gain reduction, though they fare better with intense MP3 compression.

## Citation

```bibtex
@misc{plachouras2023mir_ref,
  title={mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks},
  author={Plachouras et al. (2023)},
  year={2023},
  note={arXiv:2312.05994}
}
```

- arXiv: 2312.05994