Clamp2 Music Retrieval Eval

---
name: clamp2-music-retrieval-eval
description: Evaluates a model's ability to classify symbolic music into genres, emotions, or composer styles, and to perform cross-modal semantic search between music scores (ABC/MIDI) and textual descriptions. It also probes multilingual retrieval capabilities by testing performance across machine-translated text queries. Use when the user wants to benchmark on WikiMT, VGMIDI, Pianist8, MidiCaps, or asks about evaluating this task. Reports Accuracy, MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13267
  bibtex_key: wu2024clamp2
  confidence: high
---

# clamp2-music-retrieval-eval

> CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models — Shangda Wu et al. (2024) (arXiv:2410.13267, 2024)

## What this evaluates

Evaluates a model's ability to classify symbolic music into genres, emotions, or composer styles, and to perform cross-modal semantic search between music scores (ABC/MIDI) and textual descriptions. It also probes multilingual retrieval capabilities by testing performance across machine-translated text queries.

## Datasets

- **WikiMT** — total 1010; splits: test (1010)
- **VGMIDI** — total 204; splits: test (204)
- **Pianist8** — total 411; splits: test (411)
- **MidiCaps** — total 1010; splits: test (1010)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances in the test set.
- `F1-macro` — range: [0, 1]
  - Unweighted mean of recall (or F1) for each class, treating all classes equally regardless of their support.
- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: average of 1/rank for the first correctly retrieved item across all queries.
- `HR@K` — range: [0, 1]
  - Hit Rate at top K: fraction of queries where the relevant item appears in the top K ranked results.

## Input / output format

**Input**: Per instance: either a symbolic music representation (ABC notation or MIDI file) for classification, or a text query paired with a candidate music piece for semantic search.

**Output**: Per instance: a predicted class label (genre, emotion, or composer) for classification; or a ranked list of music-text pairs for retrieval.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, ranked_lists=None):
    if ranked_lists is None:
        correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
        accuracy = correct / len(gold_labels)
        f1_macro = average_f1_per_class(predictions, gold_labels)
        return {'accuracy': accuracy, 'f1_macro': f1_macro}
    else:
        mrr_scores = []
        hr_counts = {1: 0, 10: 0, 100: 0}
        for ranks, gold in zip(ranked_lists, gold_labels):
            rank = ranks.index(gold) + 1 if gold in ranks else len(ranks) + 1
            mrr_scores.append(1.0 / rank)
            for k in [1, 10, 100]:
                if rank <= k:
                    hr_counts[k] += 1
        mrr = sum(mrr_scores) / len(mrr_scores)
        hr = {k: v / len(gold_labels) for k, v in hr_counts.items()}
        return {'mrr': mrr, 'hr@1': hr[1], 'hr@10': hr[10], 'hr@100': hr[100]}
```

## Common pitfalls

- Data leakage: The pre-training set includes the Lakh MIDI dataset, so the test set for MidiCaps must be carefully sampled (1,010 pieces) to avoid overlap with training data.
- Modality mismatch: Baseline models like CLaMP do not natively support MIDI, requiring conversion to ABC notation for fair comparison, which may lose performance details.
- Translation quality dependency: Multilingual retrieval results are heavily influenced by the quality of machine-translated queries, measured via BLEU scores of back-translations.

## Evidence (verbatim from paper)

> Table 1: Classification performance for ABC notation and MIDI was assessed across three datasets: WikiMT (1,010 pieces, 8 genres), VGMIDI (204 pieces, 4 emotions), and Pianist8 (411 pieces, 8 composers). Table 2 shows semantic search results on the WikiMT and MidiCaps benchmarks, using Mean Reciprocal Rank (MRR) and Hit Rate at Top K (HR@K) to assess model performance in retrieving and ranking relevant music-text pairs.

## Citation

```bibtex
@misc{wu2024clamp2,
  title={CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models},
  author={Shangda Wu et al. (2024)},
  year={2024},
  note={arXiv:2410.13267}
}
```

- arXiv: 2410.13267