Musicsem Eval

---
name: musicsem-eval
description: Evaluates multimodal models on their ability to understand, generate, and retrieve music based on semantically rich, context-aware natural language descriptions. It probes fine-grained musical semantics beyond technical attributes, including atmospheric, situational, and contextual cues. Use when the user wants to benchmark on MusicSem, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.17769
  bibtex_key: salganik2026musicsem
  confidence: medium
---

# musicsem-eval

> MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions — Salganik et al. (2026) (arXiv:2602.17769, 2026)

## What this evaluates

Evaluates multimodal models on their ability to understand, generate, and retrieve music based on semantically rich, context-aware natural language descriptions. It probes fine-grained musical semantics beyond technical attributes, including atmospheric, situational, and contextual cues.

## Datasets

- **MusicSem** — total ?; splits: test (-1), train (-1); repo https://github.com/Rsalganik1123/MusicSem

## Metrics

- `BLEU` **(primary)** — range: [0, 1]
  - Measures n-gram precision (typically unigrams, bigrams, and trigrams) between the generated caption and the ground-truth description.
- `ROUGE` — range: [0, 1]
  - Measures recall-based overlap of n-grams between the generated and reference captions.
- `METEOR` — range: [0, 1]
  - A metric that combines precision and recall with synonymy and stemming matching to evaluate semantic similarity.

## Input / output format

**Input**: Text-to-Music: text prompt. Music-to-Text: audio clip. Retrieval: paired text and audio.

**Output**: Text-to-Music: audio waveform. Music-to-Text: natural language caption. Retrieval: similarity score or ranked list.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    bleu = nltk.bleu_score.corpus_bleu(references, predictions)
    rouge = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
    rouge_scores = [rouge.score(ref, pred)['rouge1'].fmeasure for ref, pred in zip(references, predictions)]
    meteor = compute_meteor(predictions, references)
    return {'bleu': bleu, 'rouge': sum(rouge_scores)/len(rouge_scores), 'meteor': meteor}
```

## Common pitfalls

- MusicCaps and Song Describer lack official train/test splits, so models are evaluated on the full public data, risking overfitting assessment.
- Generation duration varies significantly across models, making direct latency comparisons difficult without normalizing by output length.
- Proprietary models (e.g., Mureka) are evaluated via API, limiting reproducibility of exact generation parameters.

## Evidence (verbatim from paper)

> For all evaluations on MusicSem, we use only this held-out portion of the data for testing, while releasing the remaining entries as the public training set. ... Following canonical works in music-to-text generation, we first consider three n-gram-based metrics originally developed for machine translation: BLEU, ROUGE, and METEOR. BLEU (B) measures precision by computing the overlap of n-grams (typically unigrams, bigrams, and trigrams; i.e., B1, B2, and B3) between the

## Citation

```bibtex
@misc{salganik2026musicsem,
  title={MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions},
  author={Salganik et al. (2026)},
  year={2026},
  note={arXiv:2602.17769}
}
```

- arXiv: 2602.17769