Musiccaps Eval

---
name: musiccaps-eval
description: Evaluates a model's ability to generate high-fidelity, long-form music from complex text descriptions. It probes both audio quality/plausibility and the model's adherence to specific textual constraints such as genre, mood, tempo, and instrumentation. Use when the user wants to benchmark on MusicCaps, or asks about evaluating this task. Reports FAD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.11325
  bibtex_key: agostinelli2023musiclm
  confidence: high
---

# musiccaps-eval

> MusicLM: Generating Music From Text — Andrea Agostinelli et al. (arXiv:2301.11325, 2023)

## What this evaluates

Evaluates a model's ability to generate high-fidelity, long-form music from complex text descriptions. It probes both audio quality/plausibility and the model's adherence to specific textual constraints such as genre, mood, tempo, and instrumentation.

## Datasets

- **MusicCaps** — total 5500; splits: train (-1), eval (-1), genre-balanced (1000)

## Metrics

- `FAD` **(primary)** — range: other
  - Fréchet Audio Distance computed between audio embeddings (using Trill2 or VGGish models) of generated and reference clips. Lower scores indicate higher audio quality and realism.
- `KLD` — range: other
  - KL divergence between the class prediction probability distributions of a LEAF classifier on AudioSet for generated versus reference music. Measures adherence to acoustic characteristics implied by the text.
- `MCC` — range: [0, 1]
  - Average cosine similarity between MuLan embeddings of the input text captions and the generated music. Higher values indicate better text-music alignment.

## Input / output format

**Input**: Free-text caption (average 4 sentences) describing music, including genre, mood, tempo, instrumentation, and other aspects.

**Output**: 10-second audio clip (generated via autoregressive token sampling with temperature 1.0/0.95/0.4 across stages).

## Scoring recipe

```python
def score(predictions, gold):
    # FAD: Fréchet distance between audio embeddings
    fad = frechet_distance(embed_audio(predictions), embed_audio(gold))
    # KLD: KL divergence of LEAF classifier predictions
    pred_probs = classifier.predict(predictions)
    gold_probs = classifier.predict(gold)
    kld = kl_divergence(pred_probs, gold_probs)
    # MCC: Cosine similarity of MuLan text/audio embeddings
    text_embs = embed_text(gold['caption'])
    music_embs = embed_audio(predictions)
    mcc = cosine_similarity(text_embs, music_embs).mean()
    return {'FAD': fad, 'KLD': kld, 'MCC': mcc}
```

## Common pitfalls

- FAD measures audio quality/plausibility, not text adherence; a low FAD does not guarantee the generated music matches the prompt.
- KLD relies on a proxy LEAF classifier trained on AudioSet, which may not capture nuanced musical aspects or genre-specific details.
- Human evaluation explicitly instructs raters to ignore audio quality to isolate text adherence, which can skew subjective preferences.

## Evidence (verbatim from paper)

> We compute different metrics to evaluate MusicLM, capturing two important aspects of music generation: the audio quality and the adherence to the text description. We report the FAD based on two audio embedding models, both of which are publicly available: (1) Trill2... and (2) VGGish... we use a LEAF classifier trained for multi-label classification on AudioSet, to compute class predictions for both the generated and the reference music and measure the KL divergence between probability distributions of class predictions... define the MCC metric as the average cosine similarity between these embeddings.

## Citation

```bibtex
@misc{agostinelli2023musiclm,
  title={MusicLM: Generating Music From Text},
  author={Andrea Agostinelli et al.},
  year={2023},
  note={arXiv:2301.11325}
}
```

- arXiv: 2301.11325