Music Caption Retrieval Eval

---
name: music-caption-retrieval-eval
description: Evaluates a model's ability to generate fine-grained, temporally-aware music captions and retrieve corresponding audio segments using those captions. It probes the model's capacity for temporal reasoning, structural music understanding, and cross-modal alignment. Use when the user wants to benchmark on MusicCaps, Song Descriptor, or asks about evaluating this task. Reports BLEU-1/2/3, METEOR, ROUGE-L, BERTScore, Recall@K, Median Rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.20445
  bibtex_key: wu2024futga
  confidence: high
---

# music-caption-retrieval-eval

> Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation — Junda Wu et al. (arXiv:2407.20445, 2024)

## What this evaluates

Evaluates a model's ability to generate fine-grained, temporally-aware music captions and retrieve corresponding audio segments using those captions. It probes the model's capacity for temporal reasoning, structural music understanding, and cross-modal alignment.

## Datasets

- **MusicCaps** — total ?; splits: test (-1)
- **Song Descriptor** — total ?; splits: test (-1)

## Metrics

- `BLEU-1/2/3, METEOR, ROUGE-L, BERTScore` **(primary)** — range: percent
  - Standard NLP sequence-to-sequence metrics comparing generated captions against human annotations. BLEU measures n-gram precision, METEOR adds synonymy and stemming, ROUGE-L uses longest common subsequence, and BERTScore uses contextual embeddings for semantic similarity.
- `Recall@K, Median Rank` **(primary)** — range: percent
  - For retrieval, compute IoU between caption and audio time segments. Use CLAP to extract features and calculate IoU-weighted average cosine similarity. Rank audio clips by similarity; Recall@K is the fraction of correct clips in the top K, and Median Rank is the median position of the correct clip.
- `CLAP Score` — range: [0, 1]
  - Measures audio-text similarity by computing the cosine similarity between audio and text embeddings extracted from the CLAP model.

## Input / output format

**Input**: Audio clips with associated time segments and text captions (global or segment-level with time boundaries).

**Output**: Generated text captions (global or temporally segmented), ranked list of retrieved audio clips, or generated audio tracks.

## Scoring recipe

```python
def evaluate_caption(pred, gold):
    return bleu(pred, gold), meteor(pred, gold), rouge(pred, gold), bertscore(pred, gold)

def evaluate_retrieval(preds, gold, clips, segments):
    ious = compute_iou(preds.segments, clips.segments)
    feats = clap_encode([preds.text, clips.audio])
    sims = ious * cosine_similarity(feats.text, feats.audio)
    ranks = rank_by_similarity(sims)
    return recall_at_k(ranks, k=[1,5,10]), median_rank(ranks)

def evaluate_generation(gen_audio, prompt):
    return clap_score(gen_audio, prompt)
```

## Common pitfalls

- Comparing FUTGA's temporally segmented captions directly to global MusicCaps captions without accounting for formal differences, which artificially penalizes the model.
- Using LLM-generated captions for retrieval without considering that CLAP's text encoder may be out-of-distribution for complex LLM language, leading to lower-than-expected retrieval scores.
- Failing to separate global vs. complete (segmented) caption evaluations, which conflates different levels of temporal granularity.

## Evidence (verbatim from paper)

> We follow the previous works [14, 22] and report the metrics, BLEU (B),METEOR (M), ROUGE (R), and BERT-score (B-S), in Table 3. Since our captions are formally different from original music captions, we report the evaluation metrics for the global and the complete captions in our dataset separately.

## Citation

```bibtex
@misc{wu2024futga,
  title={Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation},
  author={Junda Wu et al.},
  year={2024},
  note={arXiv:2407.20445}
}
```

- arXiv: 2407.20445