Tinymu Music Eval

---
name: tinymu-music-eval
description: Evaluates a compact audio-language model's ability to perform music information retrieval (genre and instrument classification), generate descriptive music captions, and answer complex multiple-choice questions about musical theory and structure. Use when the user wants to benchmark on GTZAN, Medley-Solos-DB, MusicCaps, MuChoMusic, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.15849
  bibtex_key: li2026tinymu
  confidence: high
---

# tinymu-music-eval

> TinyMU: A Compact Audio-Language Model for Music Understanding — Li et al. (2026) (arXiv:2604.15849, 2026)

## What this evaluates

Evaluates a compact audio-language model's ability to perform music information retrieval (genre and instrument classification), generate descriptive music captions, and answer complex multiple-choice questions about musical theory and structure.

## Datasets

- **GTZAN** — total 1000; splits: test (1000)
- **Medley-Solos-DB** — total 12236; splits: test (12236)
- **MusicCaps** — total 2800; splits: test (2800)
- **MuChoMusic** — total 1187; splits: test (1187)

## Metrics

- `classification accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels. For MIR tasks, the model's text output is matched to candidate labels using CLAP text encoder similarity; the highest-scoring label is selected as the prediction.
- `METEOR` — range: [0, 1]
  - Automatic evaluation metric for text generation that measures lexical overlap, synonymy, and stemming between generated and reference captions.
- `BERTScore` — range: [0, 1]
  - Evaluates text generation by computing cosine similarity between contextual embeddings of generated and reference texts using a BERT model.
- `accuracy` — range: percent
  - Percentage of correctly answered multiple-choice questions on the reasoning benchmark.

## Input / output format

**Input**: Audio clip paired with a text prompt (candidate labels for classification, open-ended question for captioning, or multiple-choice question for reasoning).

**Output**: Text string corresponding to the predicted label, generated caption, or selected answer choice.

## Scoring recipe

```python
# MIR Tasks (GTZAN, Medley-Solos-DB)
pred = max(candidates, key=lambda c: clap_text_encoder_similarity(model_output, c))
acc = 1.0 if pred == gold_label else 0.0

# Captioning (MusicCaps)
meteor = compute_meteor(generated_caption, gold_caption)
bertscore = compute_bertscore(generated_caption, gold_caption)

# Reasoning (MuChoMusic)
acc = 1.0 if predicted_choice == gold_choice else 0.0
```

## Common pitfalls

- MIR tasks rely on CLAP text encoder similarity to match model outputs to candidate labels rather than exact string matching, which can cause mismatches if the model paraphrases the label.
- The evaluation assumes zero-shot transfer for GTZAN and Medley-Solos-DB as they do not overlap with training data, but dataset splits are not explicitly detailed beyond 'test'.
- Freezing the language model severely degrades QA performance; full fine-tuning or LoRA is required for reasoning tasks, as shown in ablation studies.

## Evidence (verbatim from paper)

> Following prior work [[18], [20]], we compute the similarity between TinyMU’s outputs and candidate labels using the text encoder of CLAP [[14]], and select the highest-scoring label as model output. We then report classification accuracy for these two benchmarks. We report the commonly adopted metrics: METEOR [[4]], and BERTScore [[31]]. We report the accuracy of models on the benchmark.

## Citation

```bibtex
@misc{li2026tinymu,
  title={TinyMU: A Compact Audio-Language Model for Music Understanding},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.15849}
}
```

- arXiv: 2604.15849