Music Controlnet Eval

---
name: music-controlnet-eval
description: This evaluation probes a diffusion-based music generation model's ability to precisely follow time-varying control signals (melody, dynamics, rhythm) and global style tags (genre/mood). It measures how faithfully the generated audio adheres to these inputs while maintaining overall audio realism and diversity. Use when the user wants to benchmark on In-domain test set, MusicCaps, MusicCaps+ChatGPT, Created Controls dataset, or asks about evaluating this task. Reports Melody accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.07069
  bibtex_key: wu2023musiccontrolnet
  confidence: high
---

# music-controlnet-eval

> Music ControlNet: Multiple Time-varying Controls for Music Generation — Shih-Lun Wu et al. (2023) (arXiv:2311.07069, 2023)

## What this evaluates

This evaluation probes a diffusion-based music generation model's ability to precisely follow time-varying control signals (melody, dynamics, rhythm) and global style tags (genre/mood). It measures how faithfully the generated audio adheres to these inputs while maintaining overall audio realism and diversity.

## Datasets

- **In-domain test set** — total 2000; splits: test (2000)
- **MusicCaps** — total 5000; splits: test (5000)
- **MusicCaps+ChatGPT** — total 5000; splits: test (5000)
- **Created Controls dataset** — total 6400; splits: test (6400)

## Metrics

- `Melody accuracy` **(primary)** — range: percent
  - Percentage of frame-wise pitch classes (out of 12) that match between the input melody control and the pitch classes extracted from the generated audio.
- `Dynamics correlation` — range: [-1, 1]
  - Pearson’s correlation coefficient between input dynamics values and those computed from the generation. Computed per-generation (micro) or pooled across all generations (macro).
- `Rhythm F1` — range: percent
  - F1 score measuring alignment between beat/downbeat timestamps from the input control and the generation. Timestamps are considered aligned if they differ by less than 70 milliseconds.
- `CLAP score` — range: [-1, 1]
  - Cosine similarity between text embeddings (formatted as 'An audio of [mood] [genre] music') and audio embeddings extracted from the generated audio using the CLAP dual-encoder model.
- `FAD` — range: other
  - Fréchet Audio Distance measuring the distributional distance between embeddings of reference audios (typically the in-domain test set) and generated audios using a VGGish model. Lower is better.

## Input / output format

**Input**: Mel-scaled spectrograms (160 bins, ~86 Hz frame rate), global style embeddings (genre/mood), and time-varying control signals (melody pitch classes, dynamics curves, or rhythm/downbeat probabilities).

**Output**: Generated audio waveform (22.05 kHz) produced by a DiffWave vocoder from the model's predicted spectrogram.

## Scoring recipe

```python
def score_melody_acc(input_pitch, gen_pitch):
    return sum(1 for i in range(len(input_pitch)) if input_pitch[i] == gen_pitch[i]) / len(input_pitch)

def score_dynamics_corr(input_dyn, gen_dyn):
    return pearsonr(input_dyn, gen_dyn).statistic

def score_rhythm_f1(input_beats, gen_beats, threshold=0.07):
    tp = sum(1 for g in gen_beats if any(abs(g - i) < threshold for i in input_beats))
    prec = tp / len(gen_beats) if gen_beats else 0
    rec = tp / len(input_beats) if input_beats else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def score_clap(text_prompt, gen_audio):
    return cosine_similarity(clap_text_embed(text_prompt), clap_audio_embed(gen_audio))

def score_fad(ref_audios, gen_audios):
    return frechet_distance(vggish_embed(ref_audios), vggish_embed(gen_audios))
```

## Common pitfalls

- Micro vs. macro dynamics correlation are computed differently (per-sample vs. pooled across all samples); mixing them invalidates cross-paper comparisons.
- FAD scores are highly sensitive to the reference dataset; the paper specifies using the in-domain test set, so using a different reference will produce incomparable scores.
- Rhythm F1 relies on a strict 70ms alignment threshold; relaxing this threshold significantly inflates scores and deviates from the paper's protocol.
- CLAP evaluation requires the exact prompt template 'An audio of [mood] [genre] music'; using raw free-form text will yield lower, non-comparable adherence scores.

## Evidence (verbatim from paper)

> We use the following metrics to evaluate time-varying controllability, adherence to global text (i.e., mood \& genre tags) control, and overall audio realism. Melody accuracy examines whether the frame-wise pitch classes (C, C#,…, B; 12 in total) match between the input melody control and that extracted from the generation. Dynamics correlation is the Pearson’s correlation between the frame-wise input dynamics values to the values computed from the generation. Rhythm F1 follows the standard evaluation methodology for beat/downbeat detection. It quantifies the alignment between the beat/downbeat timestamps estimated from the input rhythm control, and those from the generation.

## Citation

```bibtex
@misc{wu2023musiccontrolnet,
  title={Music ControlNet: Multiple Time-varying Controls for Music Generation},
  author={Shih-Lun Wu et al. (2023)},
  year={2023},
  note={arXiv:2311.07069}
}
```

- arXiv: 2311.07069