Vidmuse Video To Music Eval

---
name: vidmuse-video-to-music-eval
description: Evaluates a model's ability to generate high-fidelity, semantically aligned music conditioned on video input. It probes audio quality, diversity, and cross-modal alignment using statistical distance metrics, beat alignment scores, and subjective human preference tests. Use when the user wants to benchmark on V2M, AIST++, LORIS, TikTok, or asks about evaluating this task. Reports FAD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.04321
  bibtex_key: tian2024vidmuse
  confidence: high
---

# vidmuse-video-to-music-eval

> VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling — Tian et al. (2024) (arXiv:2406.04321, 2024)

## What this evaluates

Evaluates a model's ability to generate high-fidelity, semantically aligned music conditioned on video input. It probes audio quality, diversity, and cross-modal alignment using statistical distance metrics, beat alignment scores, and subjective human preference tests.

## Datasets

- **V2M** — total ?; splits: test (-1)
- **AIST++** — total ?; splits: test (-1)
- **LORIS** — total ?; splits: test (-1)
- **TikTok** — total ?; splits: test (-1)

## Metrics

- `FAD` **(primary)** — range: [0, inf)
  - Frechet Audio Distance; measures the distance between feature distributions of generated and real audio in a pretrained feature space. Lower is better.
- `ImageBind Score` — range: [0, 1]
  - Cross-modal alignment score computed using the ImageBind model to measure semantic consistency between video and generated music. Higher is better.
- `BCS` — range: [0, 100]
  - Beats Coverage Score; measures the proportion of ground truth beats covered by generated music beats. Higher is better.
- `BHS` — range: [0, 100]
  - Beats Hit Score; measures the proportion of generated beats that match ground truth beats. Higher is better.
- `Density` — range: [0, inf)
  - Diversity metric indicating the average pairwise distance between generated samples. Higher indicates greater diversity.
- `Coverage` — range: [0, 1]
  - Diversity metric indicating the fraction of ground truth samples covered by generated samples. Higher indicates better coverage.

## Input / output format

**Input**: 30-second video segments; frames sampled at 2 fps for short-term modeling and 32 uniformly sampled frames for long-term modeling.

**Output**: Autoregressive music tokens sampled via top-k (k=250, temp=1.0), decoded into 32 kHz monophonic audio waveforms.

## Scoring recipe

```python
def evaluate(predictions, gold):
    fad = frechet_audio_distance(predictions, gold)
    img_score = imagebind_alignment(predictions, gold)
    density, coverage = diversity_metrics(predictions)
    bcs, bhs = beat_coverage_hit(predictions, gold)
    return {'FAD': fad, 'ImageBind': img_score, 'Density': density, 'Coverage': coverage, 'BCS': bcs, 'BHS': bhs}
```

## Common pitfalls

- ImageBind Score is not trained on music data, so it may poorly capture semantic alignment for audio.
- BCS/BHS metrics are borrowed from non-public baselines and may lack standardized open-source implementations.
- Sliding window inference (30s window, 0.5s overlap) can introduce boundary artifacts that skew diversity metrics if not evaluated on full untrimmed generations.

## Evidence (verbatim from paper)

> To quantitatively evaluate the effectiveness of our model, we employ a series of metrics to assess different models in terms of quality, fidelity, and diversity of the generated music. These metrics include the Frechet Audio Distance (FAD), Frechet Distance (FD), Kullback-Leibler Divergence (KL), as well as Density and Coverage*[[55]]*. Additionally, we utilize the ImageBind Score*[[22]]* to examine the alignment between the video and the generated music.

## Citation

```bibtex
@misc{tian2024vidmuse,
  title={VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2406.04321}
}
```

- arXiv: 2406.04321