Mmaudiosep Separation Eval

---
name: mmaudiosep-separation-eval
description: Evaluates a generative model's ability to separate target sounds from mixture audio using video and text queries, while preserving video-to-audio generation capabilities. It probes multimodal conditioning, cross-domain knowledge transfer, and the perceptual quality of generated separated audio against discriminative baselines. Use when the user wants to benchmark on VGGSound-Clean, MUSIC, VGGSound, or asks about evaluating this task. Reports FAD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.09065
  bibtex_key: takahashi2025mmaudiosep
  confidence: high
---

# mmaudiosep-separation-eval

> MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation — Akira Takahashi et al. (2025) (arXiv:2510.09065, 2025)

## What this evaluates

Evaluates a generative model's ability to separate target sounds from mixture audio using video and text queries, while preserving video-to-audio generation capabilities. It probes multimodal conditioning, cross-domain knowledge transfer, and the perceptual quality of generated separated audio against discriminative baselines.

## Datasets

- **VGGSound-Clean** — total 5004; splits: test (5004); repo https://github.com/Audio-AGI/AudioSep
- **MUSIC** — total 1000; splits: test (1000); repo https://github.com/Audio-AGI/AudioSep
- **VGGSound** — total 15000; splits: test (15000)

## Metrics

- `FAD` **(primary)** — range: other
  - Fréchet Audio Distance computed on PaSST features to measure feature distribution similarity between generated and reference audio. Lower values indicate better distribution matching.
- `CLAPScore` — range: [0, 1]
  - Semantic alignment score between output audio and text query using CLAP embeddings. Higher values indicate better text-audio alignment.
- `CLAPScore_A` — range: [0, 1]
  - Similarity score between output audio and ground-truth target audio using CLAP embeddings. Higher values indicate better target reconstruction.
- `IB-Score` — range: [0, 1]
  - Average cosine similarity of ImageBind features between input video and generated audio to measure AV semantic alignment.
- `DeSync` — range: other
  - Temporal misalignment predicted by Synchformer features between input video and generated audio. Lower values indicate better synchronization.
- `IS` — range: other
  - Inception Score calculated with a PANNs classifier to evaluate the quality and diversity of generated audio.
- `FD` — range: other
  - Fréchet Distance computed on PaSST features for distribution matching between generated and reference audio.
- `KL` — range: other
  - Kullback-Leibler divergence computed on PaSST features for distribution matching between generated and reference audio.

## Input / output format

**Input**: 10-second mixture audio, 10-second video query (padded with black frames if shorter), and text query (class or instrument label).

**Output**: 10-second separated target audio (for separation task) or 8-second generated audio (for V2A task).

## Scoring recipe

```python
def evaluate_separation(pred_audio, gold_audio, text_query, video_query):
    # Compute generative metrics as specified in the paper
    fad = frechet_audio_distance(pred_audio, gold_audio, model='PaSST')
    clap_score = clap_similarity(pred_audio, text_query)
    clap_a = clap_similarity(pred_audio, gold_audio)
    ib_score = cosine_similarity(imagebind(pred_audio), imagebind(video_query))
    desync = synchformer_misalignment(pred_audio, video_query)
    is_score = inception_score(pred_audio, classifier='PANNs')
    fd = frechet_distance(pred_audio, gold_audio, model='PaSST')
    kl = kl_divergence(pred_audio, gold_audio, model='PaSST')
    return {'FAD': fad, 'CLAPScore': clap_score, 'CLAPScore_A': clap_a,
            'IB-Score': ib_score, 'DeSync': desync, 'IS': is_score,
            'FD': fd, 'KL': kl}
```

## Common pitfalls

- Using discriminative metrics like SDR instead of generative metrics (FAD, CLAP), which the authors explicitly note are suboptimal for this generative approach.
- Including PANNs in FD/KL computation, as the paper explicitly excludes it due to reported robustness issues in some scenarios.
- Failing to pad videos to exactly 10 seconds, which breaks temporal consistency between audio and video inputs required for the evaluation protocol.

## Evidence (verbatim from paper)

> As MMAudioSep is a generative-based approach, conventional sample-level objective metrics for sound separation tasks, such as the source-to-distortion ratio (SDR), are suboptimal for assessing the proposed system. Instead, we follow the evaluation protocol of FlowSep [[5]], employing metrics suited for generative tasks. These include Fréchet Audio Distance (FAD) [[28]] for feature distribution similarity and CLAP-based scores [[29]]. Specifically, CLAPScore evaluates the semantic alignment between the output audio and the text query, while CLAPScore ${}_{\textbf{A}}$ measures the similarity to the ground-truth target audio.

## Citation

```bibtex
@misc{takahashi2025mmaudiosep,
  title={MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation},
  author={Akira Takahashi et al. (2025)},
  year={2025},
  note={arXiv:2510.09065}
}
```

- arXiv: 2510.09065