Dualbench Eval

---
name: dualbench-eval
description: Evaluates a model's ability to generate synchronized background audio and intelligible speech from video input, measuring audio quality, distribution matching, and audio-video temporal alignment. Use when the user wants to benchmark on DualBench, VGGSound, or asks about evaluating this task. Reports FAD↓.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10109
  bibtex_key: tian2025dualdub
  confidence: high
---

# dualbench-eval

> DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis — Tian et al. (2025) (arXiv:2507.10109, 2025)

## What this evaluates

Evaluates a model's ability to generate synchronized background audio and intelligible speech from video input, measuring audio quality, distribution matching, and audio-video temporal alignment.

## Datasets

- **DualBench** — total ?; splits: test (-1)
- **VGGSound** — total ?; splits: test (-1)

## Metrics

- `FAD↓` **(primary)** — range: other
  - Fréchet Audio Distance: measures the distance between feature distributions of generated and real audio using a pre-trained audio classifier. Lower is better.
- `AV-Align↑` — range: [0, 1]
  - Audio-Video Alignment score: quantifies the temporal synchronization and acoustic harmony between generated audio and input video frames. Higher is better.
- `FD↓` — range: other
  - Fréchet Distance: measures the distance between feature distributions of generated and real audio in a latent space. Lower is better.
- `KLD↓` — range: other
  - Kullback-Leibler Divergence: measures the difference between the probability distributions of generated and real audio features. Lower is better.
- `IS↑` — range: other
  - Inception Score: evaluates the quality and diversity of generated audio based on a classifier's confidence and entropy. Higher is better.

## Input / output format

**Input**: Video frames processed via a CLIP ViT-B/32 visual encoder; optionally text prompts (though baselines use video-only input).

**Output**: Synchronized audio waveform containing both background sound and speech, sampled at 24kHz with 40 tokens per second.

## Scoring recipe

```python
def compute_metrics(generated_audios, reference_audios, videos):
    # Extract features using pre-trained audio/video encoders
    gen_feats = extract_features(generated_audios)
    ref_feats = extract_features(reference_audios)
    
    # Compute distribution distances
    fd = frechet_distance(gen_feats, ref_feats)
    fad = frechet_audio_distance(gen_feats, ref_feats)
    kld = kl_divergence(gen_feats, ref_feats)
    is_score = inception_score(gen_feats)
    
    # Compute alignment
    av_align = compute_audio_video_alignment(generated_audios, videos)
    
    return {'FD': fd, 'FAD': fad, 'KLD': kld, 'IS': is_score, 'AV-Align': av_align}
```

## Common pitfalls

- Speech leaking into background audio tracks during generation or evaluation
- Mismatched frame rates between video features and audio tokens causing misalignment
- Using text prompts during inference for video-to-audio baselines, violating the video-only setting

## Evidence (verbatim from paper)

> Table 2: Objective evaluation of generated audio on the VGGSound and DualBench test sets. The best and the second best result is shown in bold and by underlined. 

| Model | VGGSound | ... | FD↓ | FAD↓ | KLD↓ | IS↑ | AV-Align↑ |

## Citation

```bibtex
@misc{tian2025dualdub,
  title={DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis},
  author={Tian et al. (2025)},
  year={2025},
  note={arXiv:2507.10109}
}
```

- arXiv: 2507.10109