Vabench Eval

---
name: vabench-eval
description: Evaluates the quality, cross-modal consistency, synchronization, and spatial audio rendering of text-to-audio-video and image-to-audio-video generation models. It probes physical plausibility, emotional expressiveness, and stereo separation across seven real-world sound categories. Use when the user wants to benchmark on VABench, or asks about evaluating this task. Reports Audio-Visual Align.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.09299
  bibtex_key: hua2025vabench
  confidence: high
---

# vabench-eval

> VABench: A Comprehensive Benchmark for Audio-Video Generation — Hua et al. (2025) (arXiv:2512.09299, 2025)

## What this evaluates

Evaluates the quality, cross-modal consistency, synchronization, and spatial audio rendering of text-to-audio-video and image-to-audio-video generation models. It probes physical plausibility, emotional expressiveness, and stereo separation across seven real-world sound categories.

## Datasets

- **VABench** — total ?; splits: test (-1); repo https://github.com/tanABCC/VABench

## Metrics

- `Audio-Visual Align` **(primary)** — range: [0, 1]
  - Cross-modal semantic alignment score between generated audio and video, assessed via expert-based and MLLM-driven evaluation.
- `Lip-Sync` — range: [0, 1]
  - Synchronization accuracy for speech and lip movements, measured via expert/MLLM assessment.
- `Audio Realism` — range: [0, 1]
  - Fidelity and naturalness of the generated audio track.
- `Video Realism` — range: [0, 1]
  - Fidelity and naturalness of the generated video sequence.
- `Audio QA` — range: [0, 1]
  - Fine-grained semantic understanding and question-answering capability on the generated audio.
- `Vision QA` — range: [0, 1]
  - Fine-grained semantic understanding and question-answering capability on the generated video.
- `Phase Coherence` — range: [0, 1]
  - Normalized acoustic metric measuring left-right channel phase consistency across low, mid, and high frequencies.
- `Mono Compatibility` — range: [0, 1]
  - Normalized acoustic metric evaluating how well the stereo mix collapses to mono without phase cancellation artifacts.
- `Soundstage Width` — range: [0, 1]
  - Normalized acoustic metric quantifying the perceived spatial width of the audio field.
- `Transient Synchronization` — range: [0, 1]
  - Normalized acoustic metric measuring temporal alignment of sharp sound events across channels.
- `Level Stability` — range: [0, 1]
  - Normalized acoustic metric assessing consistent amplitude levels across the stereo track.
- `Envelope Correlation` — range: [0, 1]
  - Normalized acoustic metric evaluating the similarity of amplitude envelope shapes between left and right channels.
- `Imaging Stability` — range: [0, 1]
  - Normalized acoustic metric measuring the steadiness of localized sound sources in the stereo field.

## Input / output format

**Input**: Text prompts (for T2AV) or reference images (for I2AV), optionally with audio prompts for stereo tasks. Models generate synchronized video and audio outputs.

**Output**: Synchronized video (720P, default frame rate/duration) and audio (48kHz stereo tracks) files.

## Scoring recipe

```python
def compute_vabench_scores(predictions, gold_prompts):
    scores = {}
    for model, (vid, aud) in predictions.items():
        scores[model] = {
            'Audio-Visual Align': eval_cross_modal(vid, aud),
            'Lip-Sync': eval_lip_sync(vid, aud),
            'Audio Realism': eval_realism(aud),
            'Video Realism': eval_realism(vid),
            'Audio QA': eval_qa(aud, gold_prompts),
            'Vision QA': eval_qa(vid, gold_prompts),
            **{m: eval_acoustic_metric(aud, m) for m in STEREO_METRICS}
        }
    return scores
# Human validation: 6 evaluators rate subset on 1-5 scale. Compute pairwise win rates (Win=1, Loss=0, Tie=0.5) for humans and benchmark scores. Calculate Pearson correlation between the two win-rate vectors.
```

## Common pitfalls

- Confusing end-to-end audio-video models with decoupled video+audio pipelines, which have fundamentally different training objectives and performance characteristics.
- Assuming high stereo acoustic metric scores imply semantic spatial awareness; the paper notes models often produce technical stereo separation without corresponding visual or semantic grounding.
- Evaluating across only a single sound category; performance varies significantly across the seven categories (e.g., strong on Music/Animals, weak on Human Sounds).

## Evidence (verbatim from paper)

> To validate VABench’s alignment with human senses, we conducted a pilot user study. Balancing evaluation fidelity and cost, we had six professional evaluators rate a representative subset of videos from three models (Veo3, Seedance+MMAudio, Wan2.2+ThinkSound) on a 1–5 scale. The evaluation focused on three key dimensions—semantics, synchronization, and realism—which directly correspond to aggregated benchmark metrics (Semantics: Text-Video Align, Text-Audio Align, Audio-Visual Align; Synchronization: Desync; Realism: Audio Realism, Video Realism).

## Citation

```bibtex
@misc{hua2025vabench,
  title={VABench: A Comprehensive Benchmark for Audio-Video Generation},
  author={Hua et al. (2025)},
  year={2025},
  note={arXiv:2512.09299}
}
```

- arXiv: 2512.09299