Versebench Eval

---
name: versebench-eval
description: Evaluates a unified multimodal model's ability to generate synchronized audio and video from text, phonemes, and reference media. It probes zero-shot voice cloning fidelity, lip-sync accuracy, acoustic quality, and cross-modal temporal alignment. Use when the user wants to benchmark on VerseBench, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.01568
  bibtex_key: qiang2026mmsonate
  confidence: high
---

# versebench-eval

> MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning — Qiang et al. (2026) (arXiv:2601.01568, 2026)

## What this evaluates

Evaluates a unified multimodal model's ability to generate synchronized audio and video from text, phonemes, and reference media. It probes zero-shot voice cloning fidelity, lip-sync accuracy, acoustic quality, and cross-modal temporal alignment.

## Datasets

- **VerseBench** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate computed by transcribing generated audio with Whisper-large-v3 and comparing against ground-truth text.
- `SIM-o` — range: [0, 1]
  - Speaker similarity score calculated using the WavLM-large speaker verification model following the SeedTTS protocol.
- `SNC` — range: other
  - SyncNet Confidence score measuring lip-synchronization quality between generated audio and video frames.
- `CLAP` — range: [0, 1]
  - Semantic alignment score between audio and text using the CLAP model.
- `FD` — range: other
  - Fréchet Distance on mel-spectrogram features extracted via PANNs, measuring distributional distance.
- `AS` — range: [0, 1]
  - Aesthetic Score derived from MANIQA, aesthetic-predictor-v2-5, and Musiq.
- `MS` — range: [0, 1]
  - Motion Score based on RAFT optical flow to quantify dynamic realism.
- `ID` — range: [0, 1]
  - Identity Consistency measured as mean DINOV3 feature similarity between reference and generated frames.
- `AV-A` — range: other
  - Audio-Video Alignment distance computed via Synchformer.
- `IB` — range: [0, 1]
  - ImageBind score measuring global cross-modal alignment.

## Input / output format

**Input**: Text instructions, phoneme sequences, and optionally reference audio (for timbre cloning) and reference images (for animation).

**Output**: Synchronized video frames and corresponding audio waveform.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    wer = whisper_transcribe(predictions.audio).wer(gold.text)
    sim_o = wavlm_similarity(predictions.audio, gold.ref_audio)
    snc = syncnet_confidence(predictions.audio, predictions.video)
    clap = clap_score(predictions.audio, gold.text)
    fd = frechet_distance(melspectrogram(predictions.audio), melspectrogram(gold.audio))
    return {'WER': wer, 'SIM-o': sim_o, 'SNC': snc, 'CLAP': clap, 'FD': fd}
```

## Common pitfalls

- Comparing joint audio-video generation models against audio-only TTS baselines without acknowledging the increased generative complexity of the multimodal task.
- Using different speaker verification backends (WavLM-large for SIM-o vs. Resemblyzer for ablation studies), which prevents direct numerical comparison of identity preservation scores across tables.
- Relying solely on automated metrics for lip-sync and identity, as the paper also supplements these with human preference tests and pass-rate thresholds.

## Evidence (verbatim from paper)

> Speech intelligibility is quantified by the Word Error Rate (WER), derived from Whisper-large-v3 transcriptions. To assess speaker identity preservation, we employ specific metrics depending on the evaluation context. For the main comparative benchmark against state-of-the-art TTS models (Table[2]), we follow the SeedTTS evaluation protocol and report SIM-o, calculated using the WavLM-large speaker verification model.

## Citation

```bibtex
@misc{qiang2026mmsonate,
  title={MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning},
  author={Qiang et al. (2026)},
  year={2026},
  note={arXiv:2601.01568}
}
```

- arXiv: 2601.01568