V2c Chem Dubbing Eval

---
name: v2c-chem-dubbing-eval
description: This evaluation probes a model's ability to generate synchronized, emotionally faithful, and speaker-identifiable speech for movie dubbing tasks. It measures audio-visual alignment, spectral similarity, and the preservation of speaker identity and emotional tone against ground-truth recordings. Use when the user wants to benchmark on V2C, Chem, or asks about evaluating this task. Reports LSE-D.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.04054
  bibtex_key: cong2022learningtodub
  confidence: high
---

# v2c-chem-dubbing-eval

> Learning to Dub Movies via Hierarchical Prosody Models — Cong et al. (2022) (arXiv:2212.04054, 2022)

## What this evaluates

This evaluation probes a model's ability to generate synchronized, emotionally faithful, and speaker-identifiable speech for movie dubbing tasks. It measures audio-visual alignment, spectral similarity, and the preservation of speaker identity and emotional tone against ground-truth recordings.

## Datasets

- **V2C** — total 10217; splits: train (-1), val (-1), test (-1)
- **Chem** — total 6640; splits: train (6240), val (200), test (200)

## Metrics

- `LSE-D` **(primary)** — range: other
  - Lip Sync Error Distance; explicitly tests for synchronization between lip motions and speech in unconstrained videos in the wild using an external pre-trained model [12,43].
- `MCD` — range: other
  - Mel Cepstral Distortion; reflects the similarity of mel-spectrograms between generated and ground-truth audio by computing the average distance between cepstral coefficients.
- `Emo. Acc.` — range: percent
  - Emotion accuracy; measures whether the generated speech carries proper emotion compared to ground truth, evaluated using a pre-trained emotion classifier [9].
- `MOS-N` — range: other
  - MOS-Naturalness; subjective metric assessing the naturalness of the generated speech via human listener ratings.

## Input / output format

**Input**: Video clips (frames/lip motion sequences), reference audio for speaker identity, text/subtitles, and scene context/facial expression features (valence/arousal).

**Output**: Generated audio waveform (produced via mel-spectrogram generation followed by HiFiGAN vocoding).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions and gold are aligned lists of generated and reference audio files
    lse_d_scores = [run_lse_d_tool(pred, gold[i]) for i, pred in enumerate(predictions)]
    mcd_scores = [run_mcd_tool(pred, gold[i]) for i, pred in enumerate(predictions)]
    emo_acc = compute_classifier_accuracy(predictions, gold, 'emotion')
    id_acc = compute_classifier_accuracy(predictions, gold, 'identity')
    return {
        'LSE-D': np.mean(lse_d_scores),
        'MCD': np.mean(mcd_scores),
        'Emo. Acc.': emo_acc,
        'Id. Acc.': id_acc
    }
```

## Common pitfalls

- LSE-D/LSE-C and MCD metrics rely on external pre-trained models (e.g., [12,43] for LSE, [24] for MCD) that must be used in strict inference mode; fine-tuning or retraining them on the test set invalidates the evaluation.
- Identity and Emotion accuracy depend on specific pre-trained classifiers from [9]; using different classifier architectures or training data causes distribution mismatch and inaccurate scores.
- Subjective metrics (MOS-N, MOS-S, AQ, AV Sync) require rigorous human evaluation protocols (listener selection, rating scales, randomization) and cannot be computed algorithmically from audio files alone.

## Evidence (verbatim from paper)

> To evaluate the synchronization between the generated speech and the video quantitatively, we adopt Lip Sync Error Distance (LSE-D) and Lip Sync Error Confidence (LSE-C) as our metrics, which can explicitly test for synchronization between lip motions and speech in unconstrained videos in the wild [12,43]. Mel Cepstral Distortion and its variants. MCD [24], MCD-DTW [4] and MCD-DTW-SL [9] are adopted, which reflect the similarity of mel-spectrograms.

## Citation

```bibtex
@misc{cong2022learningtodub,
  title={Learning to Dub Movies via Hierarchical Prosody Models},
  author={Cong et al. (2022)},
  year={2022},
  note={arXiv:2212.04054}
}
```

- arXiv: 2212.04054