V2c Animation Eval

---
name: v2c-animation-eval
description: Evaluates visually-driven voice cloning by measuring speech quality, temporal alignment, length consistency, speaker identity preservation, and emotion transfer accuracy against ground-truth audio. Use when the user wants to benchmark on V2C-Animation, or asks about evaluating this task. Reports MCD-DTW-SL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.12890
  bibtex_key: chen2021v2c
  confidence: high
---

# v2c-animation-eval

> V2C: Visual Voice Cloning — Chen et al. (2021) (arXiv:2111.12890, 2021)

## What this evaluates

Evaluates visually-driven voice cloning by measuring speech quality, temporal alignment, length consistency, speaker identity preservation, and emotion transfer accuracy against ground-truth audio.

## Datasets

- **V2C-Animation** — total 10217; splits: test (-1)

## Metrics

- `MCD-DTW-SL` **(primary)** — range: other
  - MCD-DTW weighted by Speech Length. Computes DTW minimum cumulative distance gamma_M,N between generated and ground-truth MFCC sequences, then scales by eta/R where eta = max(M,N)/min(M,N) and R is the number of DTW steps.
- `MCD` — range: other
  - Mel Cepstral Distortion. Sums Euclidean distance over first K MFCC values across T frames: (1/T) * sum(sqrt(sum((c_tk - c'_tk)^2))).
- `Id. Acc.` — range: percent
  - Speaker identity accuracy. Computes cosine similarity between generated speech embedding and pre-computed speaker centroids (from GE2E encoder), then calculates percentage of correct classifications.
- `Emo. Acc.` — range: percent
  - Emotion accuracy. Same procedure as Id. Acc. but uses emotion centroids derived from reference video labels.
- `MOS-naturalness` — range: [1, 5]
  - Mean Opinion Score for naturalness. Listeners rate generated audio on an ACR scale from 1 (Bad) to 5 (Excellent) in 0.5 increments.
- `MOS-similarity` — range: [1, 5]
  - Mean Opinion Score for similarity. Listeners rate how well generated speech aligns with desired voice/prosody compared to ground truth on a 1-5 ACR scale.

## Input / output format

**Input**: Reference audio clip, reference video clip, and target text transcript.

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
def compute_mcd_dtw_sl(c, c_gt):
    gamma = dtw_minimum_distance(c, c_gt)
    R = max(len(c), len(c_gt))
    eta = max(len(c), len(c_gt)) / min(len(c), len(c_gt))
    return (eta / R) * gamma

def compute_accuracy(audio, gold_label, centroids):
    emb = ge2e_encoder(audio)
    pred = argmax([cosine_similarity(emb, cent) for cent in centroids])
    return (pred == gold_label)

# Aggregate over test set
mcd_dtw_sl_scores = [compute_mcd_dtw_sl(gen, gt) for gen, gt in test_set]
acc_scores = [compute_accuracy(gen, label, centroids) for gen, label in test_set]
```

## Common pitfalls

- Standard MCD requires equal-length inputs; zero-padding shorter speech can artificially inflate distortion if mismatches occur early in the sequence.
- Plain MCD-DTW ignores length mismatch, allowing artificially low scores if any segment aligns, which is why the length-weighted variant (MCD-DTW-SL) is necessary.
- Identity and emotion accuracy scores are capped below 100% even on ground truth due to imperfections in the pre-trained GE2E classifier and emotion encoder.

## Evidence (verbatim from paper)

> To assess the quality of generated speech, we use Mel Cepstral Distortion (MCD) metric... To alleviate the above issues, we propose a MCD-DTW weighted by Speech Length (MCD-DTW-SL)... we use a Mean Opinion Score (MOS) evaluation approach... To evaluate whether the generated speech carries proper speaker identity and emotion, we propose an identity accuracy and an emotion accuracy, respectively.

## Citation

```bibtex
@misc{chen2021v2c,
  title={V2C: Visual Voice Cloning},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2111.12890}
}
```

- arXiv: 2111.12890