Lip2wav Eval

---
name: lip2wav-eval
description: Evaluates a model's ability to synthesize natural, speaker-specific speech from unconstrained lip movements in large-vocabulary settings. It measures how well the model captures individual speaking styles and contextual cues from video frames. Use when the user wants to benchmark on Lip2Wav, GRID, TCD-TIMIT lip speaker corpus, or asks about evaluating this task. Reports mel reconstruction loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.08209
  bibtex_key: prajwal2020lip2wav
  confidence: high
---

# lip2wav-eval

> Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis — Prajwal et al. (2020) (arXiv:2005.08209, 2020)

## What this evaluates

Evaluates a model's ability to synthesize natural, speaker-specific speech from unconstrained lip movements in large-vocabulary settings. It measures how well the model captures individual speaking styles and contextual cues from video frames.

## Datasets

- **Lip2Wav** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/Rudrabha/Lip2Wav
- **GRID** — total ?; splits: train (-1), val (-1), test (-1)
- **TCD-TIMIT lip speaker corpus** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `mel reconstruction loss` **(primary)** — range: other
  - Computed as the mean squared error (MSE) between the predicted and ground-truth mel-spectrogram frames: L = (1/N) Σ (y_pred - y_true)². Lower values indicate better spectral reconstruction fidelity.

## Input / output format

**Input**: Contiguous 3-second sequence of video frames (face crops resized to 48×48) representing lip movements.

**Output**: Generated speech waveform (produced via Griffin-Lim algorithm from predicted mel-spectrograms).

## Scoring recipe

```python
def compute_mel_reconstruction_loss(pred_mels, gt_mels):
    # pred_mels and gt_mels: tensors of shape (T, F)
    loss = torch.mean((pred_mels - gt_mels) ** 2)
    return loss.item()
```

## Common pitfalls

- Video-level splitting is mandatory to prevent data leakage between train and test sets.
- Griffin-Lim vocoder is explicitly used over neural vocoders due to lower accuracy of generated mel-spectrograms.
- Sliding window inference requires overlap to adjust for boundary effects, which must be accounted for during evaluation.

## Evidence (verbatim from paper)

> train until the mel reconstruction loss plateaus for at least 30K iterations. ... The model with the best performance on the validation set is chosen for testing and evaluation.

## Citation

```bibtex
@misc{prajwal2020lip2wav,
  title={Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis},
  author={Prajwal et al. (2020)},
  year={2020},
  note={arXiv:2005.08209}
}
```

- arXiv: 2005.08209