Lip To Speech Eval

---
name: lip-to-speech-eval
description: Evaluates a model's ability to synthesize high-fidelity, intelligible speech directly from visual lip movements. It probes perceptual audio quality, content accuracy, and speaker identity preservation in a cross-dataset generalization setting. Use when the user wants to benchmark on LRS3-TED, LRS2-BBC, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11477
  bibtex_key: liang2026sldl2s
  confidence: high
---

# lip-to-speech-eval

> SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis — Liang et al. (2026) (arXiv:2602.11477, 2026)

## What this evaluates

Evaluates a model's ability to synthesize high-fidelity, intelligible speech directly from visual lip movements. It probes perceptual audio quality, content accuracy, and speaker identity preservation in a cross-dataset generalization setting.

## Datasets

- **LRS3-TED** — total 150000; splits: train (-1), val (-1), test (-1)
- **LRS2-BBC** — total 144000; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated using the AUTO-AVSR model, measuring the percentage of incorrectly recognized words relative to the ground truth transcript.
- `UTMOS` — range: [1, 5]
  - Non-intrusive reference-free model predicting a mean opinion score (MOS) on a 1-to-5 scale to correlate with human quality judgments.
- `SCOREQ` — range: [1, 5]
  - Reference-free model predicting a MOS on a 1-to-5 scale for perceptual quality assessment.
- `D-BERT` — range: [-1, 1]
  - Cosine similarity between SSL embeddings of synthesized and target speech, measuring semantic similarity.
- `SECS` — range: [-1, 1]
  - Cosine similarity between speaker embeddings extracted from synthesized and target speech using Resemblyzer, measuring speaker identity preservation.
- `MOS` — range: [1, 5]
  - Human-rated Mean Opinion Score on a 1-to-5 scale assessing naturalness, intelligibility, and speaker similarity.

## Input / output format

**Input**: Mouth region-of-interest video frames (88x88 grayscale) processed into 1024-dimensional visual features via AV-Hubert Large, plus a 256-dimensional speaker identity embedding from a reference utterance.

**Output**: Continuous latent vectors of the X-Codec-hubert neural audio codec, which are subsequently decoded into 16 kHz audio waveforms.

## Scoring recipe

```python
def compute_metrics(synthesized_audio, target_audio, target_transcript):
    wer = auto_avsr.transcribe(synthesized_audio) / len(target_transcript)
    d_bert = cosine_similarity(hubert_embed(target_audio), hubert_embed(synthesized_audio))
    secs = cosine_similarity(resemblyzer_embed(target_audio), resemblizer_embed(synthesized_audio))
    utmos = utmos_model.predict(synthesized_audio)
    scoreq = scoreq_model.predict(synthesized_audio)
    return {'WER': wer, 'D-BERT': d_bert, 'SECS': secs, 'UTMOS': utmos, 'SCOREQ': scoreq}
```

## Common pitfalls

- Using standard ASR instead of AUTO-AVSR for WER calculation, which ignores visual cues and inflates error rates.
- Confusing reference-free metrics (UTMOS, SCOREQ) with intrusive metrics; these require no target audio and predict MOS directly.
- Subjective MOS evaluation requires exactly 15 participants rating 30 samples per criterion; deviating from this protocol invalidates comparability.

## Evidence (verbatim from paper)

> We perform a comprehensive evaluation of our proposed SLD-L2S framework using a suite of objective and subjective metrics. The evaluation is designed to assess three critical aspects of the synthesized speech: perceptual quality, content intelligibility, and speaker similarity. For objective evaluation, we employ the following metrics: Quality: We assess perceptual quality using UTMOS and SCOREQ. Both are non-intrusive, reference-free models that predict a mean opinion score (MOS) on a 1-to-5 scale, designed to correlate with human quality judgments. Intelligibility: We measure intelligibility from two perspectives. First, the word error rate (WER) is calculated using the AUTO-AVSR. Second, we utilize SpeechBERTScore (D-BERT)... Speaker Similarity: We quantify speaker identity preservation using speaker embedding cosine similarity (SECS).

## Citation

```bibtex
@misc{liang2026sldl2s,
  title={SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis},
  author={Liang et al. (2026)},
  year={2026},
  note={arXiv:2602.11477}
}
```

- arXiv: 2602.11477