Revise Eval

---
name: revise-eval
description: Evaluates generalized speech enhancement and audio-visual speech resynthesis across multiple distortion types (denoising, separation, inpainting, video-to-speech). Measures content intelligibility, audio-video synchronization, perceptual quality, and low-level signal reconstruction. Use when the user wants to benchmark on LRS3, EasyCom, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.11377
  bibtex_key: hsu2022revise
  confidence: high
---

# revise-eval

> ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement — Hsu et al. (2022) (arXiv:2212.11377, 2022)

## What this evaluates

Evaluates generalized speech enhancement and audio-visual speech resynthesis across multiple distortion types (denoising, separation, inpainting, video-to-speech). Measures content intelligibility, audio-video synchronization, perceptual quality, and low-level signal reconstruction.

## Datasets

- **LRS3** — total ?; splits: test (-1)
- **EasyCom** — total ?; splits: train (-1), valid (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: percentage of incorrectly recognized words relative to the reference transcript, computed using a public ASR model.
- `LSE-D` — range: other
  - SyncNet temporal distance: average predicted temporal distance between audio and video frames over the test set.
- `LSE-C` — range: other
  - SyncNet confidence: average confidence score of the audio-video synchronization prediction over the test set.
- `MOS` — range: [1, 5]
  - Mean Opinion Score: subjective quality rating on a 1–5 scale (0.5 increments), averaged across ≥15 raters per sample using CrowdMOS.
- `ESTOI` — range: [0, 1]
  - Extended Short-Time Objective Intelligibility: estimates speech intelligibility based on temporal envelope correlation between reference and enhanced speech.
- `MCD` — range: other
  - Mel Cepstral Distortion: measures the Euclidean distance between mel cepstral coefficients of reference and synthesized speech.

## Input / output format

**Input**: Video frames (head-crop at 25fps) and/or audio features (23-dim Mel FBank, 10ms shift, stacked every 4 frames). For EasyCom, beamformed multi-channel audio or single-channel distant mic audio paired with video.

**Output**: Resynthesized audio waveform (16kHz sampling rate).

## Scoring recipe

```python
def compute_metrics(predictions, references, videos=None):
    wer = compute_wer(predictions, references, asr_model='public_model')
    lse_d, lse_c = compute_syncnet(predictions, videos)
    mos = compute_mos(predictions, num_raters=15, scale=[1,5])
    estoi = compute_estoi(predictions, references)
    mcd = compute_mcd(predictions, references)
    return {'WER': wer, 'LSE-D': lse_d, 'LSE-C': lse_c, 'MOS': mos, 'ESTOI': estoi, 'MCD': mcd}
```

## Common pitfalls

- Using the distant microphone audio as the clean reference for EasyCom instead of the close-talking microphone.
- Assuming LRS3 contains real-world noisy speech; all enhancement tasks are artificially corrupted (mixed with AudioSet noise, other speakers, or zeroed frames).
- MOS evaluation requires exactly 50 randomly sampled test files and ≥15 raters per file via CrowdMOS, not ad-hoc listening.

## Evidence (verbatim from paper)

> For content, we use the WER computed with a speech recognition model to measure the intelligibility quantitatively similar to[[36]], [[37]]. The public model from [[56]] is used, which reports a WER of 5.6% on LRS3 test split and 35.7% on the EasyCom close-talking validation set. For synchronization, following[[21]] we use SyncNet[[9]] metrics, the predicted temporal distance between audio and video (LSE-D) and the prediction’s confidence (LSE-C) are averaged over the entire test set. For quality, we follow the tradition of text-to-speech synthesis evaluation and conduct subjective mean opinion score (MOS) studies with a scale from 1 to 5 and a 0.5 increment. We evaluate 50 randomly sampled files from the test set, where each sample was evaluated by at least 15 raters using the CrowdMOS package[[47]]. Finally, to evaluate reconstruction of low-level details as typically done in speech denoising or source separation studies, we include ESTOI[[51]] and Mel cepstral distortion (MCD)[[28]].

## Citation

```bibtex
@misc{hsu2022revise,
  title={ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement},
  author={Hsu et al. (2022)},
  year={2022},
  note={arXiv:2212.11377}
}
```

- arXiv: 2212.11377