Unispeaker Mvc Eval

---
name: unispeaker-mvc-eval
description: Evaluates multimodal voice generation and conversion capabilities across face-driven, text-driven, and attribute-based tasks. Probes the model's ability to align facial, textual, and attribute descriptions with target speech while preserving speaker identity, content clarity, and naturalness. Use when the user wants to benchmark on LRS3, or asks about evaluating this task. Reports MOS-Match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.06394
  bibtex_key: sheng2025unispeaker
  confidence: high
---

# unispeaker-mvc-eval

> Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation — Sheng et al. (2025) (arXiv:2501.06394, 2025)

## What this evaluates

Evaluates multimodal voice generation and conversion capabilities across face-driven, text-driven, and attribute-based tasks. Probes the model's ability to align facial, textual, and attribute descriptions with target speech while preserving speaker identity, content clarity, and naturalness.

## Datasets

- **LRS3** — total ?; splits: train (-1), test (-1)

## Metrics

- `MOS-Match` **(primary)** — range: other
  - Mean Opinion Score for voice suitability, measuring how well the generated voice matches the target modality (face, text, or attributes). Evaluated via human listening tests on a standard scale.
- `MOS-Nat` — range: other
  - Mean Opinion Score for naturalness, assessing the perceptual naturalness of the generated speech without reference to the target modality.
- `SST` — range: other
  - Speaker Similarity Score, quantifying the acoustic similarity between the generated voice and the target speaker's identity, typically computed via cosine similarity of speaker embeddings.
- `SSD` — range: other
  - Speech Distortion Score, measuring the level of acoustic distortion or noise introduced during generation, often using PESQ or STOI.
- `SSC` — range: other
  - Speech Clarity Score, evaluating the intelligibility and overall quality of the generated speech, typically assessed via human listening tests.

## Input / output format

**Input**: Multimodal conditioning inputs (face images, textual descriptions, or voice attributes) combined with content text for TTS or reference speech for VC.

**Output**: Generated audio waveform/speech.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions: generated audio files
    # gold: target audio / modality descriptions
    mos_match = human_listen_test(predictions, gold['modality'], aspect='suitability')
    mos_nat = human_listen_test(predictions, aspect='naturalness')
    sst = cosine_similarity(speaker_embeddings(predictions), speaker_embeddings(gold['target_speaker']))
    ssd = pesq_or_stoi(predictions, gold['target_speaker'])
    ssc = human_listen_test(predictions, aspect='clarity')
    return {'MOS-Match': mos_match, 'MOS-Nat': mos_nat, 'SST': sst, 'SSD': ssd, 'SSC': ssc}
```

## Common pitfalls

- MOS-Match and MOS-Nat are distinct subjective metrics; confusing them leads to incorrect suitability vs. quality reporting.
- ABX test results are highly dependent on listener demographics and audio playback conditions, requiring strict standardization.
- Performance scales non-linearly with multimodal dataset size; evaluating on small subsets without reporting scale may misrepresent capability.

## Evidence (verbatim from paper)

> Across five tasks, UniSpeaker outperformed previous approaches on all three metrics, except for MOS-Match in the AVE task.

## Citation

```bibtex
@misc{sheng2025unispeaker,
  title={Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation},
  author={Sheng et al. (2025)},
  year={2025},
  note={arXiv:2501.06394}
}
```

- arXiv: 2501.06394