Hyface Vc Eval

---
name: hyface-vc-eval
description: This protocol evaluates face-based voice conversion models by measuring how well synthesized audio matches the target speaker's identity and pitch characteristics using only facial images as input. It probes cross-modal alignment, speaker homogeneity, diversity, and explicit fundamental frequency (F0) estimation accuracy. Use when the user wants to benchmark on LRS3, or asks about evaluating this task. Reports Pitch deviation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.09802
  bibtex_key: lee2024hearyourface
  confidence: high
---

# hyface-vc-eval

> Hear Your Face: Face-based voice conversion with F0 estimation — Jaejun Lee et al. (2024) (arXiv:2408.09802, 2024)

## What this evaluates

This protocol evaluates face-based voice conversion models by measuring how well synthesized audio matches the target speaker's identity and pitch characteristics using only facial images as input. It probes cross-modal alignment, speaker homogeneity, diversity, and explicit fundamental frequency (F0) estimation accuracy.

## Datasets

- **LRS3** — total 5502; splits: trainval (-1), test (-1)

## Metrics

- `Pitch deviation` **(primary)** — range: Hz
  - Mean absolute difference between the frame-wise F0 of the synthesized audio and the average F0 of the ground-truth target speaker audio.
- `Consistency(obj)` — range: [-1, 1]
  - Cosine similarity between speaker embeddings of the synthesized audio and the ground-truth audio from the same target speaker.
- `Homogeneity` — range: [-1, 1]
  - Cosine similarity of speaker embeddings in synthesized audio generated from different facial images of the same speaker.
- `Diversity` — range: [-1, 1]
  - Cosine similarity of speaker embeddings in synthesized audio generated from different target speakers.
- `Consistency(sub)` — range: [1, 5]
  - 5-point Mean Opinion Score (MOS) assessing alignment between synthesized audio and corresponding facial images.
- `Naturalness` — range: [1, 5]
  - 5-point MOS assessing sound quality of synthesized audio.
- `ABX test` — range: percent
  - Subjective preference test where participants choose which of two synthesized audio samples better matches a given face image.

## Input / output format

**Input**: Source audio clip and a set of frontal facial images (224×224, 25 fps) of the target speaker.

**Output**: Synthesized speech audio matching the target speaker's voice characteristics.

## Scoring recipe

```python
def compute_metrics(pred_audio, gold_audio):
    pred_f0 = extract_f0(pred_audio)
    gold_f0_avg = np.mean(extract_f0(gold_audio))
    pitch_dev = np.mean(np.abs(pred_f0 - gold_f0_avg))
    pred_emb = Resemblyzer.encode(pred_audio)
    gold_emb = Resemblyzer.encode(gold_audio)
    consistency_obj = cosine_similarity(pred_emb, gold_emb)
    return {'pitch_deviation': pitch_dev, 'consistency_obj': consistency_obj}
```

## Common pitfalls

- Homogeneity and Diversity measure opposite goals (similarity within same speaker vs. distinctiveness across speakers); confusing them inverts the desired score direction.
- Consistency(rnd) is a control metric, not a performance target; high scores here indicate the model fails to capture the target speaker's identity.
- Ground-truth (GT) audio cannot be evaluated for heterogeneous gender pairings (HTG), so GT scores are only reported for homogeneous sets (HMG).

## Evidence (verbatim from paper)

> Following Sheng et al. [9] and other conventional VC studies, for objective evaluation, we assess the homogeneity, diversity, and objective consistency. For subjective evaluation, we examine subjective consistency, naturalness, and ABX tests. Furthermore, we propose a new evaluation metric: pitch deviation.

## Citation

```bibtex
@misc{lee2024hearyourface,
  title={Hear Your Face: Face-based voice conversion with F0 estimation},
  author={Jaejun Lee et al. (2024)},
  year={2024},
  note={arXiv:2408.09802}
}
```

- arXiv: 2408.09802