Preset Voice Matching Eval

---
name: preset-voice-matching-eval
description: Evaluates a privacy-regulated speech-to-speech translation framework that replaces voice cloning with matching to pre-consented preset voices. Probes classifier robustness, cross-lingual speech naturalness, and inference efficiency across multilingual scenarios. Use when the user wants to benchmark on RAVDESS, CGDD, CAFE, EmoDB, CREMA-D, or asks about evaluating this task. Reports NISQA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.13153
  bibtex_key: platnick2024preset
  confidence: high
---

# preset-voice-matching-eval

> Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems — Platnick et al. (2024) (arXiv:2407.13153, 2024)

## What this evaluates

Evaluates a privacy-regulated speech-to-speech translation framework that replaces voice cloning with matching to pre-consented preset voices. Probes classifier robustness, cross-lingual speech naturalness, and inference efficiency across multilingual scenarios.

## Datasets

- **RAVDESS** — total ?; splits: train (-1), val (-1), test (-1)
- **CGDD** — total ?; splits: train (-1), val (-1), test (-1)
- **CAFE** — total ?; splits: (unstated)
- **EmoDB** — total ?; splits: (unstated)
- **CREMA-D** — total ?; splits: (unstated)

## Metrics

- `NISQA` **(primary)** — range: [0, 5]
  - Non-intrusive Objective Speech Quality Assessment predicts speech quality scores without a reference signal, typically mapping to a [0, 5] scale similar to Mean Opinion Score (MOS).
- `accuracy and precision` — range: [0, 1]
  - Standard classification metrics computed per classifier (gender, male-emotion, female-emotion) to assess GEMO-Match feature extraction robustness.
- `inference run-time` — range: seconds
  - Average time taken to process 10 identical 15-second utterances per model, measured in seconds.

## Input / output format

**Input**: Audio utterances for classifier evaluation; English text transcriptions and emotive English audio for multilingual S2ST testing; 15-second audio clips with transcriptions for run-time benchmarking.

**Output**: Predicted gender/emotion labels; target-language (French/German) synthesized speech audio; average inference time in seconds.

## Scoring recipe

```python
def compute_metrics(predictions, golds, audio_outputs, run_times):
    # Accuracy & Precision for classifiers
    accuracy = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    true_positives = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 1)
    predicted_positives = sum(1 for p in predictions if p == 1)
    precision = true_positives / max(predicted_positives, 1)
    
    # NISQA for speech naturalness
    nisqa_scores = [nisqa_model.predict(audio) for audio in audio_outputs]
    avg_nisqa = sum(nisqa_scores) / len(nisqa_scores)
    
    # Run-time
    avg_run_time = sum(run_times) / len(run_times)
    return {'accuracy': accuracy, 'precision': precision, 'nisqa': avg_nisqa, 'run_time': avg_run_time}
```

## Common pitfalls

- The multilingual experiment bypasses the Speech-to-Text (STT) stage by feeding ground-truth English text directly to the translation model, which isolates GEMO-Match but ignores STT error propagation.
- NISQA provides objective quality predictions that may not fully capture perceptual differences in cross-lingual voice matching or emotional fidelity.
- Run-time comparisons use fixed 15-second utterances and may not reflect latency scaling for variable-length or highly dynamic speech.

## Evidence (verbatim from paper)

> We measured speaker naturalness using the standard metric Non-intrusive Objective Speech Quality Assessment (NISQA) [[42], [43]].

## Citation

```bibtex
@misc{platnick2024preset,
  title={Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems},
  author={Platnick et al. (2024)},
  year={2024},
  note={arXiv:2407.13153}
}
```

- arXiv: 2407.13153