Cloned Voice Detection Eval

---
name: cloned-voice-detection-eval
description: Evaluates the ability of audio classifiers to distinguish between real human speech and AI-generated cloned voices across single and multi-speaker scenarios. It also probes robustness against adversarial audio laundering, including additive noise and AAC transcoding, to assess how well different feature representations (learned, spectral, perceptual) generalize and resist degradation. Use when the user wants to benchmark on ElevenLabs (EL), Uberduck (UD), WaveFake (WF), TIMIT-ElevenLabs, or asks about evaluating this task. Reports EER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.07683
  bibtex_key: barrington2023clonedvoice
  confidence: high
---

# cloned-voice-detection-eval

> Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features — Barrington et al. (2023) (arXiv:2307.07683, 2023)

## What this evaluates

Evaluates the ability of audio classifiers to distinguish between real human speech and AI-generated cloned voices across single and multi-speaker scenarios. It also probes robustness against adversarial audio laundering, including additive noise and AAC transcoding, to assess how well different feature representations (learned, spectral, perceptual) generalize and resist degradation.

## Datasets

- **ElevenLabs (EL)** — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
- **Uberduck (UD)** — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
- **WaveFake (WF)** — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
- **TIMIT-ElevenLabs** — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection

## Metrics

- `EER (%)` **(primary)** — range: percent
  - Equal Error Rate is the point on the receiver operating characteristic (ROC) curve where the false acceptance rate (FAR, incorrectly classifying a synthetic voice as real) equals the false rejection rate (FRR, incorrectly classifying a real voice as synthetic).
- `Synthetic Accuracy (%)` — range: percent
  - Percentage of synthetic audio samples correctly classified as synthetic.
- `Real Accuracy (%)` — range: percent
  - Percentage of real audio samples correctly classified as real.

## Input / output format

**Input**: Raw audio samples labeled as either real human speech or AI-generated synthetic speech (optionally subjected to adversarial laundering like additive Gaussian noise or AAC transcoding).

**Output**: Binary classification label (real vs. synthetic) or multi-class label (real vs. specific synthesis engine). Accuracy is computed separately for synthetic and real classes, with EER reported for single-class classifiers.

## Scoring recipe

```python
def compute_metrics(predictions, synth_mask, real_mask):
    synth_acc = sum(predictions[synth_mask]) / synth_mask.sum()
    real_acc = sum(1 - predictions[real_mask]) / real_mask.sum()
    
    # EER calculation
    thresholds = np.linspace(0, 1, 1000)
    eer = None
    for t in thresholds:
        far = sum(predictions[synth_mask] > t) / synth_mask.sum()
        frr = sum(predictions[real_mask] <= t) / real_mask.sum()
        if abs(far - frr) < 1e-3:
            eer = (far + frr) / 2
            break
    return synth_acc, real_acc, eer
```

## Common pitfalls

- Laundering (additive noise + AAC transcoding) significantly degrades spectral and perceptual features more than learned features, skewing comparisons if laundering is not controlled.
- Classifiers trained on a single dataset or synthesis engine do not generalize well to others; multi-dataset training is required for robust detection.
- EER is only reported for single-class classifiers; multi-class setups report separate per-class accuracies instead of EER.

## Evidence (verbatim from paper)

> The far-right columns report the equal error rate (the EER is the point on the receiver operating curve (ROC) where the false acceptance rate (incorrectly classifying a synthetic voice as real) and false rejection rate (incorrectly classifying a real voice as synthetic) are equal).

## Citation

```bibtex
@misc{barrington2023clonedvoice,
  title={Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features},
  author={Barrington et al. (2023)},
  year={2023},
  note={arXiv:2307.07683}
}
```

- arXiv: 2307.07683