Deepfake Speech Auth Eval

---
name: deepfake-speech-auth-eval
description: Evaluates the robustness of audio-based biometric authentication systems against deepfake speech synthesis attacks. It measures how easily voice cloning models can bypass speaker verification and how effectively anti-spoofing detectors can distinguish genuine from synthetic speech. Use when the user wants to benchmark on AISHELL-3, or asks about evaluating this task. Reports Bypass Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02914
  bibtex_key: hong2026vulnerabilities
  confidence: high
---

# deepfake-speech-auth-eval

> Vulnerabilities of Audio-Based Biometric Authentication Systems Against Deepfake Speech Synthesis — Hong et al. (2026) (arXiv:2601.02914, 2026)

## What this evaluates

Evaluates the robustness of audio-based biometric authentication systems against deepfake speech synthesis attacks. It measures how easily voice cloning models can bypass speaker verification and how effectively anti-spoofing detectors can distinguish genuine from synthetic speech.

## Datasets

- **AISHELL-3** — total ?; splits: train (30), val (10), test (10)

## Metrics

- `Bypass Rate` **(primary)** — range: percent
  - The fraction of deepfake speech attacks that are misclassified as the target genuine speaker by the speaker verification model.
- `Equal Error Rate (EER)` — range: percent
  - The operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Lower values indicate better discrimination between genuine and spoofed speech.

## Input / output format

**Input**: Audio clips (genuine or synthetic/deepfake speech) processed by speaker verification and deepfake detection models.

**Output**: Binary classification decision (genuine vs. spoofed) for detection, and speaker identity match probability/decision for verification.

## Scoring recipe

```python
# Bypass Rate (Speaker Verification)
bypass_rate = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold) / len(gold_labels)

# Equal Error Rate (Deepfake Detection)
# Compute FAR and FRR across thresholds, find threshold where FAR == FRR
eer = find_threshold_where_far_equals_frr(predictions, gold_labels)
```

## Common pitfalls

- Speaker leakage: The test set must consist of entirely unseen speakers to avoid overestimating robustness.
- Threshold sensitivity: The verification threshold is fixed at a 0.01% FAR on the development set, which may not reflect real-world operating conditions.
- Synthesis type variation: Text-to-speech (GPT-SoVITS, Bert-VITS2) and voice conversion (RVC) models exhibit significantly different bypass rates and similarity scores.

## Evidence (verbatim from paper)

> In speaker verification, the bypass rate denotes the fraction of attacks that are misclassified as the target speaker. For deepfake detection, performance is evaluated using the Equal Error Rate (EER) (Reis et al., 2016), defined as the point where the False Acceptance Rate equals the False Rejection Rate.

## Citation

```bibtex
@misc{hong2026vulnerabilities,
  title={Vulnerabilities of Audio-Based Biometric Authentication Systems Against Deepfake Speech Synthesis},
  author={Hong et al. (2026)},
  year={2026},
  note={arXiv:2601.02914}
}
```

- arXiv: 2601.02914