Speechfake Eval

---
name: speechfake-eval
description: This benchmark evaluates speech deepfake detection models on their ability to generalize across diverse synthesis methods (TTS, voice conversion, neural vocoders), multiple languages, and unseen speaker identities. It probes whether models learn inherent spoofing artifacts or merely memorize specific speaker voices or generation techniques. Use when the user wants to benchmark on SpeechFake, or asks about evaluating this task. Reports EER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.21463
  bibtex_key: huang2025speechfake
  confidence: high
---

# speechfake-eval

> SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods — Wen Huang et al. (2025) (arXiv:2507.21463, 2025)

## What this evaluates

This benchmark evaluates speech deepfake detection models on their ability to generalize across diverse synthesis methods (TTS, voice conversion, neural vocoders), multiple languages, and unseen speaker identities. It probes whether models learn inherent spoofing artifacts or merely memorize specific speaker voices or generation techniques.

## Datasets

- **SpeechFake** — total ?; splits: train (-1), test (-1); repo https://github.com/YMLLG/SpeechFake

## Metrics

- `EER` **(primary)** — range: percent
  - Equal Error Rate (EER) is the operating point on the detection error tradeoff (DET) curve where the False Positive Rate (FPR) equals the False Negative Rate (FNR). It is reported as a percentage.

## Input / output format

**Input**: Raw audio recordings of speech, labeled as real or fake, with metadata indicating the generation method, language, and speaker identity.

**Output**: Binary classification prediction (real/fake) or a continuous spoofing score used to compute the EER across decision thresholds.

## Scoring recipe

```python
def compute_eer(scores, labels):
    # scores: predicted spoofing scores, labels: 1=fake, 0=real
    fpr, fnr, _ = compute_det_curve(scores, labels)
    idx = np.argmin(np.abs(fpr - fnr))
    eer = fpr[idx]
    return eer * 100  # Return as percentage
```

## Common pitfalls

- Models may overfit to specific speaker identities rather than learning general deepfake artifacts, leading to inflated performance on seen speakers but poor generalization to unseen ones.
- Training on a single generation method (e.g., TTS) causes significant performance drops when tested on unseen methods (e.g., VC or NV), highlighting a lack of cross-generator generalization.
- Evaluating only on in-domain test sets (e.g., ASVspoof2019-LA when trained on it) masks real-world generalization failures and inflates reported accuracy.

## Evidence (verbatim from paper)

> For evaluation, we use the Equal Error Rate (EER) as the metric, following previous work (Yamagishi et al., 2021; Du et al., 2024a).

## Citation

```bibtex
@misc{huang2025speechfake,
  title={SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods},
  author={Wen Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.21463}
}
```

- arXiv: 2507.21463