Fake Voice Detection Eval

---
name: fake-voice-detection-eval
description: Evaluates the robustness and cross-domain generalization of fake voice detectors against 17 state-of-the-art generators (TTS, voice conversion, audio reconstruction) using a one-to-one protocol. It quantifies both generator quality and detector effectiveness through composite scores to expose method-specific vulnerabilities that aggregated benchmarks typically mask. Use when the user wants to benchmark on LibriSpeech (test-clean), ASVspoof-21LA, ASVspoof-21DF, ASVspoof-5, Fake or Real (FoR), CFAD, or asks about evaluating this task. Reports EER, minDCF.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06544
  bibtex_key: mao2025benchmarkingfakevoice
  confidence: high
---

# fake-voice-detection-eval

> Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race — Mao et al. (2025) (arXiv:2510.06544, 2025)

## What this evaluates

Evaluates the robustness and cross-domain generalization of fake voice detectors against 17 state-of-the-art generators (TTS, voice conversion, audio reconstruction) using a one-to-one protocol. It quantifies both generator quality and detector effectiveness through composite scores to expose method-specific vulnerabilities that aggregated benchmarks typically mask.

## Datasets

- **LibriSpeech (test-clean)** — total ?; splits: test (-1)
- **ASVspoof-21LA** — total ?; splits: test (-1)
- **ASVspoof-21DF** — total ?; splits: test (-1)
- **ASVspoof-5** — total ?; splits: test (-1)
- **Fake or Real (FoR)** — total ?; splits: test (-1)
- **CFAD** — total ?; splits: test (-1)

## Metrics

- `EER` **(primary)** — range: percent
  - Equal Error Rate: the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Lower values indicate better discrimination.
- `minDCF` **(primary)** — range: [0, 1]
  - Minimum normalized Detection Cost Function: the lowest weighted cost over all possible decision thresholds, balancing miss and false alarm costs under target conditions.
- `C_llr` — range: [0, ∞)
  - Log Likelihood Ratio Cost: measures the calibration quality of detection scores by comparing the log-likelihood ratio of real vs. fake hypotheses against ground truth.
- `DNSMOS` — range: [0, 5]
  - Deep Learning Speech Mean Opinion Score: a no-reference neural network prediction of subjective audio perceptual quality.
- `SIM` — range: [0, 1]
  - Speaker Similarity: cosine similarity between embeddings of original and generated voice pairs extracted via a WavLM-based model.
- `Generator Overall Score` — range: (0, 1]
  - Weighted sum of normalized metrics: F = Σ α_i * M_i_norm, prioritizing perceptual naturalness (DNSMOS), identity fidelity (SIM), and intelligibility (WER).
- `Detector Overall Score` — range: (0, 1]
  - Composite score balancing empirical performance and complexity: S_i = α P_i + (1 - α) Ĉ_i, where P_i is weighted average normalized EER/minDCF/C_llr and Ĉ_i is normalized parameter count.

## Input / output format

**Input**: Detectors receive 24kHz audio waveforms (generated fake voices or real references). Generators receive text transcriptions and reference audio waveforms.

**Output**: Detectors output detection confidence scores or binary real/fake labels. Generators output synthesized audio waveforms.

## Scoring recipe

```python
def compute_detector_metrics(scores, labels):
    far, frr, eer = compute_eer_curve(scores, labels)
    min_dcf = compute_mindcf(scores, labels, p_target=0.01, p_false_alarm=0.01)
    c_llr = compute_llr_cost(scores, labels)
    return {"EER": eer, "minDCF": min_dcf, "C_llr": c_llr}

def compute_composite_scores(detector_scores, generator_scores, alpha=0.8):
    P_i = weighted_avg(detector_scores, weights=generator_quality_scores)
    C_i = normalize(generator_params)
    return alpha * P_i + (1 - alpha) * C_i
```

## Common pitfalls

- Aggregating results across all generators and detectors masks method-specific vulnerabilities; the protocol strictly requires a one-to-one evaluation matrix.
- Relying solely on MOS or single metrics like WER/DNSMOS without considering speaker similarity (SIM) or efficiency (RTF/Params) yields incomplete quality assessments.
- Assuming white-box access to detectors contradicts the threat model; the evaluation assumes black-box adversaries aiming for broad evasion.

## Evidence (verbatim from paper)

> We adopt standard metrics from the ASVspoof Challenge: Equal Error Rate (EER) for discrimination, and minimum normalized Detection Cost Function (minDCF) and Log Likelihood Ratio Cost $(C_{llr})$ for a combination of discrimination and score calibration. To synthesize these results, we propose a Detector Overall Score $(S_i)$ that balances empirical performance against model complexity.

## Citation

```bibtex
@misc{mao2025benchmarkingfakevoice,
  title={Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2510.06544}
}
```

- arXiv: 2510.06544