Echofake Eval

---
name: echofake-eval
description: Probes the robustness of speech deepfake detection models against physical replay attacks and cross-dataset generalization. It evaluates how well anti-spoofing systems distinguish between genuine speech, zero-shot TTS-generated deepfakes, and their physically replayed counterparts under realistic acoustic conditions. Use when the user wants to benchmark on EchoFake, or asks about evaluating this task. Reports EER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19414
  bibtex_key: zhang2025echofake
  confidence: high
---

# echofake-eval

> EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection — Zhang et al. (2025) (arXiv:2510.19414, 2025)

## What this evaluates

Probes the robustness of speech deepfake detection models against physical replay attacks and cross-dataset generalization. It evaluates how well anti-spoofing systems distinguish between genuine speech, zero-shot TTS-generated deepfakes, and their physically replayed counterparts under realistic acoustic conditions.

## Datasets

- **EchoFake** — total ?; splits: train (-1), open (-1); repo https://github.com/EchoFake/EchoFake

## Metrics

- `EER` **(primary)** — range: percent
  - Equal Error Rate: the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Reported as a percentage.
- `F1-score` — range: percent
  - Harmonic mean of precision and recall. Evaluated for four-class classification (B/RB/F/RF) and averaged across classes.

## Input / output format

**Input**: Raw audio recordings (speech samples) with ground-truth labels.

**Output**: Binary label (spoof/bonafide) or four-class label (B/RB/F/RF).

## Scoring recipe

```python
def compute_eer(pred_scores, true_labels):
    # pred_scores: probability of spoof, true_labels: 1=spoof, 0=bonafide
    thresholds = np.linspace(0, 1, 1000)
    best_diff = float('inf')
    eer = 0.0
    for t in thresholds:
        far = sum(s >= t and l == 0 for s, l in zip(pred_scores, true_labels)) / max(sum(l == 0 for l in true_labels), 1)
        frr = sum(s < t and l == 1 for s, l in zip(pred_scores, true_labels)) / max(sum(l == 1 for l in true_labels), 1)
        if abs(far - frr) < best_diff:
            best_diff = abs(far - frr)
            eer = (far + frr) / 2
    return eer
```

## Common pitfalls

- Treating Replayed Bona Fide (RB) samples as genuine in binary evaluation; the protocol explicitly treats RB as spoofed to test replay robustness.
- Ignoring the distinction between closed-set and open-set conditions; performance drops sharply in open-set scenarios where replayed samples are the primary error source.
- Assuming spectral artifacts are the sole detection cue; high-fidelity replay attacks mask these artifacts, requiring models to learn channel distortion features.

## Evidence (verbatim from paper)

> We evaluate system performance using two standard metrics: F1-score for four-class classification (B/RB/F/RF) and equal error rate (EER) for binary spoofing detection (spoof/bonafide). The F1-score provides balanced accuracy measurement, while EER optimizes the trade-off between false acceptance and rejection rates.

## Citation

```bibtex
@misc{zhang2025echofake,
  title={EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.19414}
}
```

- arXiv: 2510.19414