Replaydf Eval

---
name: replaydf-eval
description: Evaluates the robustness of audio deepfake detection models against replay attacks where deepfake audio is played back and re-recorded through real-world hardware, introducing acoustic distortions and room impulse responses. Use when the user wants to benchmark on ReplayDF, or asks about evaluating this task. Reports EER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14862
  bibtex_key: muller2025replay
  confidence: high
---

# replaydf-eval

> Replay Attacks Against Audio Deepfake Detection — Müller et al. (2025) (arXiv:2505.14862, 2025)

## What this evaluates

Evaluates the robustness of audio deepfake detection models against replay attacks where deepfake audio is played back and re-recorded through real-world hardware, introducing acoustic distortions and room impulse responses.

## Datasets

- **ReplayDF** — total ?; splits: test (-1)

## Metrics

- `EER (%)` **(primary)** — range: percent
  - Equal Error Rate: the operating point where the false acceptance rate (FAR) equals the false rejection rate (FRR).
- `Accuracy (%)` — range: percent
  - Percentage of correctly classified instances (bona fide vs. spoofed).

## Input / output format

**Input**: Raw audio files containing either bona fide speech or TTS-generated spoofed speech.

**Output**: Binary classification label indicating whether the audio is bona fide or spoofed.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    accuracy = np.mean(predictions == labels) * 100
    # EER is the threshold where FAR = FRR
    thresholds = np.linspace(0, 1, 1000)
    best_eer = 1.0
    for t in thresholds:
        far = np.mean((predictions >= t) & (labels == 0))
        frr = np.mean((predictions < t) & (labels == 1))
        if abs(far - frr) < best_eer:
            best_eer = (far + frr) / 2
    return accuracy, best_eer
```

## Common pitfalls

- Models trained on datasets with differing noise patterns between genuine and spoofed audio learn shortcuts equating poor quality with spoofing, leading to misclassification of replayed audio.
- Performance degradation from replay attacks is not merely due to added noise but rather the loss of distinctive deepfake-specific characteristics caused by the physical air-gap.
- RIR augmentation improves resilience but does not fully mitigate the vulnerability, often leaving EER significantly higher than baseline.

## Evidence (verbatim from paper)

> We evaluate ReplayDF across multiple scenarios to assess the impact of replay attacks on audio deepfake detection models. We define two key data partitions: first, ReplayDF (set  $R$ ): All audio files generated as in Section 3, containing equal amounts of bona fide and spoofed instances. Second, the Baseline dataset (set  $O$ ): the original input instances from MLAAD v5 and M-AILABS, serving as a comparative baseline against ReplayDF. Table 2: Performance of Open-Source models with publicly available checkpoints in mean accuracy and EER over ReplayDF, as well as the original audio files (Baseline).

## Citation

```bibtex
@misc{muller2025replay,
  title={Replay Attacks Against Audio Deepfake Detection},
  author={Müller et al. (2025)},
  year={2025},
  note={arXiv:2505.14862}
}
```

- arXiv: 2505.14862