Snr Bench Eval

---
name: snr-bench-eval
description: Evaluates the robustness of audio deepfake detection models under varying signal-to-noise ratios (SNRs) by testing binary (real vs. spoof) and four-class (real+clean, real+noisy, spoof+clean, spoof+noisy) classification tasks on ASVspoof 2021 utterances augmented with MS-SNSD ambient noise. Use when the user wants to benchmark on ASVspoof 2021 (DF), or asks about evaluating this task. Reports EER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.13744
  bibtex_key: sen2025toward
  confidence: high
---

# snr-bench-eval

> Toward Noise-Aware Audio Deepfake Detection: Survey, SNR-Benchmarks, and Practical Recipes — Sen et al. (2025) (arXiv:2512.13744, 2025)

## What this evaluates

Evaluates the robustness of audio deepfake detection models under varying signal-to-noise ratios (SNRs) by testing binary (real vs. spoof) and four-class (real+clean, real+noisy, spoof+clean, spoof+noisy) classification tasks on ASVspoof 2021 utterances augmented with MS-SNSD ambient noise.

## Datasets

- **ASVspoof 2021 (DF)** — total ?; splits: test (-1), development (-1)

## Metrics

- `accuracy` — range: [0, 1]
  - Proportion of correctly classified utterances under a fixed decision threshold (typically 0.5).
- `ROC-AUC` — range: [0, 1]
  - Area under the receiver operating characteristic curve; quantifies the probability that a randomly sampled spoof trial receives a lower score than a randomly sampled bonafide trial.
- `EER` **(primary)** — range: [0, 1]
  - The point on the ROC curve where the false acceptance rate equals the false rejection rate.

## Input / output format

**Input**: 2-second audio waveforms of speech, optionally mixed with ambient noise at a specific SNR from the grid [35, 30, 25, 20, 15, 10, 5, 0, -5] dB.

**Output**: Classification logits (binary or four-class) produced by a LayerNorm → Dropout(0.1) → Linear prediction head.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    # predictions: spoof scores (higher = more likely spoof)
    # labels: 1 for spoof, 0 for real
    roc_auc = roc_auc_score(labels, predictions)
    fpr, tpr, _ = roc_curve(labels, predictions)
    eer = fpr[argmin(abs((1 - tpr) - fpr))]
    preds = (predictions >= 0.5).astype(int)
    accuracy = mean(preds == labels)
    return {'accuracy': accuracy, 'roc_auc': roc_auc, 'eer': eer}
```

## Common pitfalls

- Treating accuracy as a reliable robustness metric, despite the paper explicitly noting it is sensitive to class imbalance and noise-induced score distribution shifts.
- Ignoring the four-class task formulation, which decouples authenticity from acoustic corruption and requires different evaluation logic than standard binary spoof detection.
- Assuming fixed SNR test splits are generated once; the paper specifies stochastic on-the-fly mixing during training and fixed-SNR generation per test split for controlled comparison.

## Evidence (verbatim from paper)

> We report three complementary metrics—accuracy, ROC–AUC, and EER—that together capture threshold-free ranking quality, threshold-specific operating points, and overall correctness. ... We therefore use EER as the primary robustness metric in our per-SNR analyses.

## Citation

```bibtex
@misc{sen2025toward,
  title={Toward Noise-Aware Audio Deepfake Detection: Survey, SNR-Benchmarks, and Practical Recipes},
  author={Sen et al. (2025)},
  year={2025},
  note={arXiv:2512.13744}
}
```

- arXiv: 2512.13744