Sonics Eval

---
name: sonics-eval
description: This benchmark evaluates a model's ability to distinguish between real human-recorded songs and AI-generated synthetic songs. It specifically probes long-range temporal dependency modeling by testing performance on both short (5s) and long (120s) audio clips, while also measuring generalization to unseen generation algorithms and singers. Use when the user wants to benchmark on SONICS, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14080
  bibtex_key: rahman2024sonics
  confidence: high
---

# sonics-eval

> SONICS: Synthetic Or Not -- Identifying Counterfeit Songs — Md Awsafur Rahman et al. (2024) (arXiv:2408.14080, 2024)

## What this evaluates

This benchmark evaluates a model's ability to distinguish between real human-recorded songs and AI-generated synthetic songs. It specifically probes long-range temporal dependency modeling by testing performance on both short (5s) and long (120s) audio clips, while also measuring generalization to unseen generation algorithms and singers.

## Datasets

- **SONICS** — total 97000; splits: train (-1), valid (-1), test (-1); repo https://github.com/awsaf49/sonics

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Binary average F1 score computed at a fixed decision threshold of 0.5. It balances precision and recall across the real (negative) and fake (positive) classes.
- `Sensitivity` — range: [0, 1]
  - True positive rate (recall) for the fake/AI class.
- `Specificity` — range: [0, 1]
  - True negative rate for the real/human class.

## Input / output format

**Input**: Raw audio songs resampled to 16kHz, converted to mel-spectrograms (n_mels=128, hop_length=512, win_length=2048). Inputs are fixed to either 5 seconds (128x128 spectrogram) or 120 seconds (128x3744 spectrogram) via right-side zero-padding or middle cropping.

**Output**: Binary classification prediction (Real/Human vs Fake/AI) or probability score.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, threshold=0.5):
    y_pred_binary = (y_pred >= threshold).astype(int)
    tp = np.sum((y_pred_binary == 1) & (y_true == 1))
    fp = np.sum((y_pred_binary == 1) & (y_true == 0))
    fn = np.sum((y_pred_binary == 0) & (y_true == 1))
    tn = np.sum((y_pred_binary == 0) & (y_true == 0))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    sensitivity = recall
    specificity = tn / (tn + fp + 1e-8)
    return {'F1': f1, 'Sensitivity': sensitivity, 'Specificity': specificity}
```

## Common pitfalls

- Using Equal Error Rate (EER) as the primary metric can be misleading due to class imbalance in the dataset; the authors explicitly prioritize F1 score instead.
- Evaluating only on short audio clips (e.g., 5s) masks the model's ability to capture long-range temporal dependencies, which is the core capability this benchmark is designed to test.
- Failing to enforce strict train/valid-test separation for songs sharing the same (lyrics, style) inputs leads to data leakage and inflated performance on unseen algorithm/singer splits.

## Evidence (verbatim from paper)

> While existing methods (Zang et al., 2024b;a; Xie et al., 2024) use Equal Error Rate (EER) as a metric, we prioritize the F1 score (binary average, threshold = 0.5) as our primary metric due to EER's susceptibility to class imbalance. We also evaluate Sensitivity (Sens.) and Specificity (Spec.) to assess performance across fake and real classes.

## Citation

```bibtex
@misc{rahman2024sonics,
  title={SONICS: Synthetic Or Not -- Identifying Counterfeit Songs},
  author={Md Awsafur Rahman et al. (2024)},
  year={2024},
  note={arXiv:2408.14080}
}
```

- arXiv: 2408.14080