Mscadd Eval

---
name: mscadd-eval
description: Evaluates audio deepfake detection models on their ability to distinguish real from synthetically generated multi-speaker conversations. It probes robustness to conversational dynamics, speech overlap, and varying acoustic conditions. Use when the user wants to benchmark on MsCADD, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.00295
  bibtex_key: alabi2026multispeaker
  confidence: high
---

# mscadd-eval

> Multi-Speaker Conversational Audio Deepfake: Taxonomy, Dataset and Pilot Study — Alabi Ahmed et al. (2026) (arXiv:2602.00295, 2026)

## What this evaluates

Evaluates audio deepfake detection models on their ability to distinguish real from synthetically generated multi-speaker conversations. It probes robustness to conversational dynamics, speech overlap, and varying acoustic conditions.

## Datasets

- **MsCADD** — total 2830; splits: train (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for the fake class: 2 * (Precision * Recall) / (Precision + Recall).
- `Accuracy` — range: [0, 1]
  - Ratio of correctly classified instances (both real and fake) to the total number of instances.
- `True Positive Rate (TPR)` — range: [0, 1]
  - Recall for the fake class: TP / (TP + FN).
- `True Negative Rate (TNR)` — range: [0, 1]
  - Specificity for the real class: TN / (TN + FP).

## Input / output format

**Input**: Audio clips containing two-speaker conversations (clean or noisy conditions).

**Output**: Binary classification label (real vs. fake) or aggregated per-clip fake probability scores.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * precision * tpr / (precision + tpr) if (precision + tpr) > 0 else 0
    return {'accuracy': accuracy, 'tpr': tpr, 'tnr': tnr, 'f1': f1}
```

## Common pitfalls

- Decision thresholds are empirically tuned per model/dataset rather than fixed (e.g., -15.82 for LFCC-LCNN).
- Models exhibit high false positive rates on real conversational speech due to acoustic complexity and overlap.
- Pilot study does not report performance breakdowns by TTS system or speaker gender composition.

## Evidence (verbatim from paper)

> We report accuracy (percentage of correctly classified instances), true positive rate (TPR), True negative rate (TNR) and F1 score for fake class.

## Citation

```bibtex
@misc{alabi2026multispeaker,
  title={Multi-Speaker Conversational Audio Deepfake: Taxonomy, Dataset and Pilot Study},
  author={Alabi Ahmed et al. (2026)},
  year={2026},
  note={arXiv:2602.00295}
}
```

- arXiv: 2602.00295