Timit Tts Eval

---
name: timit-tts-eval
description: Evaluates deepfake detectors on distinguishing real from synthetic audio and video, and on attributing synthetic audio to specific TTS generators. It probes robustness to post-processing (DTW alignment, augmentation) and video compression. Use when the user wants to benchmark on TIMIT-TTS, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.08000
  bibtex_key: salvi2022timittts
  confidence: high
---

# timit-tts-eval

> TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection — Salvi et al. (2022) (arXiv:2209.08000, 2022)

## What this evaluates

Evaluates deepfake detectors on distinguishing real from synthetic audio and video, and on attributing synthetic audio to specific TTS generators. It probes robustness to post-processing (DTW alignment, augmentation) and video compression.

## Datasets

- **TIMIT-TTS** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `balanced accuracy` — range: [0, 1]
  - Average of recall obtained on each class, calculated as the mean of per-class recall scores for multiclass tasks or (TP/(TP+FN) + TN/(TN+FP))/2 for binary tasks.

## Input / output format

**Input**: Raw audio waveforms or video frames (optionally compressed with QP=23 or QP=40).

**Output**: Binary label (real/fake) for detection, or categorical label (TTS algorithm ID) for attribution.

## Scoring recipe

```python
def compute_auc(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_balanced_accuracy(y_true, y_pred):
    recalls = [recall_score(y_true == c, y_pred == c) for c in np.unique(y_true)]
    return np.mean(recalls)
```

## Common pitfalls

- Models may learn speaker identity rather than TTS artifacts, inflating attribution scores for multi-speaker models.
- Performance drops significantly when testing on low-compression videos (QP=40) compared to high-quality training data, revealing robustness gaps.
- DTW post-processing lowers MOS and detection AUC, requiring explicit training on aligned tracks to avoid biased evaluation.

## Evidence (verbatim from paper)

> In particular, the operation that degrades the accuracy the most is the speech-to-speech alignment, with an AUC value that drops by $0.20$ between the clean and the DTW cases.

## Citation

```bibtex
@misc{salvi2022timittts,
  title={TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection},
  author={Salvi et al. (2022)},
  year={2022},
  note={arXiv:2209.08000}
}
```

- arXiv: 2209.08000