Talkingheadbench Eval

---
name: talkingheadbench-eval
description: Evaluates the robustness and generalization of deepfake detectors on talking-head videos under distribution shifts in identity and generator type. It probes whether detectors rely on spurious background cues or robust facial artifacts, and measures performance degradation when facing unseen generators and identities. Use when the user wants to benchmark on TalkingHeadBench, or asks about evaluating this task. Reports TPR@FPR=1% (T1).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24866
  bibtex_key: xiong2025talkingheadbench
  confidence: high
---

# talkingheadbench-eval

> TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection — Xinqi Xiong et al. (2025) (arXiv:2505.24866, 2025)

## What this evaluates

Evaluates the robustness and generalization of deepfake detectors on talking-head videos under distribution shifts in identity and generator type. It probes whether detectors rely on spurious background cues or robust facial artifacts, and measures performance degradation when facing unseen generators and identities.

## Datasets

- **TalkingHeadBench** — total ?; splits: train (-1), test (-1)

## Metrics

- `TPR@FPR=1% (T1)` **(primary)** — range: [0, 1]
  - True Positive Rate (recall) calculated at a fixed False Positive Rate threshold of 1%. Prioritized for most analysis to reflect real-world deployment constraints where false accusations must be minimized.
- `TPR@FPR=0.1% (T0.1)` — range: [0, 1]
  - True Positive Rate calculated at a stricter False Positive Rate threshold of 0.1%. Used to evaluate detector reliability under extreme low-false-positive conditions.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring overall discriminative ability across all classification thresholds.
- `Brier Score` — range: [0, 1]
  - Mean squared difference between predicted probability and actual binary outcome, measuring calibration and accuracy.

## Input / output format

**Input**: Talking-head video sequences (frames) containing either real human faces or deepfake faces generated by specific diffusion/transformer models.

**Output**: Continuous confidence score or probability indicating the likelihood of the video being a deepfake (real vs. fake classification).

## Scoring recipe

```python
def compute_tpr_at_fpr(y_true, y_pred, fpr_threshold=0.01):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    idx = np.argmin(np.abs(fpr - fpr_threshold))
    return tpr[idx]

def compute_auc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

# Usage per protocol/generator:
# t1_score = compute_tpr_at_fpr(y_true, y_pred, 0.01)
# auc_score = compute_auc(y_true, y_pred)
```

## Common pitfalls

- Detectors may overfit to generator-specific artifacts (e.g., neck distortions in EMOPortraits) or background cues rather than learning robust facial manipulation signals, leading to poor generalization under combined shifts.
- Performance metrics like T1 can mask severe degradation at stricter thresholds (e.g., T0.1), leading to overoptimistic assessments of real-world reliability.
- Identity leakage between training and test sets can artificially inflate generalization scores if strict face comparison is not enforced across real and fake video sets.

## Evidence (verbatim from paper)

> Detector performance under these protocols is measured with three metrics: AUC, Brier Score, TPR@FPR=1% (T1), and TPR@FPR=0.1% (T0.1), following IJB-C face benchmark. Stricter thresholds on FPR, e.g., T0.1, are more useful when these detectors are applied at scale and only a small amount of bad detections are tolerable. We will prioritize T1 for most analysis in this paper, following its use in the IJB-C face benchmark, but later we will analyze the detector performance across various thresholds in FPR.

## Citation

```bibtex
@misc{xiong2025talkingheadbench,
  title={TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection},
  author={Xinqi Xiong et al. (2025)},
  year={2025},
  note={arXiv:2505.24866}
}
```

- arXiv: 2505.24866