Openfake Eval

---
name: openfake-eval
description: Binary classification capability for detecting AI-generated images versus real photographs. It probes a model's ability to generalize across diverse generative models (diffusion, transformer-based) and real-world social media distributions. Use when the user wants to benchmark on OpenFake, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.09495
  bibtex_key: livernoche2025openfake
  confidence: high
---

# openfake-eval

> OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection — Livernoche et al. (2025) (arXiv:2509.09495, 2025)

## What this evaluates

Binary classification capability for detecting AI-generated images versus real photographs. It probes a model's ability to generalize across diverse generative models (diffusion, transformer-based) and real-world social media distributions.

## Datasets

- **OpenFake** — total 3963000; splits: test (-1); repo https://github.com/vicliv/OpenFake

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `TNR` — range: [0, 1]
  - True Negative Rate (Specificity): proportion of actual real images correctly identified as real.
- `TPR` — range: [0, 1]
  - True Positive Rate (Recall/Sensitivity): proportion of actual fake images correctly identified as fake.

## Input / output format

**Input**: Single image (real photograph or AI-generated/deepfake image).

**Output**: Binary classification label (real vs. fake) or probability score.

## Scoring recipe

```python
tp = sum(p == 1 and g == 1 for p, g in zip(preds, golds))
fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- High true positive rates on fakes often mask unacceptably high false positive rates on real images (low TNR), as seen with legacy baselines.
- Models trained on narrow or outdated datasets (e.g., FaceForensics++) fail to generalize to modern diffusion-based generators.
- The in-the-wild social media test set is small (1,220 images) and heavily curated, potentially underestimating real-world difficulty.

## Evidence (verbatim from paper)

> Table 4: Generalization of SwinV2 detectors trained on different benchmarks when evaluated on an in-the-wild social-media set (1,057 real, 163 fake; see [Section 3]). Metrics include TNR (real) and TPR (fake). Training on OpenFake yields balanced performance, while GenImage and Semi-Truths show strong class biases.

## Citation

```bibtex
@misc{livernoche2025openfake,
  title={OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection},
  author={Livernoche et al. (2025)},
  year={2025},
  note={arXiv:2509.09495}
}
```

- arXiv: 2509.09495