Sid Generalization Eval

---
name: sid-generalization-eval
description: Evaluates a model's ability to generalize synthetic image detection across diverse generative architectures (GANs, Diffusion Models, DiTs) and real-world image sources, focusing on robustness to unseen generators and varying image resolutions. Use when the user wants to benchmark on SID Generalization Benchmark (ForenSynths, Self-Synthesis, Ojha, GenImage, DiTFake), or asks about evaluating this task. Reports ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.06741
  bibtex_key: li2024improving
  confidence: high
---

# sid-generalization-eval

> Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective — Ouxiang Li et al. (2024) (arXiv:2408.06741, 2024)

## What this evaluates

Evaluates a model's ability to generalize synthetic image detection across diverse generative architectures (GANs, Diffusion Models, DiTs) and real-world image sources, focusing on robustness to unseen generators and varying image resolutions.

## Datasets

- **SID Generalization Benchmark (ForenSynths, Self-Synthesis, Ojha, GenImage, DiTFake)** — total ?; splits: train (72000), test (-1); repo https://github.com/Ouxiang-Li/SAFE

## Metrics

- `ACC` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly classified real versus fake images out of the total test set.
- `AP` — range: [0, 1]
  - Average Precision (area under the precision-recall curve) for the binary real/fake classification task.

## Input / output format

**Input**: RGB images, pre-processed via random 256×256 cropping during training and center 256×256 cropping during testing.

**Output**: Binary classification label: 'real' or 'fake'.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    acc = (predictions.argmax(axis=1) == labels).mean()
    ap = average_precision_score(labels, predictions)
    return acc, ap
```

## Common pitfalls

- Training is strictly limited to ProGAN and LSUN (4-class), so high performance on training data does not indicate generalization.
- The GenImage testset includes highly variable resolutions (128² to 1024²), which can cause performance drops if models are not resolution-invariant.
- Metrics are averaged across 26+ generators (ACC_M / AP_M), masking per-generator weaknesses.

## Evidence (verbatim from paper)

> The classification accuracy (ACC) and average precision (AP) are introduced as the main metrics in evaluating the SID performance across various generators. To intuitively evaluate the detection performance on GANs and DMs, we also report the averaged metrics for each testset, termed ACC${}_{\text{M}}$ and AP${}_{\text{M}}$.

## Citation

```bibtex
@misc{li2024improving,
  title={Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective},
  author={Ouxiang Li et al. (2024)},
  year={2024},
  note={arXiv:2408.06741}
}
```

- arXiv: 2408.06741