Opensdi Eval

---
name: opensdi-eval
description: This benchmark evaluates the ability of vision models to detect and localize diffusion-generated images in an open-world setting. It probes cross-domain generalization across multiple diffusion architectures (SD1.5, SD2.1, SDXL, SD3, Flux.1) and tests robustness against common image degradations like Gaussian blur and JPEG compression. Use when the user wants to benchmark on OpenSDID, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.19653
  bibtex_key: wang2025opensdi
  confidence: high
---

# opensdi-eval

> OpenSDI: Spotting Diffusion-Generated Images in the Open World — Wang et al. (2025) (arXiv:2503.19653, 2025)

## What this evaluates

This benchmark evaluates the ability of vision models to detect and localize diffusion-generated images in an open-world setting. It probes cross-domain generalization across multiple diffusion architectures (SD1.5, SD2.1, SDXL, SD3, Flux.1) and tests robustness against common image degradations like Gaussian blur and JPEG compression.

## Datasets

- **OpenSDID** — total ?; splits: train (-1), test (-1); repo https://github.com/iamwangyabin/OpenSDI

## Metrics

- `IoU` — range: [0, 1]
  - Intersection over Union: ratio of the overlapping area between predicted and ground truth masks to their union area.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. Computed separately for pixel-level localization and image-level detection.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified images (real vs. diffusion-generated) out of the total test set.

## Input / output format

**Input**: Single RGB image (resized to 224×224 for feature extraction, 512×512 for augmentation).

**Output**: Binary classification label (real/fake) for image-level detection, and a binary segmentation mask for pixel-level localization.

## Scoring recipe

```python
def compute_metrics(pred_mask, gt_mask, pred_label, gt_label):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    tp = np.logical_and(pred_mask, gt_mask).sum()
    fp = np.logical_and(pred_mask, np.logical_not(gt_mask)).sum()
    fn = np.logical_and(np.logical_not(pred_mask), gt_mask).sum()
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1_pixel = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    acc = (pred_label == gt_label).mean()
    tp_img = np.logical_and(pred_label, gt_label).sum()
    fp_img = np.logical_and(pred_label, np.logical_not(gt_label)).sum()
    fn_img = np.logical_and(np.logical_not(pred_label), gt_label).sum()
    prec_img = tp_img / (tp_img + fp_img) if (tp_img + fp_img) > 0 else 0.0
    rec_img = tp_img / (tp_img + fn_img) if (tp_img + fn_img) > 0 else 0.0
    f1_img = 2 * prec_img * rec_img / (prec_img + rec_img) if (prec_img + rec_img) > 0 else 0.0
    return {'IoU': iou, 'F1_pixel': f1_pixel, 'F1_img': f1_img, 'Accuracy': acc}
```

## Common pitfalls

- Confusing in-domain (same generator as training) vs. cross-domain (unseen generators like Flux.1, SD3) evaluation splits.
- Mixing pixel-level localization metrics (IoU, F1) with image-level detection metrics (Accuracy, F1) when reporting results.
- Ignoring image degradation robustness tests (Gaussian blur, JPEG compression) which are explicitly evaluated in the paper.

## Evidence (verbatim from paper)

> Following standard practices in[[12], [37]], we evaluate methods using both pixel-level and image-level metrics on the OpenSDID’s test data (Table [2]). For pixel-level evaluation, we employ the F1-score (F1) and Intersection over Union (IoU). At the image level, we assess performance using F1 and accuracy (Acc).

## Citation

```bibtex
@misc{wang2025opensdi,
  title={OpenSDI: Spotting Diffusion-Generated Images in the Open World},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.19653}
}
```

- arXiv: 2503.19653