Open Sdi Eval

---
name: open-sdi-eval
description: Evaluates a model's ability to localize manipulated regions in images generated by diffusion models, measuring both pixel-level segmentation precision and image-level binary detection capability across multiple unseen generators. Use when the user wants to benchmark on OpenSDI, or asks about evaluating this task. Reports pixel-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12341
  bibtex_key: liang2026bridging
  confidence: high
---

# open-sdi-eval

> Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization — Xiaojie Liang et al. (arXiv:2604.12341, 2026)

## What this evaluates

Evaluates a model's ability to localize manipulated regions in images generated by diffusion models, measuring both pixel-level segmentation precision and image-level binary detection capability across multiple unseen generators.

## Datasets

- **OpenSDI** — total ?; splits: train (-1), test (-1)

## Metrics

- `pixel-level F1` **(primary)** — range: [0, 1]
  - Standard F1 score computed between predicted and ground-truth binary masks after thresholding predictions at 0.5.
- `IoU` — range: [0, 1]
  - Intersection over Union between thresholded (at 0.5) predicted and ground-truth masks.
- `image-level F1` — range: [0, 1]
  - F1 score for binary image-level detection (fake vs real).
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified images (fake vs real).

## Input / output format

**Input**: RGB images resized to 512×512 for the frequency pathway and 224×224 for the semantic pathway.

**Output**: Per-image binary prediction (real/fake) and a pixel-level probability mask for manipulation localization.

## Scoring recipe

```python
def score(pred_masks, gt_masks, pred_images, gt_images):
    pred_masks_bin = (pred_masks > 0.5).astype(int)
    intersection = (pred_masks_bin & gt_masks).sum()
    union = (pred_masks_bin | gt_masks).sum()
    iou = intersection / (union + 1e-8)
    tp, fp, fn = intersection, pred_masks_bin.sum() - intersection, gt_masks.sum() - intersection
    f1 = 2 * tp / (2 * tp + fp + fn + 1e-8)
    pred_label = (pred_images > 0.5).astype(int)
    acc = (pred_label == gt_images).mean()
    tp_i, fp_i, fn_i = (pred_label & gt_images).sum(), (pred_label & (1 - gt_images)).sum(), ((1 - pred_label) & gt_images).sum()
    f1_i = 2 * tp_i / (2 * tp_i + fp_i + fn_i + 1e-8)
    return {'pixel_f1': f1, 'pixel_iou': iou, 'image_f1': f1_i, 'image_acc': acc}
```

## Common pitfalls

- Thresholding predicted localization masks at exactly 0.5 before computing F1 and IoU.
- Evaluating cross-generator generalization without fine-tuning on unseen diffusion models.
- Using different input resolutions (224×224 vs 512×512) for different model branches during evaluation.

## Evidence (verbatim from paper)

> Pixel-level localization is evaluated with pixel-level F1 and IoU after thresholding the predicted masks at 0.5, while image-level detection is evaluated with image-level F1 and Accuracy.

## Citation

```bibtex
@misc{liang2026bridging,
  title={Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization},
  author={Xiaojie Liang et al.},
  year={2026},
  note={arXiv:2604.12341}
}
```

- arXiv: 2604.12341