Diffseg30k Eval

---
name: diffseg30k-eval
description: Pixel-level localization of diffusion-based AI edits in images, shifting from whole-image classification to semantic segmentation to identify precisely which regions have been altered by generative models. Use when the user wants to benchmark on DiffSeg30k, or asks about evaluating this task. Reports localization accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19111
  bibtex_key: ci2025diffseg30k
  confidence: high
---

# diffseg30k-eval

> DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection — Ci et al. (2025) (arXiv:2511.19111, 2025)

## What this evaluates

Pixel-level localization of diffusion-based AI edits in images, shifting from whole-image classification to semantic segmentation to identify precisely which regions have been altered by generative models.

## Datasets

- **DiffSeg30k** — total 30000; splits: test (-1)

## Metrics

- `localization accuracy` **(primary)** — range: [0, 1]
  - Pixel-level semantic segmentation overlap between predicted and ground-truth edit masks, typically computed as mean Intersection-over-Union (mIoU) or Dice coefficient across all edited regions.

## Input / output format

**Input**: RGB images (base images, half real from COCO, half AI-generated) with corresponding ground-truth binary masks indicating the exact regions edited by diffusion models.

**Output**: Pixel-level binary segmentation mask predicting the edited regions in the input image.

## Scoring recipe

```python
def compute_localization_accuracy(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    return iou
# Aggregate over dataset to compute mIoU
```

## Common pitfalls

- Multi-turn editing compounds artifacts, making ground-truth masks harder to align with model predictions.
- Automated pipeline may produce low-quality edits or mask errors, requiring quality filtering (score < 3) that can bias evaluation if not standardized.
- Cross-generator generalization is tested across 8 different diffusion models, but robustness to image distortions remains a known weakness.

## Evidence (verbatim from paper)

> It shifts AIGC detection from whole-image classification to pixel-level semantic segmentation, revealing that segmentation models outperform traditional forgery classifiers in both localization accuracy and cross-generator generalization, despite challenges in robustness to image distortions and multi-turn editing complexity.

## Citation

```bibtex
@misc{ci2025diffseg30k,
  title={DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection},
  author={Ci et al. (2025)},
  year={2025},
  note={arXiv:2511.19111}
}
```

- arXiv: 2511.19111