Co Spy Eval

---
name: co-spy-eval
description: Evaluates the ability of AI-generated image detectors to distinguish between real and synthetic images across diverse generative models, lossy compression formats, and real-world in-the-wild sources. It specifically probes out-of-distribution generalization and robustness to common post-processing transformations like JPEG compression, blurring, and noise. Use when the user wants to benchmark on Co-SpyBench, Co-SpyBench/in-the-wild, AIGCDetectBenchmark, GenImage, or asks about evaluating this task. Reports AP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.18286
  bibtex_key: cheng2025cospy
  confidence: high
---

# co-spy-eval

> CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI — Cheng et al. (2025) (arXiv:2503.18286, 2025)

## What this evaluates

Evaluates the ability of AI-generated image detectors to distinguish between real and synthetic images across diverse generative models, lossy compression formats, and real-world in-the-wild sources. It specifically probes out-of-distribution generalization and robustness to common post-processing transformations like JPEG compression, blurring, and noise.

## Datasets

- **Co-SpyBench** — total ?; splits: test (-1)
- **Co-SpyBench/in-the-wild** — total ?; splits: test (-1)
- **AIGCDetectBenchmark** — total ?; splits: test (-1)
- **GenImage** — total ?; splits: test (-1)

## Metrics

- `AP` **(primary)** — range: [0, 1]
  - Average Precision (AP) is computed as the area under the precision-recall curve across all classification thresholds. It summarizes detector performance across varying confidence levels.
- `Accuracy@0.5` — range: [0, 1]
  - Accuracy is calculated as the fraction of correctly classified samples using a fixed decision threshold of 0.5 on the detector's output probability score.

## Input / output format

**Input**: Single RGB image resized to 224×224 pixels. Inputs are optionally subjected to random JPEG compression (quality 75–95, 50% probability) to simulate real-world upload conditions and prevent format overfitting.

**Output**: Continuous probability score indicating the likelihood of the image being synthetic. A binary prediction is derived by thresholding the score at 0.5.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    # Accuracy@0.5
    preds_binary = [1 if p >= 0.5 else 0 for p in predictions]
    accuracy = sum(p == g for p, g in zip(preds_binary, labels)) / len(labels)
    
    # Average Precision (AP)
    precisions, recalls, _ = precision_recall_curve(labels, predictions)
    ap = np.trapz(precisions, recalls)
    return ap, accuracy
```

## Common pitfalls

- Skipping the 50% probability JPEG compression (quality 75-95) applied to inputs, which significantly impacts artifact-based detectors and alters reported scores.
- Ignoring the strict 224×224 resizing applied to all images, which can degrade high-frequency artifacts crucial for some baseline methods.
- Evaluating on the same generative model used for training (e.g., SD-v1.4) instead of the intended out-of-distribution unseen models, leading to inflated and non-comparable results.

## Evidence (verbatim from paper)

> We employ four test sets for a comprehensive assessment of the detection performance: (1) AIGCDetectBenchmark*[[71]]* including synthetic images from 16 different generative models (half GANs and half diffusion models) with equal number of real images from the source training set of each generative model; (2) GenImage*[[73]]* consisting of 8 types of synthetic images with real ones; (3) Co-SpyBench; and (4) Co-SpyBench/in-the-wild, introduced in [Appendix Q]. Following the existing settings*[[59], [43], [5]]*, all images are resized to 224 $	imes$ 224 for both training and test phases, to ensure a fair comparison. In addition, we randomly apply JPEG compression with quality ranging from 75 to 95 with 50% possibility to the inputs. ... We use AP (Average Precision) and Acc. (Accuracy) with threshold 0.5, as two main metrics to evaluate the detection performance.

## Citation

```bibtex
@misc{cheng2025cospy,
  title={CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2503.18286}
}
```

- arXiv: 2503.18286