Omniaid Eval

---
name: omniaid-eval
description: Evaluates the ability of AI-generated image detectors to generalize across different semantic domains (human, animal, object, scene) and resist modern, photorealistic generative models. It probes whether models rely on content-agnostic artifacts versus semantic features for robust real-vs-fake classification. Use when the user wants to benchmark on GenImage, Chameleon, Mirage-Test, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08423
  bibtex_key: guo2025omniaid
  confidence: high
---

# omniaid-eval

> OmniAID: Decoupling Semantic and Artifacts for Universal AI-Generated Image Detection in the Wild — Guo et al. (2025) (arXiv:2511.08423, 2025)

## What this evaluates

Evaluates the ability of AI-generated image detectors to generalize across different semantic domains (human, animal, object, scene) and resist modern, photorealistic generative models. It probes whether models rely on content-agnostic artifacts versus semantic features for robust real-vs-fake classification.

## Datasets

- **GenImage** — total ?; splits: test (-1)
- **Chameleon** — total ?; splits: test (-1)
- **Mirage-Test** — total ?; splits: test (-1)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Standard binary classification accuracy: (True Positives + True Negatives) / Total Samples. Reported per semantic category and as a mean across categories.
- `Average Precision (AP)` — range: percent
  - Mean Average Precision computed over the precision-recall curve for binary real-vs-fake classification. Reported per category and as a mean.

## Input / output format

**Input**: RGB images resized to 512×512, then downsampled to 336×336 for the CLIP-ViT-L/14 backbone. Each instance is a single image labeled as Real or AI-Generated.

**Output**: Binary classification label (Real/Fake) or confidence score for each image.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    # predictions: list of predicted labels or probabilities
    # labels: list of ground truth labels (0=Real, 1=Fake)
    accuracy = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    ap = average_precision_score(labels, predictions)
    return {'accuracy': accuracy, 'ap': ap}
```

## Common pitfalls

- Models often overfit to specific generator fingerprints, causing catastrophic failure on unseen or modern generators (Real/Fake bias).
- Evaluating only on outdated benchmarks like GenImage-SD v1.4 masks poor generalization to contemporary diffusion models.
- The 'Anime' category in Mirage-Test contains only generated samples, making overall accuracy for that category meaningless without separate real/fake reporting.

## Evidence (verbatim from paper)

> To ensure a fair comparison, we follow the protocol of [genimage, aide], training all models (including our standard OmniAID) exclusively on the GenImage-SD v1.4 dataset to assess generalization from a limited, standard benchmark. Alongside this, to evaluate performance in a realistic, modern scenario, we also train our OmniAID-Mirage model on our modern Mirage-Train dataset. All models are then evaluated on the GenImage [genimage] test set, the in-the-wild Chameleon [aide] dataset, and our new Mirage-Test. Unless otherwise specified, we report classification Accuracy (%) as the primary metric. More Average Precision (AP) results are available in the Supplementary Material.

## Citation

```bibtex
@misc{guo2025omniaid,
  title={OmniAID: Decoupling Semantic and Artifacts for Universal AI-Generated Image Detection in the Wild},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2511.08423}
}
```

- arXiv: 2511.08423