Ms Cocoai Eval

---
name: ms-cocoai-eval
description: Evaluates a model's ability to distinguish real images from AI-generated ones, and to identify the specific generative model that produced a synthetic image. It probes robustness to semantic alignment and fine-grained model attribution. Use when the user wants to benchmark on MS COCOAI, or asks about evaluating this task. Reports baseline_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.00553
  bibtex_key: roy2026cocoai
  confidence: medium
---

# ms-cocoai-eval

> A Comprehensive Dataset for Human vs. AI Generated Image Detection — Roy et al. (2026) (arXiv:2601.00553, 2026)

## What this evaluates

Evaluates a model's ability to distinguish real images from AI-generated ones, and to identify the specific generative model that produced a synthetic image. It probes robustness to semantic alignment and fine-grained model attribution.

## Datasets

- **MS COCOAI** — total 96000; splits: test (-1)

## Metrics

- `baseline_score` **(primary)** — range: [0, 1]
  - Correct predictions divided by total predictions across all test instances.

## Input / output format

**Input**: Image files, optionally paired with their generation captions.

**Output**: Task A: Binary label ('AI-generated' or 'human-created'). Task B: Categorical label corresponding to one of the five generative models (SD3, SD2.1, SDXL, DALL-E 3, MidJourney v6).

## Scoring recipe

```python
def compute_baseline_score(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Semantic alignment via shared captions makes distinguishing real vs. AI images significantly harder than in unaligned datasets.
- Multi-class model attribution (Task B) is substantially more difficult than binary detection, often yielding scores near random chance for similar architectures.
- Performance may drop sharply under systematic image perturbations, requiring robustness testing beyond standard accuracy.

## Evidence (verbatim from paper)

> Baseline performance metrics, given in Table [3] establish benchmarks for both authenticity detection and model attribution tasks, serving as reference points for subsequent research developments.

| Task | Description | Baseline Score |
| --- | --- | --- |
| Task A | Classify each image as either AI-generated or created by a human | 0.80144 |
| Task B | Given an AI-generated image, determine which specific model produced it | 0.44913 |

## Citation

```bibtex
@misc{roy2026cocoai,
  title={A Comprehensive Dataset for Human vs. AI Generated Image Detection},
  author={Roy et al. (2026)},
  year={2026},
  note={arXiv:2601.00553}
}
```

- arXiv: 2601.00553