Magicmirror Eval

---
name: magicmirror-eval
description: Evaluates text-to-image generation models on their ability to produce images free of fine-grained artifacts, specifically probing subject anatomy, attributes, and interactions. It measures detection accuracy using a hierarchical taxonomy of artifact types to benchmark model robustness against visual inconsistencies. Use when the user wants to benchmark on MagicData340K, or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.10260
  bibtex_key: wang2025magicmirror
  confidence: high
---

# magicmirror-eval

> MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation — Wang et al. (2025) (arXiv:2509.10260, 2025)

## What this evaluates

Evaluates text-to-image generation models on their ability to produce images free of fine-grained artifacts, specifically probing subject anatomy, attributes, and interactions. It measures detection accuracy using a hierarchical taxonomy of artifact types to benchmark model robustness against visual inconsistencies.

## Datasets

- **MagicData340K** — total 340000; splits: train (-1)

## Metrics

- `F1-Score` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (P * R) / (P + R). Calculated per class and macro/micro averaged across L2 categories.
- `Precision` — range: [0, 1]
  - Ratio of correctly identified artifact instances to all instances predicted as artifacts.
- `Recall` — range: [0, 1]
  - Ratio of correctly identified artifact instances to all actual artifact instances.
- `Overall Score` — range: [0, 100]
  - Aggregated benchmark score across interaction, human, animal, and object categories for MagicBench evaluation.

## Input / output format

**Input**: A text prompt and the corresponding generated image.

**Output**: Binary classification label (artifact present/absent), hierarchical L2 category labels, and a Chain-of-Thought inspection explanation.

## Scoring recipe

```python
def calculate_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- Larger VLMs often perform worse on artifact detection due to increased conservatism, leading to significantly lower recall.
- Interaction and object morphology categories are inherently harder; models struggle to distinguish actual artifacts from low-quality image regions or diverse object types.
- Without multi-level reward consistency, VLMs may produce disorganized explanations and suffer from reward hacking during GRPO training.

## Evidence (verbatim from paper)

> We evaluate model performance using Precision, Recall, and F1-Score, calculated for both the overall artifact detection task and for each of our primary L2 categories.

## Citation

```bibtex
@misc{wang2025magicmirror,
  title={MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.10260}
}
```

- arXiv: 2509.10260