Geneval Eval

---
name: geneval-eval
description: Evaluates text-to-image alignment by testing whether models correctly render specified objects, attributes, and spatial relations across six categories: single object, two objects, counting, colors, position, and attribute binding. Use when the user wants to benchmark on GenEval, or asks about evaluating this task. Reports Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12163
  bibtex_key: akiti2026nucleusimage
  confidence: high
---

# geneval-eval

> Nucleus-Image: Sparse MoE for Image Generation — Akiti et al. (2026) (arXiv:2604.12163, 2026)

## What this evaluates

Evaluates text-to-image alignment by testing whether models correctly render specified objects, attributes, and spatial relations across six categories: single object, two objects, counting, colors, position, and attribute binding.

## Datasets

- **GenEval** — total ?; splits: test (-1)

## Metrics

- `Overall` **(primary)** — range: [0, 1]
  - Mean accuracy across six categories: single object, two objects, counting, colors, position, and attribute binding.

## Input / output format

**Input**: Text prompt specifying objects, attributes, and spatial relations to be rendered.

**Output**: Generated image at 1024x1024 resolution, 50 inference steps, CFG scale 8.0.

## Scoring recipe

```python
category_scores = {cat: [] for cat in ['single', 'two', 'count', 'colors', 'position', 'attr']}
for prompt in prompts:
    img = model.generate(prompt, steps=50, cfg=8.0, res=1024)
    cat = get_category(prompt)
    correct = vqa_model.verify(img, prompt)
    category_scores[cat].append(correct)
overall = mean(mean(v) for v in category_scores.values())
return overall
```

## Common pitfalls

- Models frequently struggle with spatial position and counting categories compared to single-object or color rendering.
- Automated evaluation relies on external VQA models that may introduce measurement noise or bias.

## Evidence (verbatim from paper)

> Nucleus-Image achieves an overall score of 0.865 (reported as 0.87), matching Qwen-Image and surpassing all other reported models including GPT Image 1 High (0.84) and Seedream 3.0 (0.84).

## Citation

```bibtex
@misc{akiti2026nucleusimage,
  title={Nucleus-Image: Sparse MoE for Image Generation},
  author={Akiti et al. (2026)},
  year={2026},
  note={arXiv:2604.12163}
}
```

- arXiv: 2604.12163