T2i Reasoning Eval

---
name: t2i-reasoning-eval
description: Evaluates text-to-image generation models on their ability to align with complex, compositional prompts through iterative fine-grained reasoning and self-refinement. It probes capabilities in object counting, attribute binding, spatial relationships, and handling long, dense prompts. Use when the user wants to benchmark on GenEval, T2I-CompBench, DPGBench, or asks about evaluating this task. Reports GenEval, T2I-CompBench, and DPGBench alignment scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13491
  bibtex_key: kim2026enhanced
  confidence: high
---

# t2i-reasoning-eval

> Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning — Kim et al. (2026) (arXiv:2604.13491, 2026)

## What this evaluates

Evaluates text-to-image generation models on their ability to align with complex, compositional prompts through iterative fine-grained reasoning and self-refinement. It probes capabilities in object counting, attribute binding, spatial relationships, and handling long, dense prompts.

## Datasets

- **GenEval** — total ?; splits: test (-1)
- **T2I-CompBench** — total ?; splits: test (-1)
- **DPGBench** — total ?; splits: test (-1)

## Metrics

- `GenEval, T2I-CompBench, and DPGBench alignment scores` **(primary)** — range: [0, 1]
  - Models generate images from prompts and optionally refine them up to three times based on self-judgment. Alignment is measured using official benchmark implementations (with a corrected DPGBench script for multi-GPU stability). Scores are reported as accuracy/alignment percentages per category and overall, ranging from 0 to 1.

## Input / output format

**Input**: Text prompts ranging from simple object descriptions to long, dense compositional instructions.

**Output**: Generated images (iteratively refined up to 3 steps based on self-judgment).

## Scoring recipe

```python
scores = []
for prompt, gold in dataset:
    img = model.generate(prompt)
    for step in range(3):
        feedback = model.self_judge(img, prompt)
        if feedback == 'aligned': break
        img = model.refine(img, feedback)
    scores.append(benchmark_score(img, prompt))
return mean(scores)
```

## Common pitfalls

- DPGBench official evaluation code has known bugs under multi-GPU configurations; a corrected version must be used for reliable results.
- CLIP-T score used in T2I-CompBench may not capture fine-grained semantic improvements, potentially showing marginal declines in Non-Spatial categories despite actual visual refinements.
- Iterative refinement is conditional: if the self-judge deems an image aligned, subsequent steps are skipped, which affects average scores across the dataset.

## Evidence (verbatim from paper)

> To assess the image reasoning capabilities of our models, we utilize three comprehensive benchmarks: DPGBench, GenEval, and T2I-CompBench. For GenEval and T2I-Compbench, we strictly follow the official implementations to ensure consistency and comparability. For these two models, we report performance across three sequential stages: the initial T2I generation (denoted as 1st) and two subsequent refinement rounds (2nd and 3rd). Specifically, if the self-judge determines an image is aligned, subsequent refinement steps are skipped and the preceding image is retained.

## Citation

```bibtex
@misc{kim2026enhanced,
  title={Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning},
  author={Kim et al. (2026)},
  year={2026},
  note={arXiv:2604.13491}
}
```

- arXiv: 2604.13491