Igenbench Eval

---
name: igenbench-eval
description: Probes the reliability of text-to-infographic generation models by decomposing visual fidelity into atomic yes/no checks. It evaluates whether generated images accurately encode data, follow structural constraints, and maintain consistency across multiple verification questions. Use when the user wants to benchmark on IGenBench, or asks about evaluating this task. Reports Q-ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04498
  bibtex_key: tang2026igenbench
  confidence: high
---

# igenbench-eval

> IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation — Tang et al. (2026) (arXiv:2601.04498, 2026)

## What this evaluates

Probes the reliability of text-to-infographic generation models by decomposing visual fidelity into atomic yes/no checks. It evaluates whether generated images accurately encode data, follow structural constraints, and maintain consistency across multiple verification questions.

## Datasets

- **IGenBench** — total 600; splits: test (600)

## Metrics

- `Q-ACC` **(primary)** — range: [0, 1]
  - Average accuracy across 10 atomic yes/no verification questions per infographic, evaluated by a multimodal LLM.
- `I-ACC` — range: [0, 1]
  - Proportion of infographics where all verification questions are answered correctly, measuring end-to-end correctness.

## Input / output format

**Input**: Text prompt specifying data values, chart type, layout constraints, and accompanying verification questions.

**Output**: Generated infographic image.

## Scoring recipe

```python
q_accs, i_accs = [], []
for instance in dataset:
    image = generate(instance.prompt)
    q_answers = [eval_mllm(image, q) for q in instance.questions]
    q_accs.append(mean(q_answers))
    i_accs.append(all(q_answers))
Q_ACC = mean(q_accs)
I_ACC = mean(i_accs)
```

## Common pitfalls

- High Q-ACC does not imply a reliable infographic; I-ACC is significantly lower due to long-tail failures where one critical error invalidates the whole image.
- Data encoding and completeness are consistently the hardest dimensions, often failing even in top-performing models optimized for aesthetics.
- Automatic evaluation relies heavily on the specific MLLM used for verification (Gemini-2.5-Pro), which may not perfectly match human judgment in all cases.

## Evidence (verbatim from paper)

> As shown in Table 1, there exists a dramatic gap between Q-ACC and I-ACC across all models, with I-ACC consistently much lower. The best-performing model achieves a Q-ACC of 0.90 but only 0.49 I-ACC.

## Citation

```bibtex
@misc{tang2026igenbench,
  title={IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2601.04498}
}
```

- arXiv: 2601.04498