Figurebench Eval

---
name: figurebench-eval
description: Evaluates the ability of text-to-illustration models to generate publication-ready scientific figures that balance structural fidelity, visual aesthetics, and communicative clarity based on long-form scientific text. Use when the user wants to benchmark on FigureBench, or asks about evaluating this task. Reports Overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.03828
  bibtex_key: zhu2026autofigure
  confidence: high
---

# figurebench-eval

> AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations — Zhu et al. (2026) (arXiv:2602.03828, 2026)

## What this evaluates

Evaluates the ability of text-to-illustration models to generate publication-ready scientific figures that balance structural fidelity, visual aesthetics, and communicative clarity based on long-form scientific text.

## Datasets

- **FigureBench** — total 3300; splits: test (-1); repo https://github.com/ResearAI/AutoFigure

## Metrics

- `Overall score` **(primary)** — range: [0, 5]
  - Composite 1–5 Likert scale rating assessing visual design excellence, communication effectiveness, and content fidelity.
- `Win-Rate` — range: percent
  - Percentage of times a model's generated figure is preferred over a baseline in blind pairwise comparisons.

## Input / output format

**Input**: Natural language text or instructions extracted from scientific documents (blogs, surveys, textbooks, or papers) describing a figure to be generated.

**Output**: Generated scientific illustration (image, SVG, or HTML code) representing the described figure.

## Scoring recipe

```python
# For Overall Score (automated or human)
scores = [rating for rating in ratings if 1 <= rating <= 5]
overall_score = sum(scores) / len(scores)

# For Win-Rate
wins = sum(1 for pred, baseline in pairwise_comparisons if pred > baseline)
win_rate = (wins / len(pairwise_comparisons)) * 100
```

## Common pitfalls

- Win-Rate is derived from blind pairwise comparisons against baselines, not absolute scoring, so it cannot be directly compared to Likert scale scores.
- The 1–5 Likert scale measures composite design/communication/fidelity, not standard accuracy or FID/KID metrics used in general image generation.
- Human evaluation was restricted to 10 domain experts assessing only 21 of their own first-author publications, limiting generalizability to the full 3,300-item benchmark.

## Evidence (verbatim from paper)

> As detailed in Table 2, AUTOFIGURE achieves the highest Overall score across all four document categories: Blog (7.60), Survey (6.99), Textbook (8.00), and Paper (7.03). Notably, AUTOFIGURE also dominates in Win-Rate evaluations through blind pairwise comparisons, achieving  $75.0\%$  for Blog,  $78.1\%$  for Survey, an exceptional  $97.5\%$  for Textbook, and  $53.0\%$  for Paper.

## Citation

```bibtex
@misc{zhu2026autofigure,
  title={AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2602.03828}
}
```

- arXiv: 2602.03828