Gebench Eval

---
name: gebench-eval
description: Evaluates image generation models' ability to function as dynamic GUI environments, probing temporal coherence, multi-step interaction logic, spatial grounding, and visual fidelity across sequential state transitions. Use when the user wants to benchmark on GEBench, or asks about evaluating this task. Reports GE-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09007
  bibtex_key: li2026gebench
  confidence: high
---

# gebench-eval

> GEBench: Benchmarking Image Generation Models as GUI Environments — Haodong Li et al. (2026) (arXiv:2602.09007, 2026)

## What this evaluates

Evaluates image generation models' ability to function as dynamic GUI environments, probing temporal coherence, multi-step interaction logic, spatial grounding, and visual fidelity across sequential state transitions.

## Datasets

- **GEBench** — total 700; splits: test (700); repo https://github.com/stepfun-ai/GEBench

## Metrics

- `GE-Score` **(primary)** — range: [0, 100]
  - Composite score averaging five dimensions: Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Each dimension is scored by VLM judges, averaged across dimensions and tasks, then scaled to a 0–100 range.

## Input / output format

**Input**: Initial GUI state image + textual instruction/prompt specifying the target interaction or transition.

**Output**: Generated GUI image(s) representing the resulting state after executing the instruction.

## Scoring recipe

```python
def compute_ge_score(predictions, gold):
    all_scores = []
    for traj in predictions:
        traj_scores = []
        for _ in range(3):
            dim_scores = [judge_score(traj, dim) for dim in ['Goal Achievement', 'Interaction Logic', 'Content Consistency', 'UI Plausibility', 'Visual Quality']]
            traj_scores.append(mean(dim_scores))
        all_scores.append(mean(traj_scores))
    return mean(all_scores) * 100
```

## Common pitfalls

- Models excel at single-step transitions but suffer severe performance drops in multi-step planning due to error accumulation.
- VLM judges may inherit biases from their base models; the paper mitigates this by using three independent cross-evaluators and averaging results.
- Precise spatial grounding (e.g., mapping abstract [0,1000] coordinates to pixel locations) is a critical bottleneck often overlooked in standard image generation benchmarks.

## Evidence (verbatim from paper)

> To ensure the objectivity and robustness of GEBench, we deploy 3 state-of-the-art VLMs as independent cross-evaluators: 2 commercial models Gemini-3-Flash-Native, GPT-4o and 1 open-source model Qwen3-vl-235b-a22b-thinking. By utilizing these evaluators, we mitigate potential bias inherent in a single judge model. To ensure fair and reproducible comparisons, we use official default configurations for evaluated models and perform evaluation three times for each generated GUIs trajectory. Experimental results in Table [1] show that Nano Banana Pro delivers the most robust performance, particularly on Chinese subset with a top-ranking GE-Score of 69.62.

## Citation

```bibtex
@misc{li2026gebench,
  title={GEBench: Benchmarking Image Generation Models as GUI Environments},
  author={Haodong Li et al. (2026)},
  year={2026},
  note={arXiv:2602.09007}
}
```

- arXiv: 2602.09007