Planviz Eval

---
name: planviz-eval
description: Evaluates multimodal models' ability to generate and edit images that correctly follow planning-oriented instructions (route planning, workflow diagramming, web/UI displaying). It probes procedural reasoning, spatial consistency, and semantic alignment in visual synthesis. Use when the user wants to benchmark on PlanViz, or asks about evaluating this task. Reports Cor.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.06663
  bibtex_key: li2026planviz
  confidence: high
---

# planviz-eval

> PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks — Li et al. (2026) (arXiv:2602.06663, 2026)

## What this evaluates

Evaluates multimodal models' ability to generate and edit images that correctly follow planning-oriented instructions (route planning, workflow diagramming, web/UI displaying). It probes procedural reasoning, spatial consistency, and semantic alignment in visual synthesis.

## Datasets

- **PlanViz** — total ?; splits: test (-1)

## Metrics

- `Cor` **(primary)** — range: [0, 1]
  - Score between 0 and 1 measuring whether the generated or edited image correctly follows the planning task instructions and contains useful, accurate content.
- `Vis` — range: [0, 1]
  - Score between 0 and 1 evaluating the aesthetic and visual coherence of the output image.
- `Ef` — range: [0, 1]
  - Score between 0 and 1 measuring the efficiency or absence of unwanted parts in the generated image.
- `Avg` — range: [0, 1]
  - Arithmetic mean of Cor, Vis, and Ef scores.

## Input / output format

**Input**: Text prompts describing planning tasks (route planning, workflow diagramming, or web/UI displaying) for either image generation or image editing.

**Output**: Generated or edited images.

## Scoring recipe

```python
def score_planviz(predictions, gold):
    scores = []
    for img in random.sample(predictions, 50):
        c, v, e = judge_model.evaluate(img, gold) # Qwen3-VL-235B-A22B-Instruct API
        scores.append({'Cor': c, 'Vis': v, 'Ef': e})
    avg_cor = mean([s['Cor'] for s in scores])
    avg_vis = mean([s['Vis'] for s in scores])
    avg_ef = mean([s['Ef'] for s in scores])
    return {'Cor': avg_cor, 'Vis': avg_vis, 'Ef': avg_ef, 'Avg': (avg_cor + avg_vis + avg_ef) / 3}
```

## Common pitfalls

- High visual quality (Vis) scores often mask low correctness (Cor) scores, as models frequently generate visually plausible but semantically misaligned images.
- Thinking/reasoning mechanisms do not consistently improve performance and can sometimes degrade results, especially in editing tasks.
- Editing tasks are significantly harder than generation tasks due to dual constraints (preserving original content while following new instructions).

## Evidence (verbatim from paper)

> Note that in our experiments, $Cor$ is the most important score to measure whether models can plan tasks and generate with correct and useful images. We randomly select a subset of 50 images from all generations and utilize the Python API of Qwen3-VL-235B-A22B-Instruct, mentioned in Section 3.3 to judge them 10 times. All scores are between 0 and 1.

## Citation

```bibtex
@misc{li2026planviz,
  title={PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.06663}
}
```

- arXiv: 2602.06663