Showui Gui Eval

---
name: showui-gui-eval
description: Evaluates a vision-language-action model's ability to perform GUI visual grounding and task-oriented navigation across web, mobile, and online environments. It probes the model's capacity to interpret screenshots, locate interactive elements, and execute correct action sequences to complete user instructions. Use when the user wants to benchmark on Screenspot, Mind2Web, AITW, MiniWob, or asks about evaluating this task. Reports Zero-shot grounding accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.17465
  bibtex_key: lin2024showui
  confidence: high
---

# showui-gui-eval

> ShowUI: One Vision-Language-Action Model for GUI Visual Agent — Kevin Qinghong Lin et al. (2024) (arXiv:2411.17465, 2024)

## What this evaluates

Evaluates a vision-language-action model's ability to perform GUI visual grounding and task-oriented navigation across web, mobile, and online environments. It probes the model's capacity to interpret screenshots, locate interactive elements, and execute correct action sequences to complete user instructions.

## Datasets

- **Screenspot** — total ?; splits: test (-1)
- **Mind2Web** — total ?; splits: test (-1)
- **AITW** — total ?; splits: test (-1)
- **MiniWob** — total ?; splits: test (-1)

## Metrics

- `Zero-shot grounding accuracy` **(primary)** — range: percent
  - Percentage of correctly grounded text or icon elements in a screenshot, evaluated separately for text and icon tracks.
- `Element Accuracy (Ele.Acc)` — range: percent
  - Percentage of correctly identified interactive elements in a navigation step.
- `Operation F1 (Op.F1)` — range: percent
  - F1 score of the predicted operation type (e.g., click, type) against the ground truth.
- `Step Success Rate (Step.SR)` — range: percent
  - Percentage of navigation steps where the predicted action exactly matches the ground truth action.
- `Overall Accuracy` — range: percent
  - Percentage of tasks completed successfully from start to finish.

## Input / output format

**Input**: RGB screenshot of a GUI interface paired with a natural language instruction or query.

**Output**: For grounding: bounding box coordinates or element identifiers for the target UI component. For navigation: a sequence of discrete actions (e.g., click, type, scroll) formatted as JSON or structured tokens.

## Scoring recipe

```python
def score_grounding(pred_boxes, gt_boxes):
    correct = sum(1 for p, g in zip(pred_boxes, gt_boxes) if iou(p, g) > 0.5)
    return correct / len(gt_boxes) * 100

def score_navigation(pred_actions, gt_actions):
    ele_acc = sum(1 for p, g in zip(pred_actions, gt_actions) if p.element == g.element) / len(gt_actions) * 100
    op_f1 = f1_score([g.op for g in gt_actions], [p.op for p in pred_actions])
    step_sr = sum(1 for p, g in zip(pred_actions, gt_actions) if p == g) / len(gt_actions) * 100
    return ele_acc, op_f1, step_sr
```

## Common pitfalls

- Grounding accuracy is reported separately for text and icon tracks; averaging them without tracking the split can misrepresent performance.
- Navigation metrics evaluate different granularities; Step.SR requires exact match of both element and operation, making it stricter than Ele.Acc or Op.F1.
- MiniWob evaluation strictly follows a 35-task split; deviating from this split breaks comparability with reported baselines.

## Evidence (verbatim from paper)

> In Tab.[4] for web navigation, we found that: (i) Instruction-tuning has a significant effect, brings 4.6% Avg. Step SR. boost over Qwen2-VL-2B. Notably, ShowUI-2B’s zero-shot yield comparable with SeeClick-9.6B which has pretrained and fine-tuning, and achieves relatively high Op. F1 (80%+).

## Citation

```bibtex
@misc{lin2024showui,
  title={ShowUI: One Vision-Language-Action Model for GUI Visual Agent},
  author={Kevin Qinghong Lin et al. (2024)},
  year={2024},
  note={arXiv:2411.17465}
}
```

- arXiv: 2411.17465