Guicourse Gui Nav Eval

---
name: guicourse-gui-nav-eval
description: Evaluates vision-language models' ability to navigate graphical user interfaces by predicting correct action types and precise screen coordinates. It probes OCR, pixel-level grounding, and multi-step task planning across web and mobile environments. Use when the user wants to benchmark on GUIAct, Mind2Web, AITW, or asks about evaluating this task. Reports StepSR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.11317
  bibtex_key: chen2024guicourse
  confidence: high
---

# guicourse-gui-nav-eval

> GUICourse: From General Vision Language Models to Versatile GUI Agents — Chen et al. (2024) (arXiv:2406.11317, 2024)

## What this evaluates

Evaluates vision-language models' ability to navigate graphical user interfaces by predicting correct action types and precise screen coordinates. It probes OCR, pixel-level grounding, and multi-step task planning across web and mobile environments.

## Datasets

- **GUIAct** — total 4400; splits: test (4400)
- **Mind2Web** — total 2000; splits: test (2000)
- **AITW** — total ?; splits: test (-1)

## Metrics

- `StepSR` **(primary)** — range: percent
  - Average success rate across all actions in a trajectory. Calculated as the mean of per-action success rates.
- `Type EM` — range: percent
  - Accuracy of predicting the correct action type (e.g., click, swipe, scroll).
- `Cli.Acc` — range: percent
  - Average success rate specifically for 'click' and 'tap' actions, reflecting pixel-level grounding ability.
- `Ele.Acc` — range: percent
  - Accuracy of predicting correct position parameters for actions, used primarily on Mind2Web.

## Input / output format

**Input**: Screenshot/image of the GUI interface and a natural language instruction (optionally with multi-turn dialogue history).

**Output**: A structured action command specifying the action type (e.g., 'click', 'swipe') and its target coordinates (e.g., <point>x y</point> or bounding box).

## Scoring recipe

```python
def compute_metrics(pred_traj, gold_traj):
    n = len(gold_traj)
    type_em = sum(1 for p, g in zip(pred_traj, gold_traj) if p['type'] == g['type']) / n
    click_steps = [g for g in gold_traj if g['type'] in ['click', 'tap']]
    cli_acc = sum(1 for p, g in zip(pred_traj, gold_traj) if g['type'] in ['click', 'tap'] and iou(p['bbox'], g['bbox']) > 0.5) / max(1, len(click_steps))
    step_sr = sum(1 for p, g in zip(pred_traj, gold_traj) if p['type'] == g['type'] and iou(p['bbox'], g['bbox']) > 0.5) / n
    return type_em, cli_acc, step_sr
```

## Common pitfalls

- Static evaluation assumes a single golden trajectory, causing 'false errors' when multiple valid action sequences exist for the same instruction.
- Performance heavily depends on input resolution; models trained at 448x448 significantly underperform compared to 1344x1344 versions on fine-grained grounding tasks.
- Evaluations on AITW and Mind2Web use processed subsets from SeeClick, excluding the 'General' column for AITW to avoid data leakage with the training GUIAct dataset.

## Evidence (verbatim from paper)

> We use the type exact match score (Type EM), click accuracy (Cli. Acc), and step success rate (StepSR) as metrics. Type EM is the accuracy of the actions’ names. Cli.Acc is the average score of the success rate of the "click" and "tap" actions, which reflects the grounding ability. StepSR is the average score of all the actions’ success rates.

## Citation

```bibtex
@misc{chen2024guicourse,
  title={GUICourse: From General Vision Language Models to Versatile GUI Agents},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.11317}
}
```

- arXiv: 2406.11317