Gui Grounding Agent Eval

---
name: gui-grounding-agent-eval
description: Evaluates a GUI agent's ability to precisely locate UI elements (grounding) and execute multi-step tasks in real-world desktop/web environments. It probes spatial reasoning, text/icon matching, and long-horizon planning robustness without lookahead. Use when the user wants to benchmark on ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, OSWorld, WindowsAgentArena, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.05791
  bibtex_key: yang2025gta1
  confidence: high
---

# gui-grounding-agent-eval

> GTA1: GUI Test-time Scaling Agent — Yan Yang et al. (2025) (arXiv:2507.05791, 2025)

## What this evaluates

Evaluates a GUI agent's ability to precisely locate UI elements (grounding) and execute multi-step tasks in real-world desktop/web environments. It probes spatial reasoning, text/icon matching, and long-horizon planning robustness without lookahead.

## Datasets

- **ScreenSpot-Pro** — total ?; splits: test (-1)
- **ScreenSpot-V2** — total ?; splits: test (-1)
- **OSWorld-G** — total ?; splits: test (-1)
- **OSWorld** — total ?; splits: test (-1)
- **WindowsAgentArena** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of test instances where the predicted click coordinate falls within the target UI element's bounding box or matches the ground truth coordinate exactly.
- `task success rate` — range: percent
  - Percentage of multi-step agent tasks that are fully completed successfully in the dynamic environment, judged by a separate model (o3).

## Input / output format

**Input**: GUI screen capture (image), task instruction text, and optionally a chain-of-thought reasoning trace.

**Output**: For grounding: a single (x, y) coordinate. For agent tasks: a sequence of discrete actions (e.g., click, type, scroll) executed step-by-step.

## Scoring recipe

```python
def score_grounding(pred, gold):
    correct = sum(1 for p, g in zip(pred, gold) if is_within_target(p, g))
    return correct / len(gold) * 100

def score_agent(pred_actions, gt_tasks, judge):
    success = sum(1 for p, t in zip(pred_actions, gt_tasks) if judge.evaluate(t, p) == 'success')
    return success / len(gt_tasks) * 100
```

## Common pitfalls

- Grounding accuracy relies on a specific coordinate-matching threshold; bounding box regression metrics are explicitly avoided in favor of click rewards.
- Task success rate is heavily dependent on the external planner (o3) and judge model, so it measures the full agent pipeline rather than the grounding model alone.
- OSWorld-Verified is a distinct, updated variant of OSWorld; using the original split will yield different results.

## Evidence (verbatim from paper)

> We evaluate our method on two sets of benchmarks: i) GUI Grounding, where we use ScreenSpot-Pro [3], ScreenSpot-V2 [4, 5], and OSWorld-G [11] datasets, evaluating by the metric of accuracy; ii) Agent Task Execution, where we use OSWorld [11] and WindowsAgentArena [33] benchmarks, measuring performance by task success rate.

## Citation

```bibtex
@misc{yang2025gta1,
  title={GTA1: GUI Test-time Scaling Agent},
  author={Yan Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.05791}
}
```

- arXiv: 2507.05791