Gui Grounding Eval

---
name: gui-grounding-eval
description: Evaluates a model's ability to locate specific UI elements on screenshots based on natural language instructions. It measures both the recall of candidate generation and the precision of visual discrimination to select the correct bounding box. Use when the user wants to benchmark on MMBench-GUI, ScreenSpot-Pro, UI-Vision, ScreenSpot-v2, UI-I2E-Bench, OSWorld-G, or asks about evaluating this task. Reports Top-1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21268
  bibtex_key: [REDACTED]
  confidence: high
---

# gui-grounding-eval

> Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding — Wang et al. (2026) (arXiv:2604.21268, 2026)

## What this evaluates

Evaluates a model's ability to locate specific UI elements on screenshots based on natural language instructions. It measures both the recall of candidate generation and the precision of visual discrimination to select the correct bounding box.

## Datasets

- **MMBench-GUI** — total ?; splits: test (-1)
- **ScreenSpot-Pro** — total ?; splits: test (-1)
- **UI-Vision** — total ?; splits: test (-1)
- **ScreenSpot-v2** — total ?; splits: test (-1)
- **UI-I2E-Bench** — total ?; splits: test (-1)
- **OSWorld-G** — total ?; splits: test (-1)

## Metrics

- `Oracle@5` — range: percent
  - Percentage of samples where at least one of the generated candidates (N ≤ 5) falls within the ground-truth bounding box. Measures generator recall capability.
- `Top-1 Accuracy` **(primary)** — range: percent
  - Percentage of samples where the highest-ranked candidate selected by the Critic falls within the ground-truth bounding box. Measures end-to-end system performance.

## Input / output format

**Input**: Screenshot image and natural language instruction describing the target UI element.

**Output**: A ranked list of candidate bounding boxes (typically N candidates), with the top-ranked box selected as the final prediction.

## Scoring recipe

```python
def score(predictions, gold_boxes):
    oracle_hits = 0
    top1_hits = 0
    for pred_list, gt_box in zip(predictions, gold_boxes):
        # Oracle@5: check if any of top 5 candidates falls within gt box
        if any(candidate.contains(gt_box.center) for candidate in pred_list[:5]):
            oracle_hits += 1
        # Top-1: check if the highest-ranked candidate falls within gt box
        if pred_list[0].contains(gt_box.center):
            top1_hits += 1
    return oracle_hits / len(predictions), top1_hits / len(predictions)
```

## Common pitfalls

- Oracle@K measures generator recall, not final selection accuracy, so high Oracle@K does not guarantee high Top-1 accuracy.
- Spatial aggregation baselines (mean/median) often fail on multi-modal distributions or select invalid background regions, unlike visual critics.
- SFT training degrades visual discrimination capability, widening the gap between Oracle@K and Top-1 accuracy.

## Evidence (verbatim from paper)

> We employ two metrics to decouple the evaluation of generation coverage and discrimination precision: Oracle@K: Evaluates the Proposer’s recall capability. It calculates the percentage of samples where at least one of the generated candidates ($N\leq K$) falls within the ground-truth bounding box. Top-1 Accuracy: Evaluates the end-to-end system performance. It calculates the percentage of samples where the highest-ranked candidate selected by the Critic falls within the ground-truth bounding box.

## Citation

```bibtex
@misc{wang2026measuretwice,
  title={Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.21268}
}
```

- arXiv: 2604.21268