Screenspot Eval

---
name: screenspot-eval
description: Evaluates a model's ability to locate specific UI elements on screenshots based on natural language instructions. It measures how accurately the model can predict click coordinates that align with ground-truth bounding boxes across mobile, desktop, and web platforms. Use when the user wants to benchmark on ScreenSpot, or asks about evaluating this task. Reports click accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.10935
  bibtex_key: cheng2024seeclick
  confidence: high
---

# screenspot-eval

> SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Cheng et al. (2024) (arXiv:2401.10935, 2024)

## What this evaluates

Evaluates a model's ability to locate specific UI elements on screenshots based on natural language instructions. It measures how accurately the model can predict click coordinates that align with ground-truth bounding boxes across mobile, desktop, and web platforms.

## Datasets

- **ScreenSpot** — total ?; splits: test (-1); repo https://github.com/njucckevin/SeeClick

## Metrics

- `click accuracy` **(primary)** — range: percent
  - Proportion of test samples where the model's predicted location (point or center of predicted bounding box) falls within the ground-truth element bounding box.

## Input / output format

**Input**: Screenshot of a GUI interface paired with a natural language instruction describing the target element.

**Output**: A predicted 2D coordinate (point) or bounding box representing the target element location.

## Scoring recipe

```python
correct = 0
for pred_box, gt_box in zip(predictions, ground_truth):
    pred_pt = center(pred_box) if is_box(pred_box) else pred_box
    if point_in_box(pred_pt, gt_box):
        correct += 1
return correct / len(predictions) * 100
```

## Common pitfalls

- Models often struggle with non-text elements like icons/widgets, leading to artificially low scores if only text grounding is evaluated.
- Using the center of a predicted bounding box vs. a direct point prediction can yield different results; the protocol explicitly allows both but requires consistent handling.

## Evidence (verbatim from paper)

> Considering that GUI agents require clicking on the correct position, we calculate the click accuracy as the metric, defined as the proportion of test samples where the model’ predicted location falls in the groundtruth element bounding box. We take the models’ generated point or the center of the generated bounding box as the predicted click location.

## Citation

```bibtex
@misc{cheng2024seeclick,
  title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2401.10935}
}
```

- arXiv: 2401.10935