Screen Spot Eval

---
name: screen-spot-eval
description: This evaluation probes a vision-language model's ability to localize specific UI elements within graphical user interfaces based on natural language instructions. It tests precise coordinate prediction and cross-resolution generalization across mobile, desktop, and web platforms. Use when the user wants to benchmark on ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23491
  bibtex_key: hsieh2025zonui
  confidence: high
---

# screen-spot-eval

> ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding — Hsieh et al. (2025) (arXiv:2506.23491, 2025)

## What this evaluates

This evaluation probes a vision-language model's ability to localize specific UI elements within graphical user interfaces based on natural language instructions. It tests precise coordinate prediction and cross-resolution generalization across mobile, desktop, and web platforms.

## Datasets

- **ScreenSpot** — total 1272; splits: test (1272)
- **ScreenSpot-v2** — total 1272; splits: test (1272)
- **ScreenSpot-Pro** — total 1272; splits: test (1272)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Success is defined by whether the predicted coordinates fall within the annotated bounding box. Accuracy is calculated as the percentage of correctly grounded tasks out of the total instances.

## Input / output format

**Input**: A GUI screenshot image paired with a natural language instruction describing the target UI element to locate.

**Output**: Predicted bounding box coordinates (e.g., [x_min, y_min, x_max, y_max]) for the target element.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_boxes):
    correct = 0
    for pred_box, gold_box in zip(predictions, gold_boxes):
        if is_within(pred_box, gold_box):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Confusing ScreenSpot with ScreenSpot-v2, which contains corrected annotations and clarified instructions, leading to different baseline scores.
- Assuming high accuracy on standard-resolution screenshots implies robustness on ScreenSpot-Pro, which features dense professional software layouts with significantly smaller UI targets.

## Evidence (verbatim from paper)

> On ScreenSpot, ZonUI-3B achieves an accuracy of 84.9%, and on the cleaned ScreenSpot-v2, it reaches 86.4%, setting a new benchmark among all sub-4B models.

## Citation

```bibtex
@misc{hsieh2025zonui,
  title={ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding},
  author={Hsieh et al. (2025)},
  year={2025},
  note={arXiv:2506.23491}
}
```

- arXiv: 2506.23491