Screen Spot Grounding Eval

---
name: screen-spot-grounding-eval
description: Evaluates a model's ability to locate specific GUI elements from a screenshot given a text instruction. It measures both coarse localization accuracy and fine-grained bounding box overlap across desktop, mobile, and web platforms. Use when the user wants to benchmark on ScreenSpot, or asks about evaluating this task. Reports grounding accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.23218
  bibtex_key: wu2024osatlas
  confidence: high
---

# screen-spot-grounding-eval

> OS-ATLAS: A Foundation Action Model for Generalist GUI Agents — Zhiyong Wu et al. (2024) (arXiv:2410.23218, 2024)

## What this evaluates

Evaluates a model's ability to locate specific GUI elements from a screenshot given a text instruction. It measures both coarse localization accuracy and fine-grained bounding box overlap across desktop, mobile, and web platforms.

## Datasets

- **ScreenSpot** — total ?; splits: test (-1)

## Metrics

- `grounding accuracy` **(primary)** — range: [0, 1]
  - Fraction of test instances where the predicted bounding box falls entirely within the ground truth bounding box.
- `IoU` — range: [0, 1]
  - Intersection over Union: area of overlap between predicted and ground truth bounding boxes divided by their union area.

## Input / output format

**Input**: GUI screenshot image paired with a natural language instruction (optionally pre-processed by a planner model in the Grounding Mode Setting).

**Output**: Bounding box coordinates (x_min, y_min, x_max, y_max) or a point coordinate indicating the target GUI element.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    accs, ious = [], []
    for pred, gold in zip(predictions, golds):
        ix1, iy1 = max(pred[0], gold[0]), max(pred[1], gold[1])
        ix2, iy2 = min(pred[2], gold[2]), min(pred[3], gold[3])
        inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
        union = (pred[2]-pred[0])*(pred[3]-pred[1]) + (gold[2]-gold[0])*(gold[3]-gold[1]) - inter
        ious.append(inter / union if union > 0 else 0.0)
        accs.append(1.0 if (pred[0]>=gold[0] and pred[1]>=gold[1] and pred[2]<=gold[2] and pred[3]<=gold[3]) else 0.0)
    return {'grounding_accuracy': sum(accs)/len(accs), 'iou': sum(ious)/len(ious)}
```

## Common pitfalls

- ScreenSpot contains ~11.32% annotation errors; using the raw dataset without correction (ScreenSpot-V2) may yield inaccurate baseline comparisons.
- Grounding accuracy is a coarse metric that ignores fine-grained localization errors; IoU should be reported alongside it to capture precise bounding box overlap.
- Performance varies significantly between the 'Standard Setting' (direct instruction) and 'Grounding Mode Setting' (planner-refined instruction); results are not directly comparable across settings.

## Evidence (verbatim from paper)

> We follow previous practices by using grounding accuracy on ScreenSpot, where a prediction is considered correct if the predicted location falls within the ground truth element’s bounding box. However, this metric does not capture more fine-grained grounding errors. Therefore, we also use Intersection over Union (IoU), a widely used metric for measuring localization accuracy in object detection. IoU quantifies the overlap between the predicted bounding box and the ground truth bounding box.

## Citation

```bibtex
@misc{wu2024osatlas,
  title={OS-ATLAS: A Foundation Action Model for Generalist GUI Agents},
  author={Zhiyong Wu et al. (2024)},
  year={2024},
  note={arXiv:2410.23218}
}
```

- arXiv: 2410.23218