Screen Parsing Eval

---
name: screen-parsing-eval
description: Evaluates a model's ability to detect, localize, and semantically label all interactable UI elements on a clean screenshot. It probes fine-grained spatial reasoning, handling of dense layouts, and UI semantics understanding. Use when the user wants to benchmark on GUI-360°-Bench, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.04307
  bibtex_key: mu2025gui360
  confidence: high
---

# screen-parsing-eval

> GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents — Jian Mu et al. (2025) (arXiv:2511.04307, 2025)

## What this evaluates

Evaluates a model's ability to detect, localize, and semantically label all interactable UI elements on a clean screenshot. It probes fine-grained spatial reasoning, handling of dense layouts, and UI semantics understanding.

## Datasets

- **GUI-360°-Bench** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall computed per image via greedy bipartite matching (IoU > 0.5), then macro-averaged across all images.
- `mean IoU` — range: [0, 1]
  - Average intersection-over-union score for matched element pairs per image, macro-averaged across the benchmark.
- `semantic name accuracy` — range: [0, 1]
  - Cosine similarity between sentence embeddings of predicted and ground-truth element names for matched pairs, macro-averaged.

## Input / output format

**Input**: Clean screenshot of the GUI state.

**Output**: Set of interactable UI elements, each consisting of a semantic name (string) and a bounding box (2D coordinates).

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    matched_pairs = []
    for pred, gt in zip(predictions, ground_truths):
        pairs = greedy_match(pred, gt, iou_threshold=0.5)
        matched_pairs.append(pairs)
    precisions, recalls, ious, sims = [], [], [], []
    for pairs in matched_pairs:
        if not pairs: continue
        precisions.append(len(pairs)/len(pred))
        recalls.append(len(pairs)/len(gt))
        ious.append(mean_iou(pairs))
        sims.append(mean_text_sim(pairs))
    return macro_avg(precisions), macro_avg(recalls), macro_avg(ious), macro_avg(sims)
```

## Common pitfalls

- General-purpose VLMs exhibit very low recall, missing many elements due to dense layouts and occlusions.
- Metrics are computed per-image then macro-averaged; failing to average per-image before aggregation will skew results toward larger images.

## Evidence (verbatim from paper)

> We measure parsing quality along three complementary axes: (i) element detection accuracy (precision / recall / F1), (ii) localization quality (mean IoU on matched pairs), and (iii) semantic name accuracy (average text embedding similarity on matched pairs).

## Citation

```bibtex
@misc{mu2025gui360,
  title={GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents},
  author={Jian Mu et al. (2025)},
  year={2025},
  note={arXiv:2511.04307}
}
```

- arXiv: 2511.04307