Screenpr Eval

---
name: screenpr-eval
description: Evaluates a model's ability to read and describe the content and layout of a GUI screenshot at a specific pointed location. It probes layout-aware screen reading, spatial reasoning, and the capacity to generate focused descriptions for mobile agent navigation. Use when the user wants to benchmark on ScreenPR, or asks about evaluating this task. Reports Content Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.19263
  bibtex_key: fan2024readanywherepointed
  confidence: high
---

# screenpr-eval

> Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding — Fan et al. (2024) (arXiv:2406.19263, 2024)

## What this evaluates

Evaluates a model's ability to read and describe the content and layout of a GUI screenshot at a specific pointed location. It probes layout-aware screen reading, spatial reasoning, and the capacity to generate focused descriptions for mobile agent navigation.

## Datasets

- **ScreenPR** — total ?; splits: test (-1)

## Metrics

- `Content Acc` **(primary)** — range: percent
  - Accuracy of the generated content description compared to human-verified ground truth descriptions.
- `Layout Acc` — range: percent
  - Accuracy of the generated layout description compared to human-verified ground truth descriptions.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for verifying whether a mobile navigation agent's action is correct or incorrect.

## Input / output format

**Input**: A GUI screenshot $S_i$ and a text prompt containing a specific point coordinate $P_i$.

**Output**: Text descriptions detailing the content and layout of the GUI region surrounding the pointed location.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_desc in zip(predictions, gold):
        if pred == gold_desc:  # Exact match or human-verified label match
            correct += 1
    return (correct / len(gold)) * 100

def compute_f1(tp, fp, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Language similarity metrics like BERTScore and ROUGE-L often fail to distinguish model performance because generated captions can be comprehensive but redundant, whereas focused screen-reading outputs are preferred by humans.
- Human evaluation preferences may diverge from automatic accuracy scores; models producing longer, more comprehensive descriptions may score lower on accuracy but be preferred by human judges.

## Evidence (verbatim from paper)

> We evaluate our ToL agent against three baselines on the ScreenPR benchmark as shown in Table [2]. The results are consistent across both human evaluation and automatic cycle consistency evaluation, showing that our ToL agent achieves the best performance in terms of content and layout description accuracy.

## Citation

```bibtex
@misc{fan2024readanywherepointed,
  title={Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding},
  author={Fan et al. (2024)},
  year={2024},
  note={arXiv:2406.19263}
}
```

- arXiv: 2406.19263