Screen2words Eval

---
name: screen2words-eval
description: Evaluates a model's ability to automatically generate concise, coherent language summaries of mobile UI screens by fusing visual, structural, and textual modalities. It probes multimodal representation learning and language generation capabilities in the context of human-computer interaction and UI understanding. Use when the user wants to benchmark on Screen2Words, or asks about evaluating this task. Reports BLEU-4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.03353
  bibtex_key: wang2021screen2words
  confidence: high
---

# screen2words-eval

> Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning — Wang et al. (2021) (arXiv:2108.03353, 2021)

## What this evaluates

Evaluates a model's ability to automatically generate concise, coherent language summaries of mobile UI screens by fusing visual, structural, and textual modalities. It probes multimodal representation learning and language generation capabilities in the context of human-computer interaction and UI understanding.

## Datasets

- **Screen2Words** — total 22417; splits: train (-1), val (-1), test (-1)

## Metrics

- `BLEU-4` **(primary)** — range: [0, 100]
  - Computes modified 4-gram precision between the predicted summary and the reference set, combined with a brevity penalty to penalize overly short outputs.
- `CIDEr` — range: [0, 100]
  - Measures consensus-based image description evaluation using TF-IDF weighted n-gram cosine similarity between predictions and multiple human references.
- `ROUGE-L` — range: [0, 100]
  - Computes the F1 score based on the longest common subsequence (LCS) between the predicted and reference summaries, capturing sentence-level fluency.
- `METEOR` — range: [0, 100]
  - Matches unigrams based on exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean of precision and recall with a penalty for fragmentation.

## Input / output format

**Input**: Mobile UI screen image (processed via ResNet), structural layout data from the view hierarchy, screen text, and app description.

**Output**: A single text summary phrase generated sequentially via beam search (beam size 5).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    # gold_labels: list of 5 reference phrases per screen
    scores = []
    for pred_tokens, refs in zip(predictions, gold_labels):
        # Remove UNK tokens from decoded phrase before evaluation
        pred_clean = [w for w in pred_tokens if w != '<UNK>']
        # Compute n-gram/LCS overlap against all 5 references
        bleu = compute_bleu(n=4, pred=pred_clean, refs=refs)
        cider = compute_cider(pred=pred_clean, refs=refs)
        rouge = compute_rouge_l(pred=pred_clean, refs=refs)
        meteor = compute_meteor(pred=pred_clean, refs=refs)
        scores.append((bleu + cider + rouge + meteor) / 4)
    return mean(scores) * 100  # Paper reports all metrics scaled by 100
```

## Common pitfalls

- App-wise splitting is strictly enforced; failing to keep all screens from the same app within a single split causes severe data leakage and artificially inflates test performance.
- Each screen has 5 human-annotated reference summaries; evaluation must use all 5 as the reference set, not just a single randomly sampled label.
- UNK tokens in the model's decoded output are explicitly removed before metric calculation, which alters n-gram counts and must be handled during scoring.

## Evidence (verbatim from paper)

> In this section, we report our model performance based on metrics commonly used in machine translation and image captioning tasks: BLEU [34], CIDEr [43], ROUGE-L [29], and METOER [9] (see Table 2). A higher number means better model performance for these metrics—the closer distances between the predicted and the ground truth phrases.

## Citation

```bibtex
@misc{wang2021screen2words,
  title={Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2108.03353}
}
```

- arXiv: 2108.03353