Widget Captioning Eval

---
name: widget-captioning-eval
description: This benchmark evaluates a model's ability to generate natural language descriptions for individual mobile UI elements using multimodal inputs. It probes the capability to fuse visual appearance and structural hierarchy data to produce accurate, context-aware captions for accessibility and UI understanding tasks. Use when the user wants to benchmark on Widget Captioning Dataset, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.04295
  bibtex_key: [REDACTED]
  confidence: high
---

# widget-captioning-eval

> Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements — Yang Li et al. (arXiv:2010.04295, 2020)

## What this evaluates

This benchmark evaluates a model's ability to generate natural language descriptions for individual mobile UI elements using multimodal inputs. It probes the capability to fuse visual appearance and structural hierarchy data to produce accurate, context-aware captions for accessibility and UI understanding tasks.

## Datasets

- **Widget Captioning Dataset** — total 61285; splits: train (52178), val (4548), test (4559); repo https://github.com/google-research/google-research/tree/master/widget-caption

## Metrics

- `CIDEr` **(primary)** — range: [0, 1]
  - TF-IDF weighted n-gram similarity between predicted and reference captions. Higher scores indicate better alignment with ground truth.
- `BLEU-1` — range: [0, 1]
  - Unigram precision with brevity penalty to penalize overly short predictions.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence recall-based overlap between prediction and reference.
- `METEOR` — range: [0, 1]
  - Alignment-based metric combining precision, recall, and synonymy matching with a penalty for fragmentation.
- `SPICE` — range: [0, 1]
  - Semantic parser-based metric that matches objects and relations in the predicted caption to the reference.

## Input / output format

**Input**: Visual image of a target UI element (processed via ResNet) and structural representation including widget properties, local view hierarchy, and full screen context.

**Output**: A natural language caption sequence generated from a 10,000-word vocabulary. During evaluation, any <UNK> tokens in the decoded output are removed before scoring.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # predictions: list of str, references: list of list of str
    # Strip <UNK> tokens as per paper protocol
    preds_clean = [p.replace('<UNK>', '').strip() for p in predictions]
    bleu1 = nltk.bleu_score.corpus_bleu(references, [p.split() for p in preds_clean], weights=(1,0,0,0))
    cider = cider_score.compute_score(references, preds_clean)
    rouge = rouge_score.rouge_l(references, preds_clean)
    meteor = meteor_score.compute_score(references, preds_clean)
    spice = spice_score.compute_score(references, preds_clean)
    return {'BLEU-1': bleu1, 'CIDEr': cider, 'ROUGE-L': rouge, 'METEOR': meteor, 'SPICE': spice}
```

## Common pitfalls

- Splits are app-wise, not screen-wise: all screens from a given app are kept in a single split. Models must generalize to entirely unseen apps, not just unseen UI layouts.
- Multiple ground-truth captions per element: each widget has several reference captions. Metrics are computed against the full reference set, not a single gold caption.
- UNK token removal: any <UNK> in the model's output is stripped before evaluation, which can artificially inflate n-gram overlap if not handled identically to the paper.

## Evidence (verbatim from paper)

> We report our accuracy based on BLEU (unigram and bigram), CIDEr, ROUGE-L, METOER and SPICE metrics (see Table 3). For all these metrics, a higher number means better captioning accuracy—the closer distances between the predicted and the groundtruth captions.

## Citation

```bibtex
@misc{li2020widgetcaptioning,
  title={Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements},
  author={Yang Li et al.},
  year={2020},
  note={arXiv:2010.04295}
}
```

- arXiv: 2010.04295