Uibert Ui Understanding Eval

---
name: uibert-ui-understanding-eval
description: This evaluation probes a model's ability to understand and reason about user interface components across multiple modalities (images, text, structural metadata). It tests cross-modal alignment, component retrieval, synchronization detection, and classification tasks relevant to UI design and accessibility. Use when the user wants to benchmark on Rico, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.13731
  bibtex_key: bai2021uibert
  confidence: high
---

# uibert-ui-understanding-eval

> UIBert: Learning Generic Multimodal Representations for UI Understanding — Bai et al. (2021) (arXiv:2107.13731, 2021)

## What this evaluates

This evaluation probes a model's ability to understand and reason about user interface components across multiple modalities (images, text, structural metadata). It tests cross-modal alignment, component retrieval, synchronization detection, and classification tasks relevant to UI design and accessibility.

## Datasets

- **Rico** — total 1000000; splits: train (900000), dev (32000), test (32000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels or retrieved candidates out of the total number of instances.
- `macro-F1` — range: other
  - Unweighted mean of the F1 score calculated independently for each class, then averaged across all classes.

## Input / output format

**Input**: UI components represented as images (IMG), OCR text, and structural view hierarchy (VH) metadata. For retrieval tasks, an anchor UI/component and a search UI with candidate components. For classification/sync tasks, full UI embeddings or concatenated component embeddings.

**Output**: For retrieval: a single selected candidate component index. For classification/sync: a predicted class label (e.g., app type, icon type, sync/unsync).

## Scoring recipe

```python
def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    classes = sorted(set(gold + predictions))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return acc, sum(f1s) / len(f1s)
```

## Common pitfalls

- Web UI evaluation lacks view hierarchy (VH) components, forcing models to rely only on image and OCR text, which drastically changes the input modality compared to mobile UIs.
- Zero-shot evaluation uses the pretrained model directly without task-specific finetuning, so results test generalization rather than optimized task performance.
- App type classification results differ from prior work (ActionBert) because this paper uses the full 72k Rico dataset instead of a 43.5k subset.

## Evidence (verbatim from paper)

> Overall, prediction accuracy of all methods on the four task variations are reported in Tab. [1]. We observe that UIBert outperforms both baselines on all cases by 0.85%–9.26%, especially by a large margin on the zero-shot tasks.

## Citation

```bibtex
@misc{bai2021uibert,
  title={UIBert: Learning Generic Multimodal Representations for UI Understanding},
  author={Bai et al. (2021)},
  year={2021},
  note={arXiv:2107.13731}
}
```

- arXiv: 2107.13731