Ghostui Eval

---
name: ghostui-eval
description: This benchmark evaluates vision-language models' ability to detect and predict hidden interactions in mobile user interfaces. It probes whether models can infer concealed gestures (e.g., long presses, swipes) from before-interaction screenshots and task descriptions, localize the interaction target, and anticipate the resulting UI state changes. Use when the user wants to benchmark on GhostUI, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.19258
  bibtex_key: kweon2026ghostui
  confidence: high
---

# ghostui-eval

> GhostUI: Unveiling Hidden Interactions in Mobile UI — Kweon et al. (2026) (arXiv:2601.19258, 2026)

## What this evaluates

This benchmark evaluates vision-language models' ability to detect and predict hidden interactions in mobile user interfaces. It probes whether models can infer concealed gestures (e.g., long presses, swipes) from before-interaction screenshots and task descriptions, localize the interaction target, and anticipate the resulting UI state changes.

## Datasets

- **GhostUI** — total ?; splits: train (-1), test (-1); repo https://github.com/gh0stui/ghostui

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of instances where the model correctly predicts the gesture type, including directional specifications for swipes, scrolls, and pinches.
- `IoU` — range: [0, 1]
  - Intersection over Union between the predicted bounding box and the ground truth bounding box, measuring spatial localization precision.
- `cosine similarity` — range: [-1, 1]
  - Cosine similarity between the text embedding of the model's predicted UI description and the ground truth description, computed using Google's gemini-embedding-001 model.

## Input / output format

**Input**: Before-interaction screenshot, task description, and optionally simplified view hierarchy, gesture usage patterns, and app metadata.

**Output**: Predicted gesture type and bounding box coordinates (Task 1); text description of the post-gesture UI state (Task 2).

## Scoring recipe

```python
def score_task1(pred_gesture, pred_box, gold_gesture, gold_box):
    acc = 1.0 if pred_gesture == gold_gesture else 0.0
    iou = intersection_area(pred_box, gold_box) / union_area(pred_box, gold_box)
    return acc, iou

def score_task2(pred_text, gold_text):
    pred_emb = gemini_embed_001(pred_text)
    gold_emb = gemini_embed_001(gold_text)
    return cosine_similarity(pred_emb, gold_emb)
```

## Common pitfalls

- Models often memorize app-specific patterns rather than learning generalizable gesture rules due to the app-level train/test split.
- Zero-shot models exhibit a strong bias toward predicting 'tap' gestures, frequently misclassifying double taps and long presses as simple taps.
- Spatial localization (IoU) is highly sensitive to the absence of view hierarchy data, causing drastic performance drops even if gesture classification remains stable.
- Double tap outcomes are highly context-dependent and diverse, making them consistently difficult to predict accurately even after fine-tuning.

## Evidence (verbatim from paper)

> First, classification accuracy measures whether the model correctly predicts the gesture type, including directional specifications for swipes (left, right), scrolls (up, down), and pinch gestures (zoom in, zoom out). Second, Intersection over Union (IoU) quantifies the overlap between predicted and ground truth bounding boxes to assess spatial localization precision. For [Section 4.3], we measured the model’s understanding of interaction outcomes by calculating cosine similarity between predicted UI descriptions and ground truth descriptions generated from after-interaction screenshots.

## Citation

```bibtex
@misc{kweon2026ghostui,
  title={GhostUI: Unveiling Hidden Interactions in Mobile UI},
  author={Kweon et al. (2026)},
  year={2026},
  note={arXiv:2601.19258}
}
```

- arXiv: 2601.19258