Multiui Eval

---
name: multiui-eval
description: Evaluates multimodal models' ability to understand and interact with complex webpage UIs, perform text-rich visual grounding, and generalize to OCR and document understanding tasks. Use when the user wants to benchmark on VisualWebBench, Mind2Web, DocVQA, ChartQA, or asks about evaluating this task. Reports element accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13824
  bibtex_key: liu2024multiui
  confidence: high
---

# multiui-eval

> Harnessing Webpage UIs for Text-Rich Visual Understanding — Liu et al. (2024) (arXiv:2410.13824, 2024)

## What this evaluates

Evaluates multimodal models' ability to understand and interact with complex webpage UIs, perform text-rich visual grounding, and generalize to OCR and document understanding tasks.

## Datasets

- **VisualWebBench** — total ?; splits: test (-1)
- **Mind2Web** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)

## Metrics

- `element accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted UI elements or actions out of the total ground truth instances. Calculated as (correct predictions / total instances) * 100.
- `accuracy` — range: percent
  - Standard exact-match or fuzzy-match accuracy for VQA and grounding tasks, depending on the specific benchmark protocol.

## Input / output format

**Input**: High-resolution UI screenshots processed via dynamic patching, paired with text-based multimodal instructions or questions.

**Output**: Text responses containing answers, bounding box coordinates for grounding tasks, or action sequences for agent tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if normalize(pred) == normalize(gold_item):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Baselines were re-implemented with identical training data to control for backbone differences, so direct comparison with original checkpoint results is invalid.
- Dynamic high-resolution image processing splits images into patches; evaluation consistency requires replicating this exact patching strategy.
- Two-stage training means final evaluation reflects both GUI-specific knowledge and general multimodal capabilities, not just web UI understanding.

## Evidence (verbatim from paper)

> resulting in a 7.3M-sample dataset that enables significant gains in web UI tasks (48% improvement on VisualWebBench, 19.1% boost in element accuracy on Mind2Web) and unexpected generalization to non-UI domains like document understanding, OCR, and chart interpretation, demonstrating the broad utility of web UIs for text-rich visual understanding.

## Citation

```bibtex
@misc{liu2024multiui,
  title={Harnessing Webpage UIs for Text-Rich Visual Understanding},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.13824}
}
```

- arXiv: 2410.13824