Visualwebbench Eval

---
name: visualwebbench-eval
description: Evaluates multimodal LLMs' ability to understand web pages and ground UI elements. It probes capabilities across seven subtasks including image captioning, web question answering, OCR, element/action grounding, and action prediction. Use when the user wants to benchmark on VisualWebBench, or asks about evaluating this task. Reports Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.05955
  bibtex_key: liu2024visualwebbench
  confidence: high
---

# visualwebbench-eval

> VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? — Liu et al. (2024) (arXiv:2404.05955, 2024)

## What this evaluates

Evaluates multimodal LLMs' ability to understand web pages and ground UI elements. It probes capabilities across seven subtasks including image captioning, web question answering, OCR, element/action grounding, and action prediction.

## Datasets

- **VisualWebBench** — total 1500; splits: test (1500)

## Metrics

- `Average Score` **(primary)** — range: [0, 100]
  - Arithmetic mean of accuracy percentages across seven subtasks (Caption, WebQA, HeadOCR, OCR, Element Grounding, Action Prediction, Action Grounding), scaled to a maximum of 100.

## Input / output format

**Input**: High-resolution webpage screenshot (1280px width) paired with a natural language prompt or question.

**Output**: Text response for captioning/QA/prediction tasks; selected option index for multiple-choice grounding/OCR tasks; or bounding box/point coordinates for free-form grounding tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def compute_average_score(task_accuracies):
    return sum(task_accuracies) / len(task_accuracies)
```

## Common pitfalls

- Models limited to 448x448 input resolution struggle with 1280px wide screenshots, missing fine-grained details.
- GUI-specialized MLLMs often suffer catastrophic forgetting on general instruction-following after GUI grounding pretraining.
- Grounding evaluation uses two distinct protocols: multiple-choice selection (default) vs. free-form coordinate prediction (REC), yielding vastly different scores.

## Evidence (verbatim from paper)

> GPT-4V achieves an average score of only 64.6 on VisualWebBench, leaving ample room for improvement.

## Citation

```bibtex
@misc{liu2024visualwebbench,
  title={VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2404.05955}
}
```

- arXiv: 2404.05955