Mind2web Live Eval

---
name: mind2web-live-eval
description: Evaluates a GUI agent's ability to perform web browsing tasks using either HTML tree or image inputs. It measures the agent's capacity to navigate websites and complete user intents across diverse web interfaces. Use when the user wants to benchmark on Mind2Web-Live, or asks about evaluating this task. Reports task success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10387
  bibtex_key: xie2025mirage1
  confidence: high
---

# mind2web-live-eval

> Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills — Xie et al. (2025) (arXiv:2506.10387, 2025)

## What this evaluates

Evaluates a GUI agent's ability to perform web browsing tasks using either HTML tree or image inputs. It measures the agent's capacity to navigate websites and complete user intents across diverse web interfaces.

## Datasets

- **Mind2Web-Live** — total ?; splits: test (-1)

## Metrics

- `task success rate` **(primary)** — range: percent
  - Percentage of tasks completed successfully out of the total number of tasks evaluated.
- `micro-completion rate` — range: percent
  - Measures the proportion of key nodes completed across all tasks.

## Input / output format

**Input**: Web page observations provided as either HTML tree or screenshots.

**Output**: Action selection (e.g., click, type, scroll) to interact with the web interface.

## Scoring recipe

```python
success = 1 if task goal is met, else 0.
SR = mean(success) * 100.
CR = (total key nodes completed / total key nodes) * 100.
```

## Common pitfalls

- Micro-completion rate aggregates across tasks, which can mask per-task failure patterns and overstate agent capability.
- HTML tree vs image input modes significantly affect baseline performance and should be matched carefully when comparing methods.

## Evidence (verbatim from paper)

> For Web, we employ Mind2Web-Live as the evaluation benchmark. We use task success rate and micro-completion rate (measures the proportion of key nodes completed in all tasks) as evaluation metrics.

## Citation

```bibtex
@misc{xie2025mirage1,
  title={Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.10387}
}
```

- arXiv: 2506.10387