Minimwob Eval

---
name: minimwob-eval
description: Tests a visual GUI agent's ability to complete web automation tasks by interacting with simplified web environments based on screenshots. It measures task completion success rates across various interactive web widgets. Use when the user wants to benchmark on MiniWob, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.10935
  bibtex_key: cheng2024seeclick
  confidence: high
---

# minimwob-eval

> SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Cheng et al. (2024) (arXiv:2401.10935, 2024)

## What this evaluates

Tests a visual GUI agent's ability to complete web automation tasks by interacting with simplified web environments based on screenshots. It measures task completion success rates across various interactive web widgets.

## Datasets

- **MiniWob** — total ?; splits: train (2800), test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Mean success rate computed over all MiniWob tasks, where each task's success rate is averaged over 50 random seeds.

## Input / output format

**Input**: Screenshot of a web interface.

**Output**: Action sequence (click coordinates, typing text).

## Scoring recipe

```python
task_scores = []
for task in tasks:
    seed_scores = [env.step(action) for action in model.predict(task, seed=s) for s in range(50)]
    task_scores.append(mean(seed_scores))
return mean(task_scores) * 100
```

## Common pitfalls

- Evaluation task sets vary across methods; fairness requires reporting only on overlapping tasks.
- Dynamic webpage layouts and element positions significantly impact performance, making seed averaging crucial.

## Evidence (verbatim from paper)

> We compute the success rate over 50 random seeds for each task, and then compute the mean over all MiniWob tasks as the final score.

## Citation

```bibtex
@misc{cheng2024seeclick,
  title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2401.10935}
}
```

- arXiv: 2401.10935