Webgym Eval

---
name: webgym-eval
description: This benchmark evaluates visual web agents on their ability to navigate complex, multi-step tasks across diverse websites and domains. It specifically probes long-horizon interaction, information extraction, and out-of-distribution generalization to unseen websites. Use when the user wants to benchmark on WebGym, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02439
  bibtex_key: bai2026webgym
  confidence: high
---

# webgym-eval

> WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks — Hao Bai et al. (2026) (arXiv:2601.02439, 2026)

## What this evaluates

This benchmark evaluates visual web agents on their ability to navigate complex, multi-step tasks across diverse websites and domains. It specifically probes long-horizon interaction, information extraction, and out-of-distribution generalization to unseen websites.

## Datasets

- **WebGym** — total 293259; splits: train (292092), test (1167)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of tasks where the agent's trajectory satisfies all criteria in the task-specific rubric, as judged by an LLM on keypoint screenshots and the final answer. A trajectory receives a score of 1 if all criteria are met, and 0 otherwise.

## Input / output format

**Input**: Task description, target website URL, and a sequence of visual screenshots (observations) captured during the agent's interaction.

**Output**: A sequence of web actions (e.g., clicks, scrolls, typing) followed by a final answer or completion signal.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, task in zip(predictions, gold):
        keypoints = extract_keypoints(pred.screenshots)
        rubric = task.rubric
        if all(judge_criterion(c, keypoints, pred.final_answer) for c in rubric):
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Reference answers are rarely available for open-ended web tasks; evaluation relies on LLM-judged rubric satisfaction rather than exact string matching.
- The rubric-based evaluator can be overly strict, leading to lower recall compared to human judgment, which may penalize partially correct or exploratory trajectories.
- Trajectory length analysis filters out steps beyond 30 for fair comparison, which may obscure performance on extremely long-horizon tasks.

## Evidence (verbatim from paper)

> We validate the evaluator against human annotations on 80 collected trajectories sampled uniformly across difficulty levels (10 trajectories per level from difficulties 1-6, and around 5 trajectories per level from difficulties 7-10), where nearly half of the trajectories in each difficulty are marked as correct by the WebGym evaluator to balance the set, and find that rubric-guided evaluation improves agreement over task-only judging, which increases accuracy and precision for all evaluator models being tested (GPT-4o, Qwen3-VL-8B-Instruct, and Gemma3-27B-it).

## Citation

```bibtex
@misc{bai2026webgym,
  title={WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks},
  author={Hao Bai et al. (2026)},
  year={2026},
  note={arXiv:2601.02439}
}
```

- arXiv: 2601.02439