Webvoyager Eval

---
name: webvoyager-eval
description: Measures an autonomous web agent's end-to-end task completion capability across dynamic, real-world websites. It evaluates multi-step navigation, form filling, and robustness in open-web environments. Use when the user wants to benchmark on WebVoyager, or asks about evaluating this task. Reports Success Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.16987
  bibtex_key: bhathal2025websight
  confidence: high
---

# webvoyager-eval

> WebSight: A Vision-First Architecture for Robust Web Agents — Bhathal et al. (2025) (arXiv:2508.16987, 2025)

## What this evaluates

Measures an autonomous web agent's end-to-end task completion capability across dynamic, real-world websites. It evaluates multi-step navigation, form filling, and robustness in open-web environments.

## Datasets

- **WebVoyager** — total 50; splits: test (50)

## Metrics

- `Success Rate` **(primary)** — range: percent
  - Percentage of tasks successfully completed within the allowed time limit. Calculated as (completed tasks / total tasks) * 100.

## Input / output format

**Input**: A sequence of web page screenshots, DOM/UI state, and a natural language task instruction.

**Output**: A sequence of browser actions (click, type, scroll, navigate) and a final answer or completion signal.

## Scoring recipe

```python
completed = 0
for task in tasks:
    if agent.run(task, timeout=600) == 'success':
        completed += 1
success_rate = (completed / len(tasks)) * 100
```

## Common pitfalls

- Tasks may become outdated or broken over time, requiring a filtered subset for consistent evaluation.
- Agents often fail due to infinite loops or timeouts rather than incorrect logic, masking true capability.
- Verification agents may fail to catch intermediate errors, leading to false success claims if not manually audited.

## Evidence (verbatim from paper)

> The WebVoyager benchmark is a large-scale, real-world evaluation suite designed to measure the capabilities of autonomous web agents in handling interactive tasks across dynamic websites [22]. ... WEBSIGHT achieves a Success Rate of 68% on Skyvern's filtered WebVoyager Benchmark [55].

## Citation

```bibtex
@misc{bhathal2025websight,
  title={WebSight: A Vision-First Architecture for Robust Web Agents},
  author={Bhathal et al. (2025)},
  year={2025},
  note={arXiv:2508.16987}
}
```

- arXiv: 2508.16987