Molmoweb Eval

---
name: molmoweb-eval
description: Evaluates the capability of vision-language web agents to navigate live websites and complete complex, multi-step tasks using only screenshot inputs. It probes GUI perception, action grounding, and long-horizon planning under real-world web constraints. Use when the user wants to benchmark on WebVoyager, Online-Mind2Web, DeepShop, WebTailBench, ScreenSpot, ScreenSpot v2, or asks about evaluating this task. Reports pass@k accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08516
  bibtex_key: gupta2026molmoweb
  confidence: high
---

# molmoweb-eval

> MolmoWeb: Open Visual Web Agent and Open Data for the Open Web — Gupta et al. (2026) (arXiv:2604.08516, 2026)

## What this evaluates

Evaluates the capability of vision-language web agents to navigate live websites and complete complex, multi-step tasks using only screenshot inputs. It probes GUI perception, action grounding, and long-horizon planning under real-world web constraints.

## Datasets

- **WebVoyager** — total ?; splits: test (-1)
- **Online-Mind2Web** — total ?; splits: test (-1)
- **DeepShop** — total ?; splits: test (-1)
- **WebTailBench** — total ?; splits: test (-1)
- **ScreenSpot** — total ?; splits: test (-1)
- **ScreenSpot v2** — total ?; splits: test (-1)

## Metrics

- `pass@k accuracy` **(primary)** — range: [0, 1]
  - Estimates the probability that at least one of k independent rollouts succeeds. Computed as 1 - binom(m-c, k) / binom(m, k), where m is the number of collected rollouts per task and c is the number of successful rollouts.
- `task success rate` — range: [0, 1]
  - Binary success indicator per task (1 if completed within max steps and retry budget, 0 otherwise). Reported as average across 3-5 runs.

## Input / output format

**Input**: Screenshot of the current webpage state and the natural language task instruction.

**Output**: Sequential actions (e.g., click coordinates, scroll commands, text input) or final answer text.

## Scoring recipe

```python
def compute_pass_k(successes, k, m=5):
    c = sum(successes)
    if c == m:
        return 1.0
    if m - c >= k:
        return 0.0
    from math import comb
    return 1.0 - comb(m - c, k) / comb(m, k)

# Per task
successes = [judge.evaluate(rollout) for rollout in rollouts]
pass_k = compute_pass_k(successes, k)
# Aggregate
mean_pass_k = sum(pass_k for task in tasks) / len(tasks)
```

## Common pitfalls

- Time-sensitive tasks require date patching to avoid failures; unpatched dates cause immediate task failure.
- LLM-as-a-judge variability affects cross-benchmark comparability, especially when judges differ across benchmarks.
- Step limits (30 vs 100) and retry budgets (10) significantly impact reported scores and must be strictly enforced.

## Evidence (verbatim from paper)

> Because some tasks are time-sensitive, we change dates in outdated requests (e.g., find a flight on August 5, 2025) to be meaningful for the task across all benchmarks. For each benchmark and model, we run 3-5 evaluations up to 100 steps and report the average score across runs. If a model does not complete the task by the maximum number of steps, it is considered a failure. As environment errors occasionally occur, we allow up to 10 retries per trajectory; tasks that do not complete within this budget are also marked as failures. ... To get an unbiased, low variance estimate of pass@k, we collect m>k rollouts per task and compute the estimate: pass@k = 1 - binom(m-c, k)/binom(m, k), where c is the number of successful rollouts among the m attempts.

## Citation

```bibtex
@misc{gupta2026molmoweb,
  title={MolmoWeb: Open Visual Web Agent and Open Data for the Open Web},
  author={Gupta et al. (2026)},
  year={2026},
  note={arXiv:2604.08516}
}
```

- arXiv: 2604.08516