Web Agent Benchmark Eval

---
name: web-agent-benchmark-eval
description: Evaluates long-horizon web navigation and information-seeking capabilities. Probes the agent's ability to formulate search queries, browse multiple web pages, synthesize information from diverse sources, and answer complex multi-step questions. Use when the user wants to benchmark on BrowseComp-en, BrowseComp-zh, GAIA, WebWalkerQA, FRAMES, XBench-DeepSearch, HLE, or asks about evaluating this task. Reports Avg@4 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.06501
  bibtex_key: liu2025webexplorer
  confidence: high
---

# web-agent-benchmark-eval

> WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents — Junteng Liu et al. (2025) (arXiv:2509.06501, 2025)

## What this evaluates

Evaluates long-horizon web navigation and information-seeking capabilities. Probes the agent's ability to formulate search queries, browse multiple web pages, synthesize information from diverse sources, and answer complex multi-step questions.

## Datasets

- **BrowseComp-en** — total ?; splits: (unstated)
- **BrowseComp-zh** — total ?; splits: (unstated)
- **GAIA** — total 103; splits: val (103); HF `gaia-benchmark/GAIA`
- **WebWalkerQA** — total ?; splits: (unstated)
- **FRAMES** — total ?; splits: (unstated)
- **XBench-DeepSearch** — total ?; splits: (unstated)
- **HLE** — total ?; splits: (unstated)

## Metrics

- `Avg@4 Accuracy` **(primary)** — range: percent
  - Average correctness score across up to 4 independent agent attempts per query. Each attempt is judged by an LLM-as-Judge (DeepSeek-V3) for factual correctness, and the mean of the binary success scores is reported as a percentage.

## Input / output format

**Input**: Natural language information-seeking query or task. The agent interacts with a unified scaffold providing two tools: 'search' (returns top-10 Google results with title, URL, snippet) and 'browse' (fetches full page content via Jina and answers via Gemini 2.5 Flash).

**Output**: Sequential tool calls (search/browse) and final answer text. The evaluation extracts the final answer from the agent's trajectory for LLM-as-Judge scoring.

## Scoring recipe

```python
def compute_avg4_accuracy(predictions, gold):
    scores = []
    for attempt in predictions[:4]:
        answer = extract_final_answer(attempt)
        is_correct = llm_judge_evaluate(answer, gold) # Uses DeepSeek-V3
        scores.append(1.0 if is_correct else 0.0)
    return (sum(scores) / len(scores)) * 100
```

## Common pitfalls

- Avg@4 averages up to 4 attempts per query, not just the best single attempt or a single run.
- Proprietary models (OpenAI-o3, Claude-4-Sonnet, Kimi-K2) were evaluated under the authors' unified search/browse scaffold, not their native interfaces, to ensure fair comparison.
- LLM-as-Judge uses DeepSeek-V3 specifically; results may vary significantly with different judge models or prompts.

## Evidence (verbatim from paper)

> We report all scores on these benchmarks using Avg@4 and employ LLM-as-Judge evaluation (Liu et al., 2023) with DeepSeek-V3 (Liu et al., 2024) following previous work (Li et al., 2025a; Tao et al., 2025).

## Citation

```bibtex
@misc{liu2025webexplorer,
  title={WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents},
  author={Junteng Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.06501}
}
```

- arXiv: 2509.06501