X Webagentbench Eval

---
name: x-webagentbench-eval
description: Evaluates LLM-based agents' ability to comprehend multilingual shopping instructions and successfully navigate interactive web environments across 14 languages. Use when the user wants to benchmark on X-WebAgentBench, or asks about evaluating this task. Reports Task Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15372
  bibtex_key: wang2025xwebagentbench
  confidence: high
---

# x-webagentbench-eval

> X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System — Wang et al. (2025) (arXiv:2505.15372, 2025)

## What this evaluates

Evaluates LLM-based agents' ability to comprehend multilingual shopping instructions and successfully navigate interactive web environments across 14 languages.

## Datasets

- **X-WebAgentBench** — total 2800; splits: test (2800); repo https://github.com/WPENGxs/X-WebAgentBench

## Metrics

- `Task Score` **(primary)** — range: percent
  - The percentage of tasks successfully completed by the agent within the interactive web environment, following the WebShop evaluation protocol.
- `Average Action Steps` — range: other
  - The mean number of interaction steps (e.g., Search, Click) taken by the agent per task.

## Input / output format

**Input**: Multilingual natural language instruction (in one of 14 languages) specifying a shopping goal, provided within an interactive web interface.

**Output**: Sequential web interaction actions (e.g., Search, Click) executed within the simulated e-commerce environment.

## Scoring recipe

```python
def compute_task_score(predictions, gold):
    success = sum(1 for p in predictions if p['success'])
    return (success / len(predictions)) * 100
```

## Common pitfalls

- Cross-lingual alignment methods (CLP, Self-Translate-en) often degrade performance on smaller open-source LLMs (<8B) rather than improving it.
- Translating the web environment to English provides only partial gains and fails to close the performance gap for low-resource languages.

## Evidence (verbatim from paper)

> Following WebShop (Yao et al., 2022), we use the Task Score to evaluate the performance of language agents.

## Citation

```bibtex
@misc{wang2025xwebagentbench,
  title={X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.15372}
}
```

- arXiv: 2505.15372