Miniwob Wge Eval

---
name: miniwob-wge-eval
description: Evaluates an agent's ability to navigate and interact with semi-structured web interfaces to complete goal-directed tasks. It probes relational reasoning over DOM trees, handling of natural language instructions, and sample efficiency in sparse-reward reinforcement learning settings. Use when the user wants to benchmark on MiniWoB, MiniWoB++, Alaska, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1802.08802
  bibtex_key: liu2018wge
  confidence: high
---

# miniwob-wge-eval

> Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration — Liu et al. (2018) (arXiv:1802.08802, 2018)

## What this evaluates

Evaluates an agent's ability to navigate and interact with semi-structured web interfaces to complete goal-directed tasks. It probes relational reasoning over DOM trees, handling of natural language instructions, and sample efficiency in sparse-reward reinforcement learning settings.

## Datasets

- **MiniWoB** — total 40; splits: test (-1)
- **MiniWoB++** — total ?; splits: test (-1)
- **Alaska** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - The percentage of test episodes that achieve a reward of +1. Partial rewards are explicitly disabled, making this metric a linear scaling of the average reward.

## Input / output format

**Input**: A visual state (160×210px or 375×667px), DOM tree access via Selenium, and a text goal (structured or natural language).

**Output**: A sequence of environment actions (click DOM elements, type strings) until task completion or step limit.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    successes = 0
    for pred, goal in zip(predictions, gold):
        reward = env.step_sequence(pred)
        if reward == 1.0:
            successes += 1
    return (successes / len(predictions)) * 100
```

## Common pitfalls

- The benchmark explicitly filters out tasks requiring specialized reasoning (e.g., computing angles, algebra), so results do not generalize to all web tasks.
- Partial rewards are disabled for consistency, which fundamentally changes the reward landscape compared to the original MiniWoB and Alaska benchmarks.
- The Alaska benchmark uses a surrogate JavaScript backend and clamped dates, meaning performance may not transfer to live production websites.

## Evidence (verbatim from paper)

> Evaluation metric. We report success rate: the percentage of test episodes with reward +1 . Since we have removed partial rewards, success rate is a linear scaling of the average reward, and is equivalent to the definition of success rate in Shi et al. (2017).

## Citation

```bibtex
@misc{liu2018wge,
  title={Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration},
  author={Liu et al. (2018)},
  year={2018},
  note={arXiv:1802.08802}
}
```

- arXiv: 1802.08802