Webarena Human Trajectory Eval

---
name: webarena-human-trajectory-eval
description: Evaluates the planning and execution quality of LLM-based web agents by comparing their action trajectories against human-demonstrated gold standards. It measures recovery from deviations, action repetitiveness, step fulfillment, partial task completion, and alignment between planned and executed actions. Use when the user wants to benchmark on WebArena Human Trajectory Dataset, or asks about evaluating this task. Reports Recovery Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.12710
  bibtex_key: shahnovsky2026aiplanning
  confidence: high
---

# webarena-human-trajectory-eval

> AI Planning Framework for LLM-Based Web Agents — Shahnovsky et al. (2026) (arXiv:2603.12710, 2026)

## What this evaluates

Evaluates the planning and execution quality of LLM-based web agents by comparing their action trajectories against human-demonstrated gold standards. It measures recovery from deviations, action repetitiveness, step fulfillment, partial task completion, and alignment between planned and executed actions.

## Datasets

- **WebArena Human Trajectory Dataset** — total 794; splits: full (794); repo https://github.com/shahnovsky/WebArena-Human-Trajectory-Dataset

## Metrics

- `Recovery Rate` **(primary)** — range: [0, 1]
  - Average ratio of recoveries to deviation incidents across tasks: (1/#tasks) * sum(# recoveries in task t / # deviation incidents in task t). A recovery occurs when an agent action fulfills a subsequent human gold step after a deviation.
- `Repetitiveness Rate` — range: [0, 1]
  - Inverted proportion of consecutive duplicate actions: 1 - (1/#tasks) * sum(# repetitive actions in task t / # actions in task t). Higher values indicate fewer redundant steps.
- `Step Success Rate` — range: [0, 1]
  - Average proportion of human gold steps matched by the agent: (1/#tasks) * sum(# matched human gold steps in task t / # human gold steps in task t). Matches are determined via LLM semantic equivalence.
- `Partial Success Rate` — range: [0, 1]
  - Average proportion of completed requirements for multi-output tasks: (1/#req_tasks) * sum(# completed requirements in task t / # requirements in task t). Inapplicable to single-output tasks.
- `Element Accuracy Rate` — range: [0, 1]
  - Average proportion of planned actions that match the actually executed actions: (1/#tasks) * sum(# matching steps predicted and actual / # agent steps).

## Input / output format

**Input**: Agent execution trajectory (sequence of actions), human gold trajectory (reference sequence), task instructions, and task-specific requirements (for multi-output tasks).

**Output**: Numerical scores between 0 and 1 for each of the five trajectory quality metrics, computed via rule-based counting and LLM-as-a-judge semantic matching.

## Scoring recipe

```python
def compute_metrics(agent_traj, gold_traj, task_reqs):
    scores = {}
    for t in tasks:
        devs = count_deviations(agent_traj[t], gold_traj[t])
        recs = count_recoveries(agent_traj[t], gold_traj[t], lookahead=5)
        scores['recovery_rate'] = recs / max(devs, 1)
        reps = count_consecutive_duplicates(agent_traj[t])
        scores['repetitiveness_rate'] = 1 - (reps / len(agent_traj[t]))
        matches = llm_semantic_match(agent_traj[t], gold_traj[t])
        scores['step_success_rate'] = matches / len(gold_traj[t])
        if len(task_reqs[t]) > 1:
            completed = count_completed_requirements(agent_traj[t], task_reqs[t])
            scores['partial_success_rate'] = completed / len(task_reqs[t])
        planned = agent_traj[t].planned
        actual = agent_traj[t].actual
        scores['element_accuracy_rate'] = count_matching(planned, actual) / len(actual)
    return {k: np.mean(v) for k, v in scores.items()}
```

## Common pitfalls

- Relies on an LLM-as-a-judge for semantic equivalence, which may introduce variability or bias compared to exact string matching.
- Recovery Rate depends on a user-defined lookahead parameter to check future human steps, significantly altering scores if set too high or low.
- Partial Success Rate is explicitly inapplicable to tasks with only a single required output; applying it to such tasks yields undefined results.

## Evidence (verbatim from paper)

> Definition 4.1 (Recovery Rate). Measures how well an agent can return to the expected human-demonstrated sequence of actions after deviating from it. A deviation incident is recorded when an agent step does not directly fulfill the current human gold step. A recovery occurs when the agent performs an action that successfully fulfills a subsequent human gold step.

## Citation

```bibtex
@misc{shahnovsky2026aiplanning,
  title={AI Planning Framework for LLM-Based Web Agents},
  author={Shahnovsky et al. (2026)},
  year={2026},
  note={arXiv:2603.12710}
}
```

- arXiv: 2603.12710