Odysseys Eval

---
name: odysseys-eval
description: Probes an agent's ability to perform realistic, long-horizon web navigation tasks that require sustained cross-site reasoning, context maintenance across multiple tabs, and efficient action execution. It evaluates whether models can complete complex, multi-step user journeys derived from real browsing behavior within strict step budgets. Use when the user wants to benchmark on Odysseys, or asks about evaluating this task. Reports Perfect Rubrics (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.24964
  bibtex_key: jang2026odysseys
  confidence: high
---

# odysseys-eval

> Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks — Jang et al. (2026) (arXiv:2604.24964, 2026)

## What this evaluates

Probes an agent's ability to perform realistic, long-horizon web navigation tasks that require sustained cross-site reasoning, context maintenance across multiple tabs, and efficient action execution. It evaluates whether models can complete complex, multi-step user journeys derived from real browsing behavior within strict step budgets.

## Datasets

- **Odysseys** — total 200; splits: test (200)

## Metrics

- `Perfect Rubrics (%)` **(primary)** — range: percent
  - Binary pass/fail per task. A task is scored as passing only if every one of its predefined success rubrics are satisfied during the agent's trajectory.
- `Trajectory Efficiency (%)` — range: percent
  - Average per-task ratio of rubric score to step count: (1/N) * sum(s_i / n_i), where s_i is the averaged rubric score for task i and n_i is the number of agent steps (LLM calls) taken.

## Input / output format

**Input**: Task description, starting URL (if provided), and full access to a Google Chrome window within a virtual Ubuntu environment.

**Output**: Action trajectories consisting of browser interactions, system commands, or Python scripts, executed sequentially up to a maximum of 100 steps.

## Scoring recipe

```python
def score_perfect_rubrics(task_rubrics_satisfied):
    return 1.0 if all(task_rubrics_satisfied) else 0.0

def score_trajectory_efficiency(rubric_scores, step_counts):
    return sum(s / n for s, n in zip(rubric_scores, step_counts)) / len(rubric_scores)
```

## Common pitfalls

- Step count refers to LLM round-trips/calls, not individual UI actions; models like GPT-5.4 can batch multiple actions per call, artificially lowering step counts.
- The benchmark enforces a hard 100-step limit per task, which heavily penalizes inefficient exploration or research-heavy strategies that would succeed with more steps.
- Rubric-based scoring provides partial credit for partial completion, unlike holistic trajectory judges that give a binary pass/fail, making direct comparison with older web navigation benchmarks misleading.

## Evidence (verbatim from paper)

> To quantify this, we report Trajectory Efficiency, defined as the per-task ratio of rubric score to step count, averaged over all tasks: where $s_{i}$ is the averaged rubric score for task $i$ and $n_{i}$ is the number of agent steps taken on that task.

## Citation

```bibtex
@misc{jang2026odysseys,
  title={Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2604.24964}
}
```

- arXiv: 2604.24964