Herobench Eval

---
name: herobench-eval
description: Evaluates long-horizon planning and structured reasoning in a grid-based RPG virtual environment. Agents must generate multi-step plans involving resource gathering, crafting, and combat, requiring integration of numerical calculations with action sequencing. Use when the user wants to benchmark on HeroBench, or asks about evaluating this task. Reports Success %.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.12782
  bibtex_key: [REDACTED]
  confidence: high
---

# herobench-eval

> HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds — Anokhin et al. (2025) (arXiv:2508.12782, 2025)

## What this evaluates

Evaluates long-horizon planning and structured reasoning in a grid-based RPG virtual environment. Agents must generate multi-step plans involving resource gathering, crafting, and combat, requiring integration of numerical calculations with action sequencing.

## Datasets

- **HeroBench** — total ?; splits: base (-1), harder (-1); repo https://github.com/stefanrer/HeroBench

## Metrics

- `Success %` **(primary)** — range: percent
  - Percentage of tasks where the model's generated plan and executed actions fully satisfy the task requirements without high-level planning or low-level execution errors.
- `Score` — range: percent
  - Progress score reflecting task completion quality and efficiency, reported as mean ± standard deviation across difficulty levels.
- `pass@k` — range: percent
  - Success rate calculated over k independent attempts per task, measuring the probability of solving a task within k trials.

## Input / output format

**Input**: Grid-based RPG environment state, task instructions specifying objectives (e.g., resource gathering, crafting, combat), and environmental constraints or distractor items.

**Output**: High-level gear selection plan, low-level action sequence, and valid Python code to execute the plan within the environment.

## Scoring recipe

```python
def score_hero_bench(predictions, gold_tasks):
    success_count = 0
    total_score = 0
    for pred, task in zip(predictions, gold_tasks):
        errors = analyze_errors(pred)
        if errors['invalid_code'] == 0 and errors['high_level'] == 0 and errors['execution'] == 0:
            success_count += 1
            total_score += pred.progress_score
    return {
        'Success %': (success_count / len(gold_tasks)) * 100,
        'Score': total_score / len(gold_tasks)
    }
```

## Common pitfalls

- Models often fail at high-level planning (gear selection) rather than low-level execution, so error analysis must separate these two failure modes.
- Invalid Python code generation is counted as a distinct failure type and must be parsed before attempting environment execution.
- Adversarial distractor items in noise-augmented tasks require robust filtering, as models frequently select suboptimal gear due to irrelevant environmental information.

## Evidence (verbatim from paper)

> Table 1: Mean performance of all evaluated models across nine base task difficulty levels in HeroBench. Columns show success rate (%), score (mean ± SD), and tokens (mean ± SD).

## Citation

```bibtex
@misc{anokhin2025herobench,
  title={HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds},
  author={Anokhin et al. (2025)},
  year={2025},
  note={arXiv:2508.12782}
}
```

- arXiv: 2508.12782