Pencil Puzzle Bench Eval

---
name: pencil-puzzle-bench-eval
description: Evaluates multi-step verifiable reasoning and agentic iteration on constraint-satisfaction puzzles. It probes a model's ability to plan, execute moves, check constraints step-by-step, and course-correct over long contexts. Use when the user wants to benchmark on Pencil Puzzle Bench, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.02119
  bibtex_key: waugh2026pencil
  confidence: high
---

# pencil-puzzle-bench-eval

> Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning — Waugh (2026) (arXiv:2603.02119, 2026)

## What this evaluates

Evaluates multi-step verifiable reasoning and agentic iteration on constraint-satisfaction puzzles. It probes a model's ability to plan, execute moves, check constraints step-by-step, and course-correct over long contexts.

## Datasets

- **Pencil Puzzle Bench** — total 62231; splits: test (-1); HF `approximatelabs/pencil-puzzle-bench`

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of puzzles correctly solved by the model. Calculated as (number of solved puzzles / total puzzles evaluated) * 100.
- `uplift` — range: percent
  - Absolute percentage point gain from agentic iteration over direct-ask performance. Calculated as agentic success rate minus direct success rate.
- `cost per success` — range: other
  - Total recorded benchmark cost in USD divided by the number of successfully solved puzzles.

## Input / output format

**Input**: Constraint-satisfaction pencil puzzles presented as initial board states with associated rules. Models receive the puzzle configuration and must generate moves or solutions.

**Output**: Iterative board state updates (moves) or a final solved board configuration. In agentic mode, models output moves, receive constraint-violation feedback, and continue until solved or max turns/time reached.

## Scoring recipe

```python
def compute_metrics(predictions, gold, outcomes):
    solved = sum(1 for o in outcomes if o == 'correct')
    total = len(predictions)
    success_rate = (solved / total) * 100
    return success_rate
```

## Common pitfalls

- Infrastructure errors (timeouts, API failures) are not always filtered out, artificially lowering success rates and skewing cost analysis.
- Higher reasoning effort does not monotonically improve performance; some models perform worse at maximum effort due to reliability/capability tradeoffs.
- Cost per success is underreported when agentic runs terminate early, as token usage from completed turns before the error is not recorded.

## Evidence (verbatim from paper)

> Table 3: Top 15 models by best success rate. Direct ask on 300 puzzles; agentic on 30-puzzle baseline for most models, 60-puzzle expanded set for top 3.

## Citation

```bibtex
@misc{waugh2026pencil,
  title={Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning},
  author={Waugh (2026)},
  year={2026},
  note={arXiv:2603.02119}
}
```

- arXiv: 2603.02119