Chess Puzzle Eval

---
name: chess-puzzle-eval
description: Evaluates a model's ability to play chess and solve tactical puzzles without explicit search, testing its capacity for long-horizon planning and generalization to novel board states. Use when the user wants to benchmark on Lichess puzzles, or asks about evaluating this task. Reports Lichess Elo.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.04494
  bibtex_key: [REDACTED]
  confidence: high
---

# chess-puzzle-eval

> Amortized Planning with Large-Scale Transformers: A Case Study on Chess — Ruoss et al. (2024) (arXiv:2402.04494, 2024)

## What this evaluates

Evaluates a model's ability to play chess and solve tactical puzzles without explicit search, testing its capacity for long-horizon planning and generalization to novel board states.

## Datasets

- **Lichess puzzles** — total 11000; splits: test (10000), val (1000); repo https://github.com/google-deepmind/searchless_chess

## Metrics

- `Lichess Elo` **(primary)** — range: other
  - Standard chess rating on the Lichess platform calculated from game outcomes against human and bot opponents.
- `Puzzle accuracy` — range: [0, 1]
  - Percentage of puzzles solved correctly, requiring the full correct move sequence to be predicted greedily.
- `Action accuracy` — range: [0, 1]
  - Percentage of top-1 predicted actions matching the gold action for a given board state.
- `Kendall's τ` — range: [-1, 1]
  - Rank correlation coefficient measuring how well predicted action-values match the gold action-values.

## Input / output format

**Input**: Chess board state (position) with the set of legal moves.

**Output**: Predicted action-values for all legal moves, or a single greedy move selection derived from those values.

## Scoring recipe

```python
def score_puzzle_accuracy(pred_sequences, gold_sequences):
    correct = sum(1 for p, g in zip(pred_sequences, gold_sequences) if p == g)
    return correct / len(gold_sequences)

def score_action_accuracy(pred_actions, gold_actions):
    return sum(1 for p, g in zip(pred_actions, gold_actions) if p == g) / len(gold_actions)

def score_kendall_tau(pred_values, gold_values):
    return kendalltau(pred_values, gold_values).correlation
```

## Common pitfalls

- Models are evaluated without explicit search at test time; puzzle solving relies purely on greedy value estimation, not lookahead.
- Lichess Elo against bots differs from human Elo due to different player pools and bot exploitation strategies.
- Large models may overfit to training board states if the dataset size is insufficient, despite low training loss.

## Evidence (verbatim from paper)

> Table 1 shows the playing strength (internal tournament Elo, external Lichess Elo, and puzzle accuracy) of our large-scale transformers trained on the full (10M games) training set.

## Citation

```bibtex
@misc{ruoss2024amortized,
  title={Amortized Planning with Large-Scale Transformers: A Case Study on Chess},
  author={Ruoss et al. (2024)},
  year={2024},
  note={arXiv:2402.04494}
}
```

- arXiv: 2402.04494