Game Of 24 Eval

---
name: game-of-24-eval
description: Evaluates an LLM's ability to perform algorithmic search and recursive reasoning within a single generation window, without external tree search or iterative prompting. It probes systematic exploration, pruning, and backtracking capabilities in a mathematical constraint satisfaction task. Use when the user wants to benchmark on Game of 24, or asks about evaluating this task. Reports Success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.10379
  bibtex_key: [REDACTED]
  confidence: high
---

# game-of-24-eval

> Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models — Sel et al. (2023) (arXiv:2308.10379, 2023)

## What this evaluates

Evaluates an LLM's ability to perform algorithmic search and recursive reasoning within a single generation window, without external tree search or iterative prompting. It probes systematic exploration, pruning, and backtracking capabilities in a mathematical constraint satisfaction task.

## Datasets

- **Game of 24** — total 100; splits: test (100)

## Metrics

- `Success rate` **(primary)** — range: percent
  - Percentage of instances where the model generates a valid mathematical expression using exactly the four input numbers and allowed operations (+, -, *, /) that evaluates to 24. Calculated as (correct predictions / total instances) * 100.
- `Avg. Queries` — range: other
  - Average number of LLM generation calls or API requests required per instance to produce a final answer.

## Input / output format

**Input**: Four integers representing the card values. Prompts include a 5-shot in-context setup with DFS-style search trajectories as examples.

**Output**: A single mathematical expression string using the four numbers and +, -, *, / operators that equals 24.

## Scoring recipe

```python
def score_game_of_24(predictions, gold_numbers):
    correct = 0
    for pred in predictions:
        try:
            # Extract numbers from prediction and verify they match gold
            pred_nums = extract_numbers(pred)
            if set(pred_nums) != set(gold_numbers):
                continue
            # Evaluate expression safely
            if evaluate_expression(pred) == 24:
                correct += 1
        except Exception:
            continue
    return (correct / len(gold_numbers)) * 100
```

## Common pitfalls

- Token limits frequently cause 'out-of-token' errors before a solution is found, artificially deflating success rates.
- LLMs may internally discover the correct solution but fail to output it ('non-finalization error'), requiring manual resolution to isolate true algorithmic capability.
- Baselines like ToT rely on external memory and backtracking, making direct success-rate comparisons to single-query generation methods misleading without accounting for query overhead.

## Evidence (verbatim from paper)

> Table 1: Game of 24: success rates and the average number of LLM queries for each example. For an attempt to be considered successful, it must derive a total of 24 using the exact numbers provided and only the allowed operations.

## Citation

```bibtex
@misc{sel2023algorithmofthoughts,
  title={Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models},
  author={Sel et al. (2023)},
  year={2023},
  note={arXiv:2308.10379}
}
```

- arXiv: 2308.10379