Latent Reasoning Benchmarks Eval

---
name: latent-reasoning-benchmarks-eval
description: Evaluates a language model's reasoning, coding, and general knowledge capabilities using a suite of standard academic benchmarks. It specifically probes how test-time compute scaling (via recurrent depth) impacts performance across mathematical, coding, and commonsense reasoning tasks. Use when the user wants to benchmark on GSM8K, MATH (Minerva), MathQA, MBPP, HumanEval, ARC-E, ARC-C, HellaSwag, MMLU, OBQA, PiQA, SciQ, WinoGrande, or asks about evaluating this task. Reports flexible extract accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.05171
  bibtex_key: geiping2025scaling
  confidence: high
---

# latent-reasoning-benchmarks-eval

> Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al. (2025) (arXiv:2502.05171, 2025)

## What this evaluates

Evaluates a language model's reasoning, coding, and general knowledge capabilities using a suite of standard academic benchmarks. It specifically probes how test-time compute scaling (via recurrent depth) impacts performance across mathematical, coding, and commonsense reasoning tasks.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **MATH (Minerva)** — total ?; splits: test (-1)
- **MathQA** — total ?; splits: test (-1)
- **MBPP** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)
- **ARC-E** — total ?; splits: test (-1)
- **ARC-C** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **OBQA** — total ?; splits: test (-1)
- **PiQA** — total ?; splits: test (-1)
- **SciQ** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)

## Metrics

- `flexible extract accuracy` **(primary)** — range: [0, 1]
  - Percentage of test cases where the model's extracted answer matches the ground truth, allowing for formatting variations, synonyms, or equivalent numerical representations.
- `strict extract accuracy` — range: [0, 1]
  - Percentage of test cases where the model's extracted answer exactly matches the ground truth string without formatting tolerance.
- `extract match` — range: [0, 1]
  - Minerva evaluation rule for MATH: exact string match between the model's final answer and the gold solution after extraction.
- `normalized accuracy` — range: [0, 1]
  - Percentage of correct answers after normalizing formatting and units, as specified for MathQA.
- `pass@1` — range: [0, 1]
  - Proportion of generated code solutions that pass all provided unit tests on the first attempt.

## Input / output format

**Input**: Text prompts provided via lm-eval harness or bigcode-bench. Configurations include zero-shot, 8-way few-shot CoT (multiturn), and open-book QA (providing a relevant fact). Chat templates are optionally applied.

**Output**: Model-generated text completions. For math benchmarks, extracted numerical answers or reasoning traces. For code benchmarks, executable Python code snippets.

## Scoring recipe

```python
def score(predictions, golds, metric):
    if metric == 'pass@1':
        return sum(1 for pred, gold in zip(predictions, golds) if execute_and_pass(pred, gold)) / len(predictions)
    elif metric in ['strict extract accuracy', 'extract match']:
        return sum(1 for pred, gold in zip(predictions, golds) if extract_answer(pred) == gold) / len(predictions)
    elif metric in ['flexible extract accuracy', 'normalized accuracy']:
        return sum(1 for pred, gold in zip(predictions, golds) if normalize_and_match(extract_answer(pred), gold)) / len(predictions)
    else:
        return sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(predictions)
```

## Common pitfalls

- Test-time compute scaling (recurrence depth r) affects performance non-linearly; saturation points vary significantly by task difficulty.
- Training setup caveats: no learning rate cooldown, trained only on public data, and smaller token count compared to industrial baselines, which limits direct comparability.
- Evaluation harness differences: lm-eval default 'closed-book' format vs. open-book setups (e.g., providing facts for OBQA) drastically change results.

## Evidence (verbatim from paper)

> We execute all standard benchmarks through the lm-eval harness (Biderman et al., 2024) and code benchmarks via bigcode-bench (Zhuo et al., 2024). We report flexible and strict extract for GSM8K and GSM8K CoT, extract match for Minerva Math, and acc norm. for MathQA. We report pass@1 for both datasets.

## Citation

```bibtex
@misc{geiping2025scaling,
  title={Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach},
  author={Jonas Geiping et al. (2025)},
  year={2025},
  note={arXiv:2502.05171}
}
```

- arXiv: 2502.05171