Minerva Quant Reasoning Eval

---
name: minerva-quant-reasoning-eval
description: Evaluates large language models on quantitative reasoning and mathematical problem-solving tasks, including arithmetic, multi-step word problems, and standardized test questions, using few-shot prompting and chain-of-thought reasoning. Use when the user wants to benchmark on MATH, MMLU, GSM8k, National Math Exam in Poland, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.14858
  bibtex_key: lewkowycz2022solving
  confidence: medium
---

# minerva-quant-reasoning-eval

> Solving Quantitative Reasoning Problems with Language Models — Lewkowycz et al. (2022) (arXiv:2206.14858, 2022)

## What this evaluates

Evaluates large language models on quantitative reasoning and mathematical problem-solving tasks, including arithmetic, multi-step word problems, and standardized test questions, using few-shot prompting and chain-of-thought reasoning.

## Datasets

- **MATH** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **GSM8k** — total ?; splits: test (-1)
- **National Math Exam in Poland** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly solved problems out of the total number of problems in the dataset.
- `MMLU highest score` — range: percent
  - The score of the answer choice with the highest probability or score among the options, evaluated using a 5-shot prompt per topic.
- `majority voting accuracy` — range: percent
  - Accuracy computed by sampling k=16 model answers using a chain-of-thought prompt and taking the most frequent answer as the final prediction.

## Input / output format

**Input**: Few-shot prompts (e.g., 5-shot per topic for MMLU) with chain-of-thought instructions.

**Output**: Chain-of-thought reasoning steps followed by a final answer or LaTeX-formatted equation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def majority_voting_accuracy(predictions_list, gold):
    final_preds = [Counter(p).most_common(1)[0][0] for p in predictions_list]
    return compute_accuracy(final_preds, gold)
```

## Common pitfalls

- MMLU evaluation requires topic-specific 5-shot prompts rather than a single global prompt.
- Majority voting requires sampling k=16 answers with chain-of-thought prompts instead of using greedy decoding.

## Evidence (verbatim from paper)

> The model achieves over $80\%$ accuracy on 10-digit addition and over $20\%$ accuracy on 18-digit addition.

## Citation

```bibtex
@misc{lewkowycz2022solving,
  title={Solving Quantitative Reasoning Problems with Language Models},
  author={Lewkowycz et al. (2022)},
  year={2022},
  note={arXiv:2206.14858}
}
```

- arXiv: 2206.14858