Llama Berry Eval

---
name: llama-berry-eval
description: Evaluates LLMs on complex mathematical reasoning using search-based inference (SR-MCTS) rather than direct generation. It measures success rates across varying difficulty levels, from grade-school math to Olympiad-level problems, by testing both majority-vote and best-of-k strategies. Use when the user wants to benchmark on AIME24, AMC23, Math Odyssey, GPQA Diamond, OlympiadBench, College Math, MMLU STEM, GSM8K, GSMHard, MATH500, or asks about evaluating this task. Reports major@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.02884
  bibtex_key: zhang2024llamaberry
  confidence: high
---

# llama-berry-eval

> LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning — Di Zhang et al. (2024) (arXiv:2410.02884, 2024)

## What this evaluates

Evaluates LLMs on complex mathematical reasoning using search-based inference (SR-MCTS) rather than direct generation. It measures success rates across varying difficulty levels, from grade-school math to Olympiad-level problems, by testing both majority-vote and best-of-k strategies.

## Datasets

- **AIME24** — total 30; splits: test (30)
- **AMC23** — total ?; splits: test (-1)
- **Math Odyssey** — total ?; splits: test (-1)
- **GPQA Diamond** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)
- **College Math** — total ?; splits: test (-1)
- **MMLU STEM** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **GSMHard** — total ?; splits: test (-1)
- **MATH500** — total ?; splits: test (-1)

## Metrics

- `major@k` **(primary)** — range: percent
  - Majority voting accuracy over k inference rollouts. A problem is counted as solved if more than half of the k generated answers are graded correct.
- `rm@k` — range: percent
  - Resolve rate (best-of-k) over k inference rollouts. A problem is counted as solved if at least one of the k generated answers is graded correct.

## Input / output format

**Input**: Textual mathematical problem statement.

**Output**: Answer string conforming to the format specified in the prompt. Graded as consistent if it exactly matches the ground truth, closely approximates it numerically, or is equivalent in symbolic form.

## Scoring recipe

```python
def evaluate(predictions, gold, k):
    correct = []
    for ans in predictions:
        if exact_match(ans, gold) or close_numerical(ans, gold) or symbolic_equivalent(ans, gold):
            correct.append(1)
        else:
            correct.append(0)
    major_k = 1 if sum(correct) > k / 2 else 0
    rm_k = 1 if sum(correct) > 0 else 0
    return major_k, rm_k
```

## Common pitfalls

- Confusing major@k (majority vote) with rm@k (best-of-k/resolve rate), which yield different results especially on hard problems.
- Grading requires checking for numerical tolerance and symbolic equivalence, not just exact string matches.
- Performance is highly dependent on the number of inference-time rollouts (k), so results are not directly comparable across different k values.

## Evidence (verbatim from paper)

> We score answers as consistent if they exactly match the ground truth, closely approximate it numerically, or are equivalent in symbolic form. To ensure a comprehensive and rigorous evaluation, we adopt major@k (Kuncheva, [2014]) and rm@k (Yang et al., [2024c]), which can be unified as the solved rate of problems (Lightman et al., [2023]; Luo et al., [2024a]).

## Citation

```bibtex
@misc{zhang2024llamaberry,
  title={LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning},
  author={Di Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.02884}
}
```

- arXiv: 2410.02884