Majority K

---
name: majority-k
description: Evaluates the robustness and consistency of a model's mathematical reasoning by aggregating predictions across multiple inference runs per problem. It measures whether the most frequent prediction among k samples matches the ground truth. Use when the user has predictions and gold and needs to compute Majority@k.
metadata:
  skill_kind: metric
  source_arxiv: 2506.15710
  bibtex_key: ouyang2025rast
  confidence: high
---

# majority-k

> RAST: Reasoning Activation in LLMs via Small-model Transfer — Ouyang et al. (2025) (arXiv:2506.15710, 2025)

## What this evaluates

Evaluates the robustness and consistency of a model's mathematical reasoning by aggregating predictions across multiple inference runs per problem. It measures whether the most frequent prediction among k samples matches the ground truth.

## Datasets

- **MATH500** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **AIME 2024** — total ?; splits: test (-1)
- **AMC 2023** — total ?; splits: test (-1)
- **Minerva** — total ?; splits: test (-1)
- **Olympiad Bench** — total ?; splits: test (-1)

## Metrics

- `Majority@k` **(primary)** — range: [0, 1]
  - Accuracy based on the majority prediction among k inference runs per problem. Formula: (1/N) * sum_{i=1}^N I(majority(y_hat_{i,1}...y_hat_{i,k}) == y_i).

## Input / output format

**Input**: Mathematical problem statement.

**Output**: A predicted answer string per inference run.

## Scoring recipe

```python
def compute_majority_k(predictions_list, k, ground_truths):
    correct = 0
    for preds, gt in zip(predictions_list, ground_truths):
        majority_pred = max(set(preds), key=preds.count)
        if majority_pred == gt:
            correct += 1
    return correct / len(ground_truths)
```

## Common pitfalls

- Requires generating k independent samples per problem, which significantly increases inference cost.
- Tie-breaking in the majority function is not specified and can arbitrarily affect results when k is small.
- Accuracy is typically exact-match on the final answer, not step-level or format-tolerant.

## Evidence (verbatim from paper)

> Definition of Majority@k: This metric evaluates accuracy based on the majority prediction among multiple inference runs per problem. Formally, Majority@k = 1/N \sum_{i=1}^{N}\mathbb{I}(majority(\hat{y}_{i,1},\dots,\hat{y}_{i,k}) = y_i), where the majority function returns the prediction most frequently appearing among the k inference runs for the i-th problem.

## Citation

```bibtex
@misc{ouyang2025rast,
  title={RAST: Reasoning Activation in LLMs via Small-model Transfer},
  author={Ouyang et al. (2025)},
  year={2025},
  note={arXiv:2506.15710}
}
```

- arXiv: 2506.15710