Capbencher Eval

---
name: capbencher-eval
description: Evaluates LLMs on standard benchmarks modified with randomized answers to measure performance tracking and detect data contamination via a Bayes accuracy ceiling. The protocol compares model accuracy against a predefined theoretical maximum (Bayes accuracy) to identify overfitting or memorization. It also assesses robustness to reverse-engineering attacks and cross-lingual contamination. Use when the user wants to benchmark on GSM8K, ARC-Challenge, GPQA, MathQA, MMLU, HLE-MC, MMLU-ProX, BoolQ, GPQA (diamond), MMLU-Pro, MATH-500, HumanEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18102
  bibtex_key: [REDACTED]
  confidence: high
---

# capbencher-eval

> How Can I Publish My LLM Benchmark Without Giving the True Answers Away? — Ishida et al. (2025) (arXiv:2505.18102, 2025)

## What this evaluates

Evaluates LLMs on standard benchmarks modified with randomized answers to measure performance tracking and detect data contamination via a Bayes accuracy ceiling. The protocol compares model accuracy against a predefined theoretical maximum (Bayes accuracy) to identify overfitting or memorization. It also assesses robustness to reverse-engineering attacks and cross-lingual contamination.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **ARC-Challenge** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **MathQA** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **HLE-MC** — total ?; splits: test (-1)
- **MMLU-ProX** — total ?; splits: test (-1)
- **BoolQ** — total ?; splits: test (-1)
- **GPQA (diamond)** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **MATH-500** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions compared to the ground truth. Evaluated on both capped (randomized) and non-capped benchmarks to track model progress.
- `Kendall's tau` — range: [-1, 1]
  - Rank correlation coefficient measuring how well the capped benchmark tracks model improvements relative to the non-capped benchmark.
- `binomial test p-value` — range: [0, 1]
  - Statistical test to determine if observed accuracy significantly exceeds the predefined Bayes accuracy ceiling, indicating data contamination.

## Input / output format

**Input**: Question text, optionally with multiple-choice options. For capped benchmarks, includes a randomized answer and an instruction to randomly select a number from a list and add it to the solution.

**Output**: Direct answer string, selected choice label, or generated code snippet.

## Scoring recipe

```python
from scipy.stats import binom
def evaluate(predictions, golds, bayes_acc):
    n = len(golds)
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    acc = correct / n
    p_val = binom.sf(correct - 1, n, bayes_acc)
    return {'accuracy': acc, 'p_value': p_val, 'contaminated': p_val < 0.05}
```

## Common pitfalls

- Bayes accuracy ceiling is task-specific and must be pre-defined by the benchmark creator (e.g., 50%, 25%, 10%).
- Contamination detection requires a statistical binomial test against the Bayes ceiling, not just checking if raw accuracy exceeds a fixed threshold.
- Reverse engineering prompts (P1/P2) can allow models to recover true answers, artificially inflating accuracy and masking contamination signals.

## Evidence (verbatim from paper)

> As expected, the accuracy of capped benchmarks increased as the accuracy of non-capped benchmarks increased (Kendall’s $	au$ is $0.96$), as shown in Fig.[3]. This demonstrates that CapBencher remain effective for tracking LLM progress. As shown in Fig.[4], our method detected contamination across all models and benchmarks, as indicated by accuracies surpassing the Bayes accuracy of 50%.

## Citation

```bibtex
@misc{ishida2025capbencher,
  title={How Can I Publish My LLM Benchmark Without Giving the True Answers Away?},
  author={Ishida et al. (2025)},
  year={2025},
  note={arXiv:2505.18102}
}
```

- arXiv: 2505.18102