Standard Llm Benchmarks Eval

---
name: standard-llm-benchmarks-eval
description: Evaluates language model capabilities across knowledge, reasoning, instruction following, and safety using a standard suite of downstream benchmarks. It measures both pretraining quality and post-adaptation performance on established NLP and coding tasks. Use when the user wants to benchmark on MMLU, HellaSwag, ARC-Challenge, ARC-Easy, PIQA, WinoGrande, GSM8k, BBH, HumanEval, AlpacaEval 1.0, XSTest, IFEval, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.02060
  bibtex_key: muennighoff2024olmoe
  confidence: high
---

# standard-llm-benchmarks-eval

> OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al. (2024) (arXiv:2409.02060, 2024)

## What this evaluates

Evaluates language model capabilities across knowledge, reasoning, instruction following, and safety using a standard suite of downstream benchmarks. It measures both pretraining quality and post-adaptation performance on established NLP and coding tasks.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **ARC-Challenge** — total ?; splits: test (-1)
- **ARC-Easy** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **GSM8k** — total ?; splits: test (-1)
- **BBH** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)
- **AlpacaEval 1.0** — total ?; splits: test (-1)
- **XSTest** — total ?; splits: test (-1)
- **IFEval** — total ?; splits: test (-1)

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - EM. Computed as the fraction of predictions that exactly match the gold answer after standard normalization.
- `Pass@10` — range: [0, 1]
  - Fraction of generated code samples (out of 10 attempts) that pass all provided unit tests.
- `win rate` — range: percent
  - %win. Percentage of pairwise comparisons where the model's output is preferred over the reference by an LLM judge.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall computed over token-level matches between prediction and gold.
- `loose accuracy` — range: [0, 1]
  - Loose Acc. Fraction of predictions that match the gold answer according to a relaxed regex-based matching rule.

## Input / output format

**Input**: Text prompts containing task instructions, optionally with few-shot examples (0, 3, 5, or 8-shot depending on the benchmark), or direct questions.

**Output**: Model-generated text completion or selected option string.

## Scoring recipe

```python
def score(predictions, golds, metric):
    if metric == 'EM':
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif metric == 'Pass@10':
        return sum(1 for p in predictions[:10] if is_correct(p)) / 10
    elif metric == '%win':
        return judge_win_rate(predictions, reference_outputs)
    elif metric == 'F1':
        return compute_f1(predictions, golds)
    elif metric == 'Loose Acc':
        return sum(1 for p, g in zip(predictions, golds) if regex_match(p, g)) / len(golds)
```

## Common pitfalls

- Varying few-shot settings across tasks (0-shot, 3-shot, 5-shot, 8-shot CoT) require strict prompt formatting to reproduce.
- AlpacaEval uses a reference-free pairwise comparison with an LLM judge, not fixed gold labels, making direct score replication sensitive to the judge model version.
- Loose accuracy for IFEval relies on specific regex patterns that may differ from strict exact-match implementations.

## Evidence (verbatim from paper)

> We run all evaluations ourselves with 5 few-shots, see [Appendix C] for details. Metric ($
ightarrow$) | EM | EM | EM | Pass@10 | %win | F1 | Loose Acc

## Citation

```bibtex
@misc{muennighoff2024olmoe,
  title={OLMoE: Open Mixture-of-Experts Language Models},
  author={Niklas Muennighoff et al. (2024)},
  year={2024},
  note={arXiv:2409.02060}
}
```

- arXiv: 2409.02060