Mmlu Eval

---
name: mmlu-eval
description: Evaluates broad language understanding and reasoning capabilities across multiple academic and professional domains using multiple-choice questions. It tests the model's ability to process and answer questions in a few-shot setting. Use when the user wants to benchmark on MMLU, or asks about evaluating this task. Reports macro_avg/acc_char.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04344
  bibtex_key: zhang2025gem
  confidence: high
---

# mmlu-eval

> GEM: Empowering LLM for both Embedding Generation and Language Understanding — Caojin Zhang et al. (2025) (arXiv:2506.04344, 2025)

## What this evaluates

Evaluates broad language understanding and reasoning capabilities across multiple academic and professional domains using multiple-choice questions. It tests the model's ability to process and answer questions in a few-shot setting.

## Datasets

- **MMLU** — total ?; splits: test (-1); HF `cais/mmlu`

## Metrics

- `macro_avg/acc_char` **(primary)** — range: percent
  - Macro-averaged character-level accuracy across all subjects. Answers are evaluated using character-level similarity rather than exact string matching to handle formatting variations.

## Input / output format

**Input**: Multiple-choice questions with a 5-shot context window.

**Output**: Selected option letter or generated text answer.

## Scoring recipe

```python
# Follow Llama 3.1 model card protocol
predictions = []
for question in mmlu_test:
    prompt = build_5shot_prompt(question)
    answer = model.generate(prompt)
    predictions.append(extract_option(answer))
acc_char = character_accuracy(predictions, gold_labels)
macro_acc = mean(acc_char_per_subject)
return macro_acc
```

## Common pitfalls

- The paper reports 'acc_char' (character accuracy) instead of exact-match accuracy, which can yield different scores for synonymous or formatted answers.
- The paper incorrectly states MMLU covers '57 languages'; it actually covers 57 academic/professional subjects, which may confuse evaluators expecting language-specific splits.

## Evidence (verbatim from paper)

> For MMLU, we follow Llama 3.1 model card and use five-shot and report macro_avg/acc_char.

## Citation

```bibtex
@misc{zhang2025gem,
  title={GEM: Empowering LLM for both Embedding Generation and Language Understanding},
  author={Caojin Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.04344}
}
```

- arXiv: 2506.04344