Cmmlu Eval

---
name: cmmlu-eval
description: Evaluates large language models' Chinese language understanding and multitask knowledge across 67 subjects spanning STEM, humanities, social sciences, and China-specific domains. It probes memorization, reasoning, and instruction-following capabilities in a multiple-choice question-answering format. Use when the user wants to benchmark on CMMLU, or asks about evaluating this task. Reports macro average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09212
  bibtex_key: li2023cmmlu
  confidence: high
---

# cmmlu-eval

> CMMLU: Measuring massive multitask language understanding in Chinese — Li et al. (2023) (arXiv:2306.09212, 2023)

## What this evaluates

Evaluates large language models' Chinese language understanding and multitask knowledge across 67 subjects spanning STEM, humanities, social sciences, and China-specific domains. It probes memorization, reasoning, and instruction-following capabilities in a multiple-choice question-answering format.

## Datasets

- **CMMLU** — total ?; splits: test (-1); repo https://github.com/haonan-li/CMMLU

## Metrics

- `macro average accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted multiple-choice answers. Reported as a macro average over subjects within each category, and overall macro average across all 67 subjects.

## Input / output format

**Input**: A Chinese multiple-choice question with four options (A, B, C, D), optionally preceded by up to 5 demonstration examples. The prompt begins with '以下是关于[主题]的单项选择题，请直接给出正确答案的选项' and ends with '答案是：'.

**Output**: For open-source models: the single token with the highest logit probability among 'A', 'B', 'C', 'D'. For commercial models: free-form text parsed via regex to extract the option letter.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Chain-of-thought prompts often cause models to output answer content instead of the option letter, breaking regex matching and artificially lowering scores.
- Free generation and next-token prediction strategies yield different results; next-token prediction is more efficient but requires careful logit handling, while free generation relies on regex that can fail on complex or wrapped outputs.

## Evidence (verbatim from paper)

> Our goal is to assess the LLMs performance on CMMLU, which contains multiple-choice questions with one correct answer for each question. ... We report macro average accuracy over subjects within each category. "Overall" = macro average score over all subjects.

## Citation

```bibtex
@misc{li2023cmmlu,
  title={CMMLU: Measuring massive multitask language understanding in Chinese},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2306.09212}
}
```

- arXiv: 2306.09212