Kmmlu Eval

---
name: kmmlu-eval
description: This benchmark evaluates large language models' ability to understand and answer expert-level multiple-choice questions in Korean across diverse academic domains. It specifically probes cultural and linguistic alignment, testing whether models can handle native-language nuances and localized knowledge without relying on translated or English-centric training data. Use when the user wants to benchmark on KMMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.11548
  bibtex_key: son2024kmmlu
  confidence: high
---

# kmmlu-eval

> KMMLU: Measuring Massive Multitask Language Understanding in Korean — Son et al. (2024) (arXiv:2402.11548, 2024)

## What this evaluates

This benchmark evaluates large language models' ability to understand and answer expert-level multiple-choice questions in Korean across diverse academic domains. It specifically probes cultural and linguistic alignment, testing whether models can handle native-language nuances and localized knowledge without relying on translated or English-centric training data.

## Datasets

- **KMMLU** — total 35030; splits: test (-1); repo https://github.com/EleutherAI/lm-evaluation-harness

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the number of correctly predicted options divided by the total number of questions. The final reported score is the macro-average of accuracy across all subject categories. The multiple-choice format has 4 options, yielding a 25% random baseline.

## Input / output format

**Input**: Korean multiple-choice questions with 4 options. Each instance is presented in a 5-shot few-shot setting, where 5 exemplar questions and answers are provided as context before the target question.

**Output**: Direct method: The model generates a single option letter or text via greedy decoding. CoT method: The model generates free-form reasoning text followed by the final answer, which is extracted using RegEx.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        # Extract answer: for Direct, use raw output; for CoT, apply RegEx
        pred_answer = extract_answer(pred)
        if pred_answer == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using sampling instead of greedy decoding alters results, as the protocol mandates greedy decoding for both Direct and CoT settings.
- RegEx parsing for CoT outputs is fragile; models that do not strictly follow the expected answer format will yield missing extractions.
- Reporting micro-average instead of macro-average across subjects will skew results toward larger categories, contrary to the paper's methodology.

## Evidence (verbatim from paper)

> Average accuracy(%) calculated using the Direct method in a 5-shot setting across the entire test set. We report the macro-average accuracy across subjects within each category. Random guessing has an accuracy of 25% on all subjects.

## Citation

```bibtex
@misc{son2024kmmlu,
  title={KMMLU: Measuring Massive Multitask Language Understanding in Korean},
  author={Son et al. (2024)},
  year={2024},
  note={arXiv:2402.11548}
}
```

- arXiv: 2402.11548