Hkmmlu Eval

---
name: hkmmlu-eval
description: Evaluates large language models' multilingual comprehension of Hong Kong-specific knowledge, Cantonese linguistic capabilities, and reasoning across STEM, social sciences, and humanities in both Traditional and Simplified Chinese. Use when the user wants to benchmark on HKMMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.02177
  bibtex_key: cao2025hkmmlu
  confidence: high
---

# hkmmlu-eval

> Measuring Hong Kong Massive Multi-Task Language Understanding — Cao et al. (2025) (arXiv:2505.02177, 2025)

## What this evaluates

Evaluates large language models' multilingual comprehension of Hong Kong-specific knowledge, Cantonese linguistic capabilities, and reasoning across STEM, social sciences, and humanities in both Traditional and Simplified Chinese.

## Datasets

- **HKMMLU** — total 117248; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multi-choice questions, extracted using regular expressions from model outputs.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric for machine translation evaluation.
- `METEOR` — range: [0, 1]
  - Metric based on alignment of words between reference and hypothesis, considering synonyms and stemming.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence between reference and hypothesis translations.

## Input / output format

**Input**: Multi-choice questions with options in Traditional or Simplified Chinese; source sentences in Mandarin or Cantonese for translation tasks.

**Output**: For multi-choice: a single option letter/answer extracted via regex. For translation: a generated target language sentence.

## Scoring recipe

```python
def score_mc(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds) * 100

def score_translation(predictions, references):
    bleu = compute_bleu(references, predictions)
    meteor = compute_meteor(references, predictions)
    rouge_l = compute_rouge_l(references, predictions)
    return bleu, meteor, rouge_l
```

## Common pitfalls

- Chain-of-thought prompting significantly drops average accuracy across most models, despite improving STEM reasoning.
- Translation evaluation shows severe asymmetry; models consistently perform much worse translating Mandarin to Cantonese than vice versa.
- Few-shot prompting does not guarantee performance gains and can cause sharp accuracy drops for certain model families.

## Evidence (verbatim from paper)

> We use regular expressions to extract the answers and calculate the percentage of correct answers. For translation tasks, we directly prompt the LLM to translate the sentences and utilize BLEU, METEOR, and ROUGE-L for evaluation.

## Citation

```bibtex
@misc{cao2025hkmmlu,
  title={Measuring Hong Kong Massive Multi-Task Language Understanding},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2505.02177}
}
```

- arXiv: 2505.02177