Greekmmlu Eval

---
name: greekmmlu-eval
description: This benchmark evaluates large language models' ability to answer multiple-choice questions across 45 diverse academic, professional, and governmental subjects in Greek. It specifically probes native language fluency, cultural grounding, and domain-specific knowledge retention under zero-shot and few-shot prompting conditions. Use when the user wants to benchmark on GreekMMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.05150
  bibtex_key: [REDACTED]
  confidence: high
---

# greekmmlu-eval

> GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek — Zhang et al. (2026) (arXiv:2602.05150, 2026)

## What this evaluates

This benchmark evaluates large language models' ability to answer multiple-choice questions across 45 diverse academic, professional, and governmental subjects in Greek. It specifically probes native language fluency, cultural grounding, and domain-specific knowledge retention under zero-shot and few-shot prompting conditions.

## Datasets

- **GreekMMLU** — total 21805; splits: test (-1); repo https://github.com/mersinkonomi/GreekMMLU

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For open-weight models, the option with the highest token log-likelihood is selected. For closed-source APIs, the model generates free-form text and the predicted Greek label is extracted via regular expressions.

## Input / output format

**Input**: Greek multiple-choice question with subject-specific instruction, question stem, and four labeled options (A, B, Γ, Δ). Prompts are entirely in Greek and may include five representative in-context examples for the five-shot setting.

**Output**: For open-weight models: implicit selection via log-likelihood ranking. For closed-source models: direct extraction of the Greek answer key (A, B, Γ, Δ) from generated text using regular expressions.

## Scoring recipe

```python
correct = 0
for instance in dataset:
    if model_type == 'open_weight':
        pred = argmax([log_likelihood(instance, opt) for opt in instance.options])
    else:
        pred = extract_regex(model.generate(instance.prompt))
    if pred == instance.gold_label:
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- Confusing Greek answer labels (A, B, Γ, Δ) with standard Latin letters (A, B, C, D) during regex extraction.
- Assuming five-shot prompting universally improves performance; it only helps models >2B parameters, while smaller models remain near random baseline.
- Overlooking the distinction between public and private subsets when comparing results or citing performance.

## Evidence (verbatim from paper)

> Gemini 3 Flash reaches an average accuracy of 93.16%, while GPT-5.2 and GPT-4o achieve 87.75% and 86.81%, respectively, consistently excelling in all subjects.

## Citation

```bibtex
@misc{zhang2026greekmmlu,
  title={GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.05150}
}
```

- arXiv: 2602.05150