Haerae Bench Eval

---
name: haerae-bench-eval
description: Evaluates language models' proficiency in Korean cultural knowledge and context. It probes capabilities across vocabulary (loan words, standard nomenclature, rare words), history, general knowledge, and reading comprehension, specifically highlighting the limitations of English-trained or non-Korean-tailored models. Use when the user wants to benchmark on HAE-RAE Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.02706
  bibtex_key: son2023haeraebench
  confidence: high
---

# haerae-bench-eval

> HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models — Son et al. (2023) (arXiv:2309.02706, 2023)

## What this evaluates

Evaluates language models' proficiency in Korean cultural knowledge and context. It probes capabilities across vocabulary (loan words, standard nomenclature, rare words), history, general knowledge, and reading comprehension, specifically highlighting the limitations of English-trained or non-Korean-tailored models.

## Datasets

- **HAE-RAE Bench** — total 1500; splits: test (1500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. For open-weight models, computed via log-likelihood over five options. For proprietary models, computed via exact match of the generated option number.

## Input / output format

**Input**: Multiple-choice questions with five options, presented in 0-shot, 5-shot, or 10-shot formats with exemplars.

**Output**: For open models: log probabilities for each of the five options. For proprietary models: the number of the selected option.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Directly comparing log-likelihood accuracy (open models) with generation-based accuracy (proprietary models) is not feasible due to different evaluation methods.
- Assuming in-context learning significantly improves performance; the benchmark shows ICL is insufficient for cultural alignment.
- Assuming model size strongly predicts performance; ANOVA shows only ~26% of variance is explained by parameter count.

## Evidence (verbatim from paper)

> Unlike openly available models for which we leveraged a log probability method to gauge accuracy, these models do not provide log probabilities for individual tokens. Accordingly, we prompted the models to generate the number of the options they deemed correct. Direct comparison between these evaluation methods is not feasible. However, the method used for proprietary models is more challenging than the log-likelihood method applied to open models. The former entails generating answers from the entire vocabulary, whereas the latter restricts choices to five options.

## Citation

```bibtex
@misc{son2023haeraebench,
  title={HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models},
  author={Son et al. (2023)},
  year={2023},
  note={arXiv:2309.02706}
}
```

- arXiv: 2309.02706