Laobench Eval

---
name: laobench-eval
description: Evaluates large language models on their proficiency in Lao, a low-resource Southeast Asian language. It probes factual knowledge, K12 curriculum alignment, culturally grounded reasoning, bilingual translation fidelity, and open-ended generation quality through multiple-choice, translation, and pairwise arena tasks. Use when the user wants to benchmark on LaoBench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.11334
  bibtex_key: gao2025laobench
  confidence: high
---

# laobench-eval

> LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models — Gao et al. (2025) (arXiv:2511.11334, 2025)

## What this evaluates

Evaluates large language models on their proficiency in Lao, a low-resource Southeast Asian language. It probes factual knowledge, K12 curriculum alignment, culturally grounded reasoning, bilingual translation fidelity, and open-ended generation quality through multiple-choice, translation, and pairwise arena tasks.

## Datasets

- **LaoBench** — total ?; splits: Lao-7k (7000), Lao-500 (500)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions. Model outputs are normalized to a single option label (A/B/C/D); unparseable or multiple labels count as incorrect.
- `BLEU` — range: [0, 1]
  - Corpus-level BLEU score computed with SacreBLEU using Lao-aware tokenization to handle the scriptio continua style without explicit word boundaries.
- `Win-rate` — range: percent
  - Pairwise win rate against a fixed baseline (GPT-5-High). Ties are treated as half-wins. Scores are averaged across prompts and aggregated over two judge models, reported as a percentage.

## Input / output format

**Input**: For multiple-choice: question stem and four options in Lao. For translation: source text in Lao. For open-ended: instruction prompt in Lao.

**Output**: For multiple-choice: exactly one option label (A, B, C, or D). For translation/open-ended: generated text in Lao.

## Scoring recipe

```python
import re, sacrebleu

# 1. Accuracy (Lao-7k MC)
def normalize_mc(output):
    match = re.search(r'[A-D]', output)
    return match.group(0) if match else None
acc = sum(1 for p, g in zip(preds, golds) if normalize_mc(p) == g) / len(golds)

# 2. BLEU (Lao-7k Translation)
bleu = sacrebleu.corpus_bleu(preds, [golds], tokenize='lao')

# 3. Win-rate (Lao-500 Arena)
wins = []
for prompt in prompts:
    s1 = judge1.compare(prompt, baseline) # 1=win, 0.5=tie, 0=loss
    s2 = judge2.compare(prompt, baseline)
    wins.append((s1 + s2) / 2)
win_rate = (sum(wins) / len(wins)) * 100
```

## Common pitfalls

- Lao is written in a scriptio continua style without explicit word boundaries, so standard BLEU tokenization will severely penalize models unless Lao-aware tokenization (e.g., LaoNLP) is applied.
- Chain-of-thought (CoT) prompting improves performance on complex reasoning subdomains but yields minimal gains on factual or formulaic K12 questions, so evaluating only CoT or only direct-answer may skew results.
- Arena-style win-rates are sensitive to judge model family bias (e.g., Qwen3-Max favors Qwen-family models) and position bias; results must be averaged across judges and randomized positions to be reliable.

## Evidence (verbatim from paper)

> For K12 Education and Knowledge Application multiple-choice questions, we report Accuracy. For Translation tasks, we compute corpus-level BLEU against expert-written references under a standardized SacreBLEU configuration. Since Lao is written in a scriptio continua style without explicit word boundaries, BLEU can be sensitive to segmentation; we therefore apply Lao-aware tokenization using LaoNLP before scoring, and additionally report chrF++ in Appendix[C].

## Citation

```bibtex
@misc{gao2025laobench,
  title={LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2511.11334}
}
```

- arXiv: 2511.11334