Seaexam Eval

---
name: seaexam-eval
description: Evaluates LLMs' ability to answer local, culturally grounded multiple-choice questions in Southeast Asian languages (Indonesian, Thai, Vietnamese). It probes regional knowledge, language comprehension, and alignment with actual local usage compared to translated benchmarks. Use when the user wants to benchmark on SeaExam, or asks about evaluating this task. Reports accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.06298
  bibtex_key: liu2025seaexam
  confidence: high
---

# seaexam-eval

> SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia — Liu et al. (2025) (arXiv:2502.06298, 2025)

## What this evaluates

Evaluates LLMs' ability to answer local, culturally grounded multiple-choice questions in Southeast Asian languages (Indonesian, Thai, Vietnamese). It probes regional knowledge, language comprehension, and alignment with actual local usage compared to translated benchmarks.

## Datasets

- **SeaExam** — total ?; splits: test (-1); repo https://github.com/DAMO-NLP-SG/SeaExam

## Metrics

- `accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) × 100.

## Input / output format

**Input**: 3-shot prompt containing three example question-answer pairs followed by a multiple-choice question in a Southeast Asian language.

**Output**: Model selects the correct option letter/answer from the provided choices.

## Scoring recipe

```python
predictions = model.generate(input_text)
gold = question['answer']
accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) * 100
```

## Common pitfalls

- Directly comparing SeaExam accuracy with SeaBench judge scores without normalization, as they use fundamentally different scoring mechanisms.
- Assuming translated benchmarks like MMLU-SEA capture the same cultural and linguistic nuances as locally constructed exams.
- Overlooking that MCQ formats may provide linguistic cues that artificially inflate performance compared to open-ended tasks.

## Evidence (verbatim from paper)

> Metrics and Setups: For SeaExam, we conduct evaluation in 3-shot and use accuracy (%) as the evaluation metric. For SeaBench, we employ LLMs-as-a-Judge*Zheng et al. ([2023b]); Bai et al. ([2023]); Ying et al. ([2024])*, setting GPT-4o as the judge model to evaluate LLM’s responses based on the reference answers (construction details in Section[2.2]). Considering that different categories of questions focus on assessing different aspects of model performance, we have designed a list of priority evaluation aspects for each category to facilitate a comprehensive judgment. We prompt GPT-4o to rate each response on a scale from 1 to 10.

## Citation

```bibtex
@misc{liu2025seaexam,
  title={SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.06298}
}
```

- arXiv: 2502.06298