Seabench Eval

---
name: seabench-eval
description: Evaluates LLMs' ability to handle open-ended, daily interaction scenarios in Southeast Asian languages. It probes contextual adaptation, instruction following, and safety in real-world multilingual usage. Use when the user wants to benchmark on SeaBench, or asks about evaluating this task. Reports LLM-as-a-Judge Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.06298
  bibtex_key: liu2025seaexam
  confidence: high
---

# seabench-eval

> SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia — Liu et al. (2025) (arXiv:2502.06298, 2025)

## What this evaluates

Evaluates LLMs' ability to handle open-ended, daily interaction scenarios in Southeast Asian languages. It probes contextual adaptation, instruction following, and safety in real-world multilingual usage.

## Datasets

- **SeaBench** — total ?; splits: test (-1); repo https://github.com/DAMO-NLP-SG/SeaExam

## Metrics

- `LLM-as-a-Judge Score` **(primary)** — range: [1, 10]
  - Average score from 1 to 10 assigned by GPT-4o acting as a judge. Responses are evaluated against reference answers using category-specific priority aspects.

## Input / output format

**Input**: Zero-shot prompt containing an open-ended question or multi-turn conversation scenario in a Southeast Asian language.

**Output**: Model generates a free-form text response.

## Scoring recipe

```python
responses = model.generate(input_text)
judge = GPT-4o
scores = [judge.score(response, reference, criteria) for response, reference in zip(responses, references)]
avg_score = sum(scores) / len(scores)
```

## Common pitfalls

- Using translated benchmarks (MT-bench-SEA) as a direct proxy for local daily usage, which fails to capture region-specific safety and contextual nuances.
- Comparing open-ended judge scores directly with MCQ accuracy without converting to a common scale (e.g., high-score rate or full mark rate).
- Assuming safety performance aligns with primary language alignment, as multilingual safety often suffers due to training data bias.

## Evidence (verbatim from paper)

> Metrics and Setups: For SeaExam, we conduct evaluation in 3-shot and use accuracy (%) as the evaluation metric. For SeaBench, we employ LLMs-as-a-Judge*Zheng et al. ([2023b]); Bai et al. ([2023]); Ying et al. ([2024])*, setting GPT-4o as the judge model to evaluate LLM’s responses based on the reference answers (construction details in Section[2.2]). Considering that different categories of questions focus on assessing different aspects of model performance, we have designed a list of priority evaluation aspects for each category to facilitate a comprehensive judgment. We prompt GPT-4o to rate each response on a scale from 1 to 10.

## Citation

```bibtex
@misc{liu2025seaexam,
  title={SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.06298}
}
```

- arXiv: 2502.06298