Sea Helm Eval

---
name: sea-helm-eval
description: Evaluates Thai language models across eight competencies including instruction following, multi-turn dialogue stability, natural language understanding, generation, reasoning, safety, and code-switching resistance. Use when the user wants to benchmark on SEA-HELM, or asks about evaluating this task. Reports SEA-HELM Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.19455
  bibtex_key: pairatsuppawat2025siamgpt
  confidence: high
---

# sea-helm-eval

> SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation — Pairatsuppawat et al. (2025) (arXiv:2512.19455, 2025)

## What this evaluates

Evaluates Thai language models across eight competencies including instruction following, multi-turn dialogue stability, natural language understanding, generation, reasoning, safety, and code-switching resistance.

## Datasets

- **SEA-HELM** — total ?; splits: test (-1)

## Metrics

- `SEA-HELM Average Score` **(primary)** — range: percent
  - Arithmetic mean of normalized scores (0–100 scale) across eight sub-benchmarks: SEA-IFEval, SEA-MTBench, NLG, NLU, NLR, Safety, ThaiExam, and Code Switching. Sub-benchmarks follow their respective standard evaluation protocols.

## Input / output format

**Input**: Instruction prompts, multi-turn dialogue histories, QA pairs, translation/summarization inputs, NLI/causal reasoning statements, and safety evaluation queries in Thai and English.

**Output**: Model-generated text responses corresponding to each input prompt or task.

## Scoring recipe

```python
scores = {}
for benchmark in ["SEA-IFEval", "SEA-MTBench", "NLG", "NLU", "NLR", "Safety", "ThaiExam", "Code Switching"]:
    scores[benchmark] = compute_benchmark_score(predictions[benchmark], gold[benchmark])
final_score = sum(scores.values()) / len(scores)
return final_score
```

## Common pitfalls

- The model's lower NLG score is an intentional design trade-off prioritizing stability and formatting control over open-ended generative fluency.
- Multi-turn dialogue and safety scores are highly sensitive to context consistency and constraint adherence, which may vary significantly across different base models.

## Evidence (verbatim from paper)

> We report the performance of SiamGPT-32B on the SEA-HELM benchmark suite following the evaluation protocol described in Section[3.4], with the main results summarized in Figure[1]. ... Performance on SEA-MTBench increases from 57.94 to 75.81, indicating that SystemChat-2.0 supervision substantially improves contextual consistency across turns. Instruction following also improves markedly, with SEA-IFEval increasing from 75.47 to 83.00. Overall, the average score across benchmarks increases from 68.46 to 76.03, confirming that Quality-First fine-tuning yields consistent improvements across stability, instruction following, dialogue robustness, and language understanding.

## Citation

```bibtex
@misc{pairatsuppawat2025siamgpt,
  title={SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation},
  author={Pairatsuppawat et al. (2025)},
  year={2025},
  note={arXiv:2512.19455}
}
```

- arXiv: 2512.19455