Tmmluplus Eval

---
name: tmmluplus-eval
description: Evaluates foundation models' multitask language understanding and reasoning capabilities in Traditional Chinese across diverse academic subjects including STEM, social sciences, humanities, and other domains. Use when the user wants to benchmark on TMMLU+, or asks about evaluating this task. Reports average accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.01858
  bibtex_key: tam2024tmmluplus
  confidence: high
---

# tmmluplus-eval

> An Improved Traditional Chinese Evaluation Suite for Foundation Model — Zhi-Rui Tam et al. (2024) (arXiv:2403.01858, 2024)

## What this evaluates

Evaluates foundation models' multitask language understanding and reasoning capabilities in Traditional Chinese across diverse academic subjects including STEM, social sciences, humanities, and other domains.

## Datasets

- **TMMLU+** — total 22690; splits: test (-1), dev (-1)

## Metrics

- `average accuracy (%)` **(primary)** — range: percent
  - Calculated as the proportion of correctly predicted answers out of the total number of questions, multiplied by 100. For human baselines, it is computed as the passing rate based on the proportion of correct responses to total responses per question.

## Input / output format

**Input**: Multiple-choice questions in Traditional Chinese (or Simplified Chinese for cross-lingual experiments), formatted as zero-shot or five-shot prompts with context examples.

**Output**: Single letter/option answer extracted via regular expression parsing from the model's raw text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Chain-of-thought prompting degrades performance on this benchmark compared to direct answer prompting.
- Model performance is highly sensitive to whether the prompt language (Simplified vs. Traditional Chinese) matches the model's primary training language.
- Tokenizer fertility score strongly correlates with accuracy; models with high fertility (longer token sequences for Chinese characters) underperform.

## Evidence (verbatim from paper)

> Table 2: Zero-shot/Five-shot average accuracy (%) in an answer-only setting. The average accuracy over the subjects within each category. Responses are parsed using a modified regular expression by Huang et al. (2023) with Traditional Chinese characters added.

## Citation

```bibtex
@misc{tam2024tmmluplus,
  title={An Improved Traditional Chinese Evaluation Suite for Foundation Model},
  author={Zhi-Rui Tam et al. (2024)},
  year={2024},
  note={arXiv:2403.01858}
}
```

- arXiv: 2403.01858