Tlue Eval

---
name: tlue-eval
description: Evaluates large language models' proficiency in Tibetan across general knowledge comprehension and safety-critical domains. It probes the models' ability to handle low-resource language tasks, complex reasoning, and culturally sensitive alignment compared to English baselines. Use when the user wants to benchmark on Ti-MMLU, Ti-SafetyBench, or asks about evaluating this task. Reports Accuracy (ACC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12051
  bibtex_key: gao2025tlue
  confidence: high
---

# tlue-eval

> TLUE: A Tibetan Language Understanding Evaluation Benchmark — Gao et al. (2025) (arXiv:2503.12051, 2025)

## What this evaluates

Evaluates large language models' proficiency in Tibetan across general knowledge comprehension and safety-critical domains. It probes the models' ability to handle low-resource language tasks, complex reasoning, and culturally sensitive alignment compared to English baselines.

## Datasets

- **Ti-MMLU** — total ?; splits: test (-1); repo https://github.com/Vicentvankor/TLUE
- **Ti-SafetyBench** — total ?; splits: test (-1); repo https://github.com/Vicentvankor/TLUE

## Metrics

- `Accuracy (ACC)` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total dataset size.
- `Conditional Accuracy (CA)` — range: percent
  - Percentage of correctly answered questions among only those instances where the model provides a direct answer (excluding refusals or non-responses).
- `Response Rate (RR)` — range: percent
  - Percentage of instances where the model produces a non-refusal answer out of the total dataset size.

## Input / output format

**Input**: Multiple-choice questions in Tibetan covering 67 academic/professional subjects (Ti-MMLU) or safety-critical scenarios (Ti-SafetyBench).

**Output**: Model-generated text response, typically a selected option or a direct answer to the prompt.

## Scoring recipe

```python
def calculate_metrics(predictions, golds, refusals):
    total = len(golds)
    rr = sum(1 for p in predictions if p not in refusals) / total
    answered = [(p, g) for p, g in zip(predictions, golds) if p not in refusals]
    acc = sum(1 for p, g in answered if p == g) / total
    ca = sum(1 for p, g in answered if p == g) / len(answered) if answered else 0
    return rr, acc, ca
```

## Common pitfalls

- Models often refuse to answer safety-related prompts in Tibetan, drastically lowering raw Accuracy; Conditional Accuracy (CA) must be used to fairly assess knowledge when refusals occur.
- The benchmark adapts Chinese benchmarks (CMMLU, SafetyBench) via translation, so performance gaps may stem from translation quality or cultural mismatch rather than pure language proficiency.
- Random baselines differ between tasks (25% for Ti-MMLU, ~36.7% for Ti-SafetyBench), so comparing raw scores across domains without accounting for option counts or refusal rates is misleading.

## Evidence (verbatim from paper)

> Table 4: List of Abbreviations for Professional Terms
| Response Rate | RR |
| Accuracy | ACC |
| Conditional Accuracy | CA |

## Citation

```bibtex
@misc{gao2025tlue,
  title={TLUE: A Tibetan Language Understanding Evaluation Benchmark},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2503.12051}
}
```

- arXiv: 2503.12051