Dzen Eval

---
name: dzen-eval
description: Evaluates foundation models' ability to answer multiple-choice academic questions in both English and Dzongkha across varying grade levels and scientific subjects. It specifically probes factual recall, procedural application, and multi-step reasoning capabilities in a low-resource multilingual setting. Use when the user wants to benchmark on DZEN, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18638
  bibtex_key: hosain2025dzen
  confidence: medium
---

# dzen-eval

> Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models — Hosain et al. (2025) (arXiv:2505.18638, 2025)

## What this evaluates

Evaluates foundation models' ability to answer multiple-choice academic questions in both English and Dzongkha across varying grade levels and scientific subjects. It specifically probes factual recall, procedural application, and multi-step reasoning capabilities in a low-resource multilingual setting.

## Datasets

- **DZEN** — total 5161; splits: test (5161); repo https://github.com/kraritt/llm_dzongkha_evaluation

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy: the proportion of questions where the model's predicted option exactly matches the ground truth answer. Calculated as correct predictions divided by total questions.

## Input / output format

**Input**: Multiple-choice question presented in either Dzongkha or English, containing four possible answer options. Mathematical and chemical expressions are formatted in LaTeX.

**Output**: Selection of one of the four provided options (e.g., A, B, C, or D) corresponding to the correct answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = correct / len(gold_answers)
```

## Common pitfalls

- Question ordering differs between English and Dzongkha versions, requiring semantic matching rather than positional alignment.
- Ground truth discrepancies may arise from translation errors or annotator mistakes, necessitating manual verification of parallel pairs.
- Grammar mistakes in English translations can occasionally affect model performance, though the paper notes the effect is often minimal.

## Evidence (verbatim from paper)

> Chain-of-Thought prompting improves reasoning accuracy in Dzongkha when applied directly in the target language, especially when paired with English translations for factual and multi-step questions, while translation-augmented prompts enhance response precision without sacrificing linguistic accessibility.

## Citation

```bibtex
@misc{hosain2025dzen,
  title={Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models},
  author={Hosain et al. (2025)},
  year={2025},
  note={arXiv:2505.18638}
}
```

- arXiv: 2505.18638