Indicmmlu Pro Eval

---
name: indicmmlu-pro-eval
description: Evaluates large language models on multi-task language understanding across nine major Indic languages. It probes capabilities in reading comprehension, reasoning, and knowledge retention by adapting the English MMLU-Pro benchmark through machine translation and rigorous quality assurance. Use when the user wants to benchmark on IndicMMLU-Pro, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.15747
  bibtex_key: sankalp2025indicmmlupro
  confidence: high
---

# indicmmlu-pro-eval

> IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding — Sankalp KJ et al. (2025) (arXiv:2501.15747, 2025)

## What this evaluates

Evaluates large language models on multi-task language understanding across nine major Indic languages. It probes capabilities in reading comprehension, reasoning, and knowledge retention by adapting the English MMLU-Pro benchmark through machine translation and rigorous quality assurance.

## Datasets

- **IndicMMLU-Pro** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total number of items evaluated.
- `chrF++` — range: percent
  - Character n-gram F1 score (n=6) measuring translation quality between original and back-translated text.
- `BLEU` — range: percent
  - N-gram precision score with a brevity penalty to penalize short translations.

## Input / output format

**Input**: Multiple-choice question with four options (A, B, C, D) presented in one of nine Indic languages.

**Output**: A single selected option letter (A, B, C, or D) corresponding to the correct answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The dataset was generated via machine translation (IndicTrans2) and back-translation, so translation artifacts or cultural mismatches may affect model performance independently of true language understanding.
- Quality assessment metrics (chrF++, BLEU, etc.) are only reported for three languages, leaving the remaining six languages without quantitative translation quality validation.
- Cosine similarity scores are used as a proxy for semantic alignment but do not replace direct human evaluation of translation fidelity.

## Evidence (verbatim from paper)

> Table 1: Performance comparison of language models on the IndicMMLU-Pro benchmark across nine Indic languages, including Indo-Aryan (Hindi, Bengali, Punjabi, Marathi, Urdu, and Gujarati) and Dravidian (Telugu, Tamil, and Kannada) languages. Accuracy scores are shown as percentages.

## Citation

```bibtex
@misc{sankalp2025indicmmlupro,
  title={IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding},
  author={Sankalp KJ et al. (2025)},
  year={2025},
  note={arXiv:2501.15747}
}
```

- arXiv: 2501.15747