Indic Instruct Eval

---
name: indic-instruct-eval
description: This evaluation probes the multilingual instruction-following, natural language understanding, and generation capabilities of LLMs fine-tuned on 13 Indic languages. It measures performance on standardized academic benchmarks across NLU and NLG tasks, as well as real-world cultural relevance and helpfulness through pairwise LLM-as-a-judge comparisons. Use when the user wants to benchmark on MMLU Indic (MMLU-I), ARC Indic (ARC-I), BoolQ Indic (BoolQ-I), TriviaQA Indic (TVQA-I), BeleBele (Bele), INCLUDE (INCL), Global MMLU (GMMLU), Extreme Summarization (Xsum), Flores EnXX / XXEn, IN22-Conv-Doc, or asks about evaluating this task. Reports ELO rating.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.21294
  bibtex_key: chitale2025updesh
  confidence: high
---

# indic-instruct-eval

> UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic — Chitale et al. (2025) (arXiv:2509.21294, 2025)

## What this evaluates

This evaluation probes the multilingual instruction-following, natural language understanding, and generation capabilities of LLMs fine-tuned on 13 Indic languages. It measures performance on standardized academic benchmarks across NLU and NLG tasks, as well as real-world cultural relevance and helpfulness through pairwise LLM-as-a-judge comparisons.

## Datasets

- **MMLU Indic (MMLU-I)** — total ?; splits: test (-1)
- **ARC Indic (ARC-I)** — total ?; splits: test (-1)
- **BoolQ Indic (BoolQ-I)** — total ?; splits: test (-1)
- **TriviaQA Indic (TVQA-I)** — total ?; splits: test (-1)
- **BeleBele (Bele)** — total ?; splits: test (-1)
- **INCLUDE (INCL)** — total ?; splits: test (-1)
- **Global MMLU (GMMLU)** — total ?; splits: test (-1)
- **Extreme Summarization (Xsum)** — total ?; splits: test (-1)
- **Flores EnXX / XXEn** — total ?; splits: test (-1)
- **IN22-Conv-Doc** — total ?; splits: test (-1)

## Metrics

- `NLU average score` — range: [0, 1]
  - Likelihood-based scoring on multiple-choice questions; accuracy or log-probability selection of the correct option averaged across all NLU benchmarks.
- `NLG average score` — range: [0, 1]
  - Average of standard generation metrics (e.g., ROUGE/BERTScore) across translation and summarization tasks.
- `ELO rating` **(primary)** — range: other
  - Pairwise LLM-as-a-judge comparison score updated iteratively; higher ratings indicate better comparative performance across real-world, culturally grounded queries.

## Input / output format

**Input**: Instruction prompts in 13 Indic languages (and English for cross-lingual tasks), formatted as multiple-choice questions for NLU or translation/summarization prompts for NLG.

**Output**: Model-generated text responses (selected option for NLU, translated/summarized text for NLG).

## Scoring recipe

```python
def score_nlu(dataset, model):
    correct = 0
    for q in dataset:
        options = q.options
        best_opt = max(options, key=lambda o: model.log_prob(q.prompt + o))
        if best_opt == q.gold:
            correct += 1
    return correct / len(dataset)

def score_nlg(dataset, model):
    scores = [compute_generation_metric(q.gold, model.generate(q.prompt)) for q in dataset]
    return sum(scores) / len(scores)

def update_elo(models, battles, llm_judge):
    for a, b, prompt in battles:
        judge_win = llm_judge(prompt, models[a].generate(prompt), models[b].generate(prompt))
        elo_update(a, b, judge_win)
    return {m: elo_rating(m) for m in models}
```

## Common pitfalls

- Failing to subsample large baseline datasets (e.g., Aya-Collection, IndicAlign) to match the target dataset size, creating an unfair data/compute advantage.
- Overlooking cross-lingual transfer evaluation on languages completely absent from the training set (e.g., the 16 unseen languages in the Flores subset).
- Relying exclusively on automated academic benchmarks without incorporating LLM-as-a-judge cultural relevance evaluations, which miss real-world helpfulness.

## Evidence (verbatim from paper)

> Natural language understanding (NLU) tasks use multiple-choice questions to measure comprehension and reasoning through likelihood-based scoring. Natural language generation (NLG) tasks, such as translation and summarization, assess models’ ability to generate coherent and contextually appropriate outputs. We augment standard dataset-NLU and NLG evaluations with comparative evaluations to understand model win rates.

## Citation

```bibtex
@misc{chitale2025updesh,
  title={UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic},
  author={Chitale et al. (2025)},
  year={2025},
  note={arXiv:2509.21294}
}
```

- arXiv: 2509.21294