Milu Eval

---
name: milu-eval
description: Evaluates large language models on multi-task understanding across 11 Indic languages, covering 8 domains and 41 subjects. It probes cultural knowledge, region-specific exam data, and multilingual reasoning capabilities. Use when the user wants to benchmark on MILU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.02538
  bibtex_key: verma2024milu
  confidence: high
---

# milu-eval

> MILU: A Multi-task Indic Language Understanding Benchmark — Verma et al. (2024) (arXiv:2411.02538, 2024)

## What this evaluates

Evaluates large language models on multi-task understanding across 11 Indic languages, covering 8 domains and 41 subjects. It probes cultural knowledge, region-specific exam data, and multilingual reasoning capabilities.

## Datasets

- **MILU** — total ?; splits: validation (9000); repo https://github.com/AI4Bharat/MILU

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted answers. For multiple-choice questions, the answer with the highest conditional log-likelihood is selected. For generative API models, the parsed JSON response is compared against the gold answer.

## Input / output format

**Input**: Question text, optionally preceded by few-shot examples from a validation set. For API models, prompts request structured JSON output.

**Output**: For non-API models: the selected answer string (multiple choice). For API models: a structured JSON object containing the predicted response.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # pred is either the argmax log-likelihood answer or parsed JSON string
        if pred == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Log-likelihood scoring for non-API models differs from generative scoring for API models, making direct comparison sensitive to the evaluation setup.
- Few-shot examples are drawn from a separate ~9,000 question validation set, not the test set itself.
- API models are only evaluated in zero-shot due to cost, limiting few-shot analysis for proprietary models.

## Evidence (verbatim from paper)

> For multiple choice questions, given $k$ possible answer strings, we select the answer string $(a_{i})$ with the highest conditional log probability, i.e., $argmax(logP(a_{1}|x),...,logP(a_{k}|x))$. The API-based models are evaluated using the generative approach... explicitly prompt these models to generate the correct response in a structured JSON format. It reveals that open multilingual models outperform language-specific ones, with GPT-4o achieving 74% average accuracy.

## Citation

```bibtex
@misc{verma2024milu,
  title={MILU: A Multi-task Indic Language Understanding Benchmark},
  author={Verma et al. (2024)},
  year={2024},
  note={arXiv:2411.02538}
}
```

- arXiv: 2411.02538