Indicaleval Eval

---
name: indicaleval-eval
description: Evaluates large language models' reasoning capabilities on authentic Indian high-stakes examination questions across STEM and humanities domains. It specifically probes bilingual reasoning, cross-lingual performance differentials, and the impact of prompting strategies (Zero-Shot, Few-Shot, Chain-of-Thought) on model accuracy. Use when the user wants to benchmark on IndicEval, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16467
  bibtex_key: bharti2026indicaleval
  confidence: high
---

# indicaleval-eval

> IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models — Bharti et al. (2026) (arXiv:2602.16467, 2026)

## What this evaluates

Evaluates large language models' reasoning capabilities on authentic Indian high-stakes examination questions across STEM and humanities domains. It specifically probes bilingual reasoning, cross-lingual performance differentials, and the impact of prompting strategies (Zero-Shot, Few-Shot, Chain-of-Thought) on model accuracy.

## Datasets

- **IndicEval** — total 1509; splits: test (1509)

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of model responses that exactly match the ground-truth correct option after deterministic parsing. Reported as the mean accuracy across three independent runs per model-prompt combination.

## Input / output format

**Input**: Multiple-choice examination questions in either English or Hindi, presented via standardized prompt templates under Zero-Shot, Few-Shot, or Chain-of-Thought conditions.

**Output**: Model-generated text response containing the selected answer option, automatically parsed using a deterministic extraction module.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    matches = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = matches / len(gold)
    return accuracy
# Final reported metric is the mean accuracy across 3 independent runs
```

## Common pitfalls

- Failing to fix generation parameters (temperature, top-p, max tokens) across models, which introduces stochastic variance.
- Not preserving question order across models, potentially introducing ordering bias.
- Ignoring the need for deterministic answer parsing before exact-match scoring, leading to false negatives due to formatting differences.

## Evidence (verbatim from paper)

> Model responses were logged automatically, parsed using a deterministic extraction module, and evaluated using exact-match scoring. Mean accuracy across runs was reported as the primary performance metric, while variance was monitored to detect stochastic instability.

## Citation

```bibtex
@misc{bharti2026indicaleval,
  title={IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models},
  author={Bharti et al. (2026)},
  year={2026},
  note={arXiv:2602.16467}
}
```

- arXiv: 2602.16467