Hindi Llm Benchmark Eval

---
name: hindi-llm-benchmark-eval
description: Evaluates instruction-following, mathematical reasoning, code/function-calling, and retrieval-augmented generation capabilities of LLMs in Hindi. The benchmark specifically probes the models' ability to handle culturally and linguistically nuanced prompts that go beyond direct English translation. Use when the user wants to benchmark on IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, BFCL-Hi, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.19831
  bibtex_key: kamath2025benchmarkinghindi
  confidence: medium
---

# hindi-llm-benchmark-eval

> Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis — Kamath et al. (2025) (arXiv:2508.19831, 2025)

## What this evaluates

Evaluates instruction-following, mathematical reasoning, code/function-calling, and retrieval-augmented generation capabilities of LLMs in Hindi. The benchmark specifically probes the models' ability to handle culturally and linguistically nuanced prompts that go beyond direct English translation.

## Datasets

- **IFEval-Hi** — total ?; splits: (unstated)
- **MT-Bench-Hi** — total ?; splits: (unstated)
- **GSM8K-Hi** — total ?; splits: (unstated)
- **ChatRAG-Hi** — total ?; splits: (unstated)
- **BFCL-Hi** — total ?; splits: (unstated)

## Metrics

- `score` **(primary)** — range: percent
  - Unspecified in the provided section; generally refers to task-specific pass rate or accuracy (e.g., exact match for math, instruction-following pass rate for IFEval).

## Input / output format

**Input**: Hindi-language instruction prompts or queries, curated via a hybrid human-translation workflow to preserve linguistic and cultural nuance.

**Output**: Model-generated Hindi text responses.

## Scoring recipe

```python
for dataset in [IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, BFCL-Hi]:
  predictions = model.generate(prompts)
  metric_value = compute_task_specific_score(predictions, gold)
  record(metric_value)
aggregate_scores = mean(metric_values across datasets)
return aggregate_scores
```

## Common pitfalls

- Activating dedicated reasoning modes (e.g., in GPT-OSS or Sarvam-M) can artificially inflate scores on complex tasks like BFCL-Hi and GSM8K-Hi if not strictly controlled.
- Using automated judges like GPT-4o may introduce bias toward sibling OpenAI models, skewing evaluation scores.
- Model size alone does not guarantee performance; targeted training and architectural choices heavily influence results across different task domains.

## Evidence (verbatim from paper)

> Gemma-2-9b-it provides the best all-around performance, securing the highest scores on MT-Bench-Hi, IFEval-Hi, and ChatRAG-Hi. ... the potential for the GPT-4o judge to be biased towards a sibling OpenAI model also warrants further investigation.

## Citation

```bibtex
@misc{kamath2025benchmarkinghindi,
  title={Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis},
  author={Kamath et al. (2025)},
  year={2025},
  note={arXiv:2508.19831}
}
```

- arXiv: 2508.19831