Indicxnli Eval

---
name: indicxnli-eval
description: Evaluates multilingual natural language inference capabilities across 11 Indic languages, probing both intra-lingual reasoning and cross-lingual transfer performance of pre-trained language models. Use when the user wants to benchmark on IndicXNLI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.08776
  bibtex_key: aggarwal2022indicxnli
  confidence: high
---

# indicxnli-eval

> IndicXNLI: Evaluating Multilingual Inference for Indian Languages — Aggarwal et al. (2022) (arXiv:2204.08776, 2022)

## What this evaluates

Evaluates multilingual natural language inference capabilities across 11 Indic languages, probing both intra-lingual reasoning and cross-lingual transfer performance of pre-trained language models.

## Datasets

- **IndicXNLI** — total ?; splits: train (-1), test (-1); repo https://github.com/divyanshuaggarwal/indicxnli

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted inference labels (entailment, contradiction, neutral) out of the total number of instances.

## Input / output format

**Input**: A pair of sentences: a premise and a hypothesis. Inputs are either monolingual in one of 11 Indic languages, or mixed-language (English premise with an Indic hypothesis).

**Output**: A single predicted label from the set {entailment, contradiction, neutral}.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The dataset is machine-translated from English XNLI, so evaluation scores conflate model reasoning ability with translation quality and semantic drift.
- Performance is highly sensitive to the fine-tuning strategy (e.g., English+Indic Train vs. Cross-lingual Transfer), so results are not directly comparable across different training protocols.
- Scores vary drastically based on language resource availability and script similarity, meaning low-resource languages (e.g., Odia, Assamese) will naturally underperform regardless of model architecture.

## Evidence (verbatim from paper)

> The classifiers take two sentence as input, i.e. the premise and the hypothesis and predicts the inference label. LangAvg represents the language wise average score across models, while ModAvg average score represents the model average score across languages.

## Citation

```bibtex
@misc{aggarwal2022indicxnli,
  title={IndicXNLI: Evaluating Multilingual Inference for Indian Languages},
  author={Aggarwal et al. (2022)},
  year={2022},
  note={arXiv:2204.08776}
}
```

- arXiv: 2204.08776