Xnli Sib200 Multilingual Eval

---
name: xnli-sib200-multilingual-eval
description: Evaluates multilingual and cross-lingual capabilities of LLMs on Natural Language Inference (XNLI) and topic classification (SIB-200) across English and low-resource South Asian languages (Bangla, Hindi, Urdu) using zero-shot prompting. Use when the user wants to benchmark on XNLI, SIB-200, or asks about evaluating this task. Reports F1macro.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13153
  bibtex_key: deya2024better
  confidence: high
---

# xnli-sib200-multilingual-eval

> Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings — Deya et al. (2024) (arXiv:2410.13153, 2024)

## What this evaluates

Evaluates multilingual and cross-lingual capabilities of LLMs on Natural Language Inference (XNLI) and topic classification (SIB-200) across English and low-resource South Asian languages (Bangla, Hindi, Urdu) using zero-shot prompting.

## Datasets

- **XNLI** — total ?; splits: test (-1)
- **SIB-200** — total ?; splits: test (-1)

## Metrics

- `Accuracy` — range: percent
  - Proportion of correctly predicted class labels out of the total number of samples.
- `F1macro` **(primary)** — range: percent
  - Macro-averaged F1 score computed across all prediction classes, treating each class equally regardless of support size.

## Input / output format

**Input**: Zero-shot prompts in English, Bangla, Hindi, or Urdu, including original and cross-lingually translated variants (settings P1–P5).

**Output**: Predicted class label (e.g., 'Entailment', 'Contradiction', 'Neutral' for XNLI; topic names for SIB-200), often accompanied by extraneous text or invalid characters requiring post-processing.

## Scoring recipe

```python
def evaluate(predictions, gold):
    clean_preds = [extract_valid_label(p) for p in predictions]
    # Map invalid/unknown labels to inverse class as per paper protocol
    clean_preds = [map_invalid(p) if is_invalid(p) else p for p in clean_preds]
    acc = sum(1 for p, g in zip(clean_preds, gold) if p == g) / len(gold)
    f1 = macro_f1(gold, clean_preds)
    return {'accuracy': acc * 100, 'f1macro': f1 * 100}
```

## Common pitfalls

- LLMs frequently return extraneous characters or invalid labels alongside class predictions, requiring strict post-processing before scoring.
- Safety filters (e.g., in Gemini Pro) may block harmful content samples, resulting in missing predictions that must be tracked and handled.
- Cross-lingual prompt translation does not reliably improve performance; English prompts consistently outperform translated low-resource variants.

## Evidence (verbatim from paper)

> Table 4: Class-wise F1macro score for GPT-4, Llama 2, and Gemini across five prompt settings for the XNLI dataset. Lang.: Language, BN: Bangla, EN: English, HI: Hindi, and UR: Urdu, Cont: contradiction, Ent: Entailment, Neut: Neutral.

## Citation

```bibtex
@misc{deya2024better,
  title={Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings},
  author={Deya et al. (2024)},
  year={2024},
  note={arXiv:2410.13153}
}
```

- arXiv: 2410.13153