Sib200 Xlt Eval

---
name: sib200-xlt-eval
description: This evaluation probes a model's ability to perform zero-shot and fully-supervised cross-lingual text classification across typologically diverse languages. It specifically measures how well parameter-efficient soft prompt tuning methods transfer knowledge from high-resource source languages to low-performing or unseen target languages without language-specific fine-tuning. Use when the user wants to benchmark on SIB-200, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.10352
  bibtex_key: [REDACTED]
  confidence: high
---

# sib200-xlt-eval

> Cross-Prompt Encoder for Low-Performing Languages — Mikaberidze et al. (2025) (arXiv:2508.10352, 2025)

## What this evaluates

This evaluation probes a model's ability to perform zero-shot and fully-supervised cross-lingual text classification across typologically diverse languages. It specifically measures how well parameter-efficient soft prompt tuning methods transfer knowledge from high-resource source languages to low-performing or unseen target languages without language-specific fine-tuning.

## Datasets

- **SIB-200** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted class labels out of the total number of instances. Calculated as (number of correct predictions) / (total number of predictions).

## Input / output format

**Input**: Raw text samples in one of ~200 target languages, processed by a frozen XLM-R large encoder with a lightweight classification head.

**Output**: A single predicted topic class label per text instance.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Confusing zero-shot cross-lingual transfer (ZS-XLT), which involves training on source languages and evaluating on targets, with pure zero-shot prompting of LLMs that receive no task-specific tuning.
- Using shared or language-specific verbalizers instead of a lightweight classification head, which introduces cross-lingual bias or fails for languages outside the tokenizer's vocabulary.
- Aggregating performance across language groups (Seen, Unseen, Low-Performing) without accounting for their strict subset relationships, which can mask transfer dynamics.

## Evidence (verbatim from paper)

> We evaluate on the SIB-200 benchmark, a multilingual topic classification dataset covering 200 typologically diverse languages. ... Table 1: ZS-XLT performance (accuracy) across different target groups.

## Citation

```bibtex
@misc{mikaberidze2025crossprompt,
  title={Cross-Prompt Encoder for Low-Performing Languages},
  author={Mikaberidze et al. (2025)},
  year={2025},
  note={arXiv:2508.10352}
}
```

- arXiv: 2508.10352