Multilingual Csr Eval

---
name: multilingual-csr-eval
description: Evaluates multilingual language models on cross-lingual commonsense reasoning and plausibility. It probes whether models can rank correct assertions and answer multiple-choice questions across 11+ languages. Use when the user wants to benchmark on MickeyProbe, X-CODAH, X-CSQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.06937
  bibtex_key: lin2021commonsensebeyondenglish
  confidence: high
---

# multilingual-csr-eval

> Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning — Lin et al. (2021) (arXiv:2106.06937, 2021)

## What this evaluates

Evaluates multilingual language models on cross-lingual commonsense reasoning and plausibility. It probes whether models can rank correct assertions and answer multiple-choice questions across 11+ languages.

## Datasets

- **MickeyProbe** — total 561000; splits: test (-1)
- **X-CODAH** — total ?; splits: test (-1)
- **X-CSQA** — total ?; splits: test (-1)

## Metrics

- `Hit@1 Accuracy` — range: percent
  - Percentage of instances where the correct assertion is ranked exactly first.
- `Hit@2 Accuracy` — range: percent
  - Percentage of instances where the correct assertion is ranked within the top 2.
- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions or ranked assertions.

## Input / output format

**Input**: For MickeyProbe: a context sentence and a set of candidate assertions in a target language. For X-CODAH/X-CSQA: a question/context and multiple choice options in a target language.

**Output**: For MickeyProbe: a ranked list of assertions. For X-CODAH/X-CSQA: a single selected option label.

## Scoring recipe

```python
def compute_hit_at_k(predictions, gold, k):
    return int(gold in predictions[:k])

def compute_accuracy(predictions, gold):
    return int(predictions == gold)
```

## Common pitfalls

- Hit@2 is explicitly noted as more flexible than Hit@1, so reporting only Hit@1 underestimates performance.
- X-CODAH results are broken down by linguistic categories (Idioms, Neg., Poly., Ref., Quant., Others), which vary significantly in difficulty across languages.
- Fine-tuning hyperparameters (learning rate, epochs, batch size) differ significantly between models and datasets, requiring careful reproduction from Table 7.

## Evidence (verbatim from paper)

> Table 5 shows the Hit@2 Accuracy of the five MLLMs for the MickeyProbe. Hit@2 Accuracy evaluates whether the models can rank the correct assertion within top 2. Unlike Hit@1 which only accepts best predictions, Hit@2 is more flexible. Thus, the performances in Hit@2 increase compared to the ones in Hit@1. We can see that the discrepancies across languages still exist.

## Citation

```bibtex
@misc{lin2021commonsensebeyondenglish,
  title={Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2106.06937}
}
```

- arXiv: 2106.06937