Mmwp Eval

---
name: mmwp-eval
description: Evaluates large language models' ability to perform mathematical, commonsense, and natural language inference reasoning in low-, medium-, and high-resource languages. It specifically probes cross-lingual transfer capabilities using a zero-shot chain-of-thought setting without requiring parallel multilingual instruction data. Use when the user wants to benchmark on MMWP, MGSM, MSVAMP, X-CSQA, XNLI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.12499
  bibtex_key: zhang2024lingualift
  confidence: high
---

# mmwp-eval

> LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning — Zhang et al. (2024) (arXiv:2412.12499, 2024)

## What this evaluates

Evaluates large language models' ability to perform mathematical, commonsense, and natural language inference reasoning in low-, medium-, and high-resource languages. It specifically probes cross-lingual transfer capabilities using a zero-shot chain-of-thought setting without requiring parallel multilingual instruction data.

## Datasets

- **MMWP** — total ?; splits: test (-1)
- **MGSM** — total ?; splits: test (-1)
- **MSVAMP** — total ?; splits: test (-1)
- **X-CSQA** — total ?; splits: test (-1)
- **XNLI** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted answers out of the total number of instances, reported as a percentage. Results are aggregated by language resource level (low, medium, high) and averaged across all languages.

## Input / output format

**Input**: Multilingual natural language problem statements (math word problems, commonsense questions, or NLI premises/hypotheses) provided in the target language.

**Output**: Generated text containing reasoning steps followed by a final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize(pred) == normalize(gold))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Results are stratified by language resource level (LR/MR/HR) rather than reported as a single global average, which can obscure performance on specific languages.
- The zero-shot chain-of-thought setting does not provide task-specific prompts or examples, making performance highly sensitive to the model's inherent multilingual reasoning capabilities.
- Baselines like Translate-En and LangBridge use external translation tools or hidden-state replacements, which deviates from the pure zero-shot evaluation setting and complicates direct comparison.

## Evidence (verbatim from paper)

> We use the MMWP and the latest multilingual benchmarks, MGSM (Shi et al., [2023]) and MSVAMP (Chen et al., [2023]) to evaluate the performance of LLMs in multilingual mathematical reasoning through zero-shot chain-of-thought reasoning (Wei et al., [2022]) setting. Table 1: Results on the MMWP benchmark, where “LR.”, “MR.”, and “HR.” denote mean accuracy for low-, medium-, and high-resource languages, respectively.

## Citation

```bibtex
@misc{zhang2024lingualift,
  title={LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.12499}
}
```

- arXiv: 2412.12499