Pmmeval Eval

---
name: pmmeval-eval
description: Evaluates multilingual capabilities of LLMs across understanding, reasoning, and generation tasks in 10 languages. It probes prompt sensitivity and cross-lingual performance consistency to reveal benchmark origin bias and language-specific scaling trends. Use when the user wants to benchmark on MMMLU, MLogiQA, MGSM, MHellaSwag, XNLI, Flores-200, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.09116
  bibtex_key: zhang2024pmmeval
  confidence: high
---

# pmmeval-eval

> P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs — Zhang et al. (2024) (arXiv:2411.09116, 2024)

## What this evaluates

Evaluates multilingual capabilities of LLMs across understanding, reasoning, and generation tasks in 10 languages. It probes prompt sensitivity and cross-lingual performance consistency to reveal benchmark origin bias and language-specific scaling trends.

## Datasets

- **MMMLU** — total ?; splits: test (-1)
- **MLogiQA** — total ?; splits: test (-1)
- **MGSM** — total ?; splits: test (-1)
- **MHellaSwag** — total ?; splits: test (-1)
- **XNLI** — total ?; splits: test (-1)
- **Flores-200** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of total instances. Calculated as (number of exact matches between model output and gold label) / (total number of instances) * 100.
- `BLEU` — range: percent
  - Standard n-gram overlap metric for machine translation evaluation, typically computed as BLEU-4 with sentence-level averaging.
- `COMET` — range: [0, 1]
  - Reference-based neural translation quality score computed using the wmt22-comet-da model, outputting a score typically in [0, 1].

## Input / output format

**Input**: Instruction prompt (English, target language, or English with few-shot demonstrations) concatenated with the input instance in the target language.

**Output**: Direct answer string, or step-by-step Chain of Thought reasoning followed by a final answer (required for MGSM and used for small LLMs on MMMLU).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

def compute_bleu(references, predictions):
    return bleu_score(references, predictions) * 100

def compute_comet(references, predictions):
    return comet_model.score(references, predictions)
```

## Common pitfalls

- Prompt choice drastically changes scores (EN vs Native vs Few-shot), especially for generation tasks where English instructions cause models to output English.
- Chain-of-Thought reasoning causes high answer extraction failure rates on models with fewer than 7B parameters.
- Performance varies significantly based on whether the benchmark originates from English or Chinese training data, revealing benchmark sensitivity.

## Evidence (verbatim from paper)

> For the Flores-200 dataset, in addition to reporting BLEU scores, we also provide COMET scores measured by wmt22-comet-da *[Rei et al.]* (see Appendix, Table [5]).

## Citation

```bibtex
@misc{zhang2024pmmeval,
  title={P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.09116}
}
```

- arXiv: 2411.09116