Multilingual Vlm Bench Eval

---
name: multilingual-vlm-bench-eval
description: Evaluates vision-language models on translated benchmarks to measure cross-lingual transfer and check for performance degradation on English. It probes the model's ability to understand images and answer multiple-choice or yes/no questions in multiple European languages (DE, ES, FR, IT) while maintaining English proficiency. Use when the user wants to benchmark on MMBench (translated), ScienceQA (translated), MME (translated), POPE (translated), AI2D (translated), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18347
  bibtex_key: baiamonte2026multipixmo
  confidence: high
---

# multilingual-vlm-bench-eval

> Multilingual Training and Evaluation Resources for Vision-Language Models — Baiamonte et al. (2026) (arXiv:2604.18347, 2026)

## What this evaluates

Evaluates vision-language models on translated benchmarks to measure cross-lingual transfer and check for performance degradation on English. It probes the model's ability to understand images and answer multiple-choice or yes/no questions in multiple European languages (DE, ES, FR, IT) while maintaining English proficiency.

## Datasets

- **MMBench (translated)** — total ?; splits: test (-1)
- **ScienceQA (translated)** — total ?; splits: test (-1)
- **MME (translated)** — total ?; splits: test (-1)
- **POPE (translated)** — total ?; splits: test (-1)
- **AI2D (translated)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted options out of total instances.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall on binary yes/no answers.
- `MME composite score` — range: other
  - Aggregated score across multiple sub-tasks as defined by the original MME benchmark.

## Input / output format

**Input**: Image paired with a multilingual (or English) multiple-choice or yes/no question.

**Output**: Model generates a single token or short string corresponding to the correct option (e.g., 'A', 'B', 'C', 'D') or 'yes'/'no'.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type == 'F1':
        tp = sum(p == g == 'yes' for p, g in zip(predictions, golds))
        fp = sum(p == 'yes' and g == 'no' for p, g in zip(predictions, golds))
        fn = sum(p == 'no' and g == 'yes' for p, g in zip(predictions, golds))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif metric_type == 'MME_composite':
        return aggregate_mme_subtask_scores(predictions, golds)
```

## Common pitfalls

- Using LLM-as-a-judge for answer extraction (paper explicitly mandates heuristic extraction).
- Forgetting to average results across the four non-English languages (DE, ES, FR, IT) when reporting multilingual performance.
- Misinterpreting the MME composite score, which aggregates across multiple sub-tasks rather than being a simple accuracy.

## Evidence (verbatim from paper)

> For AI2D, ScienceQA and MMBench, we report the accuracy, while for POPE we report F1 score on binary yes/no answers, while for MME, we used the original composite score. We perform the evaluation with VLMEvalKit using heuristic answer extraction (no LLM-as-a-judge) and greedy decoding.

## Citation

```bibtex
@misc{baiamonte2026multipixmo,
  title={Multilingual Training and Evaluation Resources for Vision-Language Models},
  author={Baiamonte et al. (2026)},
  year={2026},
  note={arXiv:2604.18347}
}
```

- arXiv: 2604.18347