Maxm Eval

---
name: maxm-eval
description: This benchmark evaluates multilingual visual question answering (mVQA) by testing models on images paired with questions in seven languages. It probes a model's ability to perform cross-lingual visual reasoning and generate accurate text answers without relying on costly human annotation. Use when the user wants to benchmark on MaXM, or asks about evaluating this task. Reports Exact Match Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.05401
  bibtex_key: changpinyo2022maxm
  confidence: high
---

# maxm-eval

> MaXM: Towards Multilingual Visual Question Answering — Changpinyo et al. (2022) (arXiv:2209.05401, 2022)

## What this evaluates

This benchmark evaluates multilingual visual question answering (mVQA) by testing models on images paired with questions in seven languages. It probes a model's ability to perform cross-lingual visual reasoning and generate accurate text answers without relying on costly human annotation.

## Datasets

- **MaXM** — total ?; splits: test (-1); repo https://github.com/google-research-datasets/maxm

## Metrics

- `Exact Match Accuracy` **(primary)** — range: [0, 1]
  - The prediction is deemed correct if it exactly matches any of the provided ground-truth answers.
- `CIDEr` — range: [0, 1]
  - An n-gram based similarity metric that weights n-grams by their IDF scores across references. Each ground-truth answer is treated as a separate reference.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence between the prediction and reference answers. Each ground-truth answer is treated as a separate reference.

## Input / output format

**Input**: An image and a natural language question in one of seven target languages (en, fr, hi, iw, ro, th, zh).

**Output**: A single text string representing the predicted answer.

## Scoring recipe

```python
def score_exact_match(predictions, gold_answers):
    correct = 0
    for pred, golds in zip(predictions, gold_answers):
        if pred in golds:
            correct += 1
    return correct / len(predictions)

def score_soft_metrics(predictions, gold_answers, metric_fn):
    scores = []
    for pred, golds in zip(predictions, gold_answers):
        # Treat each GT answer as a separate reference
        score = max(metric_fn(pred, [g]) for g in golds)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- MaXM is strictly a test-only benchmark; using it for training violates the protocol.
- Soft metrics like CIDEr and ROUGE-L can mask performance gaps in certain languages (e.g., Thai, Chinese) compared to strict Exact Match Accuracy.
- Applying English-only VQA models directly to non-English questions (Translate-Test) often fails due to tokenizer/vocabulary mismatches, requiring careful adaptation or Translate-Train strategies.

## Evidence (verbatim from paper)

> We use Exact Match Accuracy as the main evaluation measure for MaXM, following previous work on VQA... We deem the answer as correct if it matches any of the ground-truth answers. To assess the degree of strictness of this measure, we also consider soft text similarity metrics CIDEr and ROUGE-L in our experiments, where we treat each of the ground-truth answers equally as one of the references (as if each of them was answered by an annotator).

## Citation

```bibtex
@misc{changpinyo2022maxm,
  title={MaXM: Towards Multilingual Visual Question Answering},
  author={Changpinyo et al. (2022)},
  year={2022},
  note={arXiv:2209.05401}
}
```

- arXiv: 2209.05401