Mm Judgebench Eval

---
name: mm-judgebench-eval
description: Evaluates the cross-lingual generalization and robustness of Large Vision-Language Models (LVLMs) acting as automated judges. It probes their ability to correctly rank paired multimodal responses across 25 languages while measuring susceptibility to positional and length biases. Use when the user wants to benchmark on MM-JudgeBench, or asks about evaluating this task. Reports average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19405
  bibtex_key: laskar2026lost
  confidence: high
---

# mm-judgebench-eval

> Lost in Translation: Do LVLM Judges Generalize Across Languages? — Md Tahmid Rahman Laskar et al. (arXiv:2604.19405, 2026)

## What this evaluates

Evaluates the cross-lingual generalization and robustness of Large Vision-Language Models (LVLMs) acting as automated judges. It probes their ability to correctly rank paired multimodal responses across 25 languages while measuring susceptibility to positional and length biases.

## Datasets

- **MM-JudgeBench** — total 60000; splits: test (-1); repo https://github.com/tahmedge/mm-judgebench

## Metrics

- `average accuracy` **(primary)** — range: [0, 1]
  - Accuracy of correctly identifying the preferred answer in a pairwise comparison. Evaluated by running the model on both original and reversed answer orders independently, then averaging the accuracy across both runs. Parsing failures are counted as incorrect.

## Input / output format

**Input**: Multimodal prompt containing an image, a question, and two candidate answers (Answer A and Answer B) in a target language. Models are evaluated twice per instance: once with the original order and once with the answers reversed.

**Output**: JSON-formatted string containing the predicted winner (e.g., 'A' or 'B') and a textual reasoning explanation.

## Scoring recipe

```python
correct = 0
total = 0
for instance in dataset:
    pred_orig = parse_json(model.generate(instance, order='original'))
    pred_rev = parse_json(model.generate(instance, order='reversed'))
    correct += (pred_orig == instance.gold) + (pred_rev == instance.gold)
    total += 2
return correct / total
```

## Common pitfalls

- If the model's output cannot be parsed into the expected JSON format, the instance is automatically scored as incorrect.
- Positional bias can artificially inflate accuracy if only the original answer order is evaluated; reversing the order is required for robust measurement.
- LaBSE scores for reasoning quality are only reported on the M-OpenCQA subset, not the full benchmark.

## Evidence (verbatim from paper)

> To ensure robustness, we generate the judgment by providing the candidate answers in both the original and the reversed order independently, and then report the average accuracy. In addition, we evaluate biases like positional bias and length bias. As the LVLMs generate the winner alongside the reasoning, we parse the LVLM-judge predicted winner from their corresponding JSON-formatted responses using a parsing script Laskar et al. ([2024](#bib.bib42 "A systematic survey and critical review on evaluating large language models: challenges, limitations, and recommendations")). If the parsing script cannot properly parse the judgment from the response, we consider the LVLM-generated answer wrong.

## Citation

```bibtex
@misc{laskar2026lost,
  title={Lost in Translation: Do LVLM Judges Generalize Across Languages?},
  author={Md Tahmid Rahman Laskar et al.},
  year={2026},
  note={arXiv:2604.19405}
}
```

- arXiv: 2604.19405