Mtvqa Eval

---
name: mtvqa-eval
description: This benchmark evaluates the multilingual visual-textual alignment and comprehension capabilities of multimodal large language models (MLLMs). It specifically probes whether models can accurately perceive, extract, and reason about text embedded within images across nine different languages without relying on translation. Use when the user wants to benchmark on MTVQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.11985
  bibtex_key: tang2024mtvqa
  confidence: high
---

# mtvqa-eval

> MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering — Tang et al. (2024) (arXiv:2405.11985, 2024)

## What this evaluates

This benchmark evaluates the multilingual visual-textual alignment and comprehension capabilities of multimodal large language models (MLLMs). It specifically probes whether models can accurately perceive, extract, and reason about text embedded within images across nine different languages without relying on translation.

## Datasets

- **MTVQA** — total ?; splits: train (-1), test (-1); repo https://github.com/bytedance/MTVQA

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Measures the percentage of questions for which the predicted answer matches any of the target answers for the question.

## Input / output format

**Input**: An image containing embedded text and a question in one of nine languages (AR, DE, FR, IT, JA, KO, RU, TH, VI). Models are prompted with: 'Answer the question using a word or phrase in the language of the question. + <Question>'.

**Output**: A concise word or phrase in the language of the question.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, golds in zip(predictions, gold_answers):
        if pred.strip().lower() in [g.strip().lower() for g in golds]:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models show significant performance drops on non-Latin scripts (AR, JA, KO, TH) due to weaker visual text perception compared to Latin-based languages.
- Using external OCR pipelines before feeding text to LLMs/MLLMs degrades performance because OCR strips crucial positional and visual context.
- Relying on the ANLS metric can overestimate model performance by 5-10% compared to exact Accuracy, masking failures in precise text extraction.

## Evidence (verbatim from paper)

> To accurately assess whether the visual text that occurs in the answer is correct, we adopt Accuracy as the metric. The Accuracy metric measures the percentage of questions for which the predicted answer matches any of the target answers for the question.

## Citation

```bibtex
@misc{tang2024mtvqa,
  title={MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2405.11985}
}
```

- arXiv: 2405.11985