Mmtr Bench Eval

---
name: mmtr-bench-eval
description: Evaluates Multimodal Large Language Models' ability to reconstruct masked text from visual context without explicit prompts. It probes layout understanding, visual grounding, and world knowledge integration by requiring models to infer missing content from surrounding text, charts, and multi-page evidence. Use when the user wants to benchmark on MMTR-Bench, or asks about evaluating this task. Reports exact-match / semantic-similarity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21277
  bibtex_key: guo2026canmllmsread
  confidence: high
---

# mmtr-bench-eval

> Can MLLMs "Read" What is Missing? — Guo et al. (2026) (arXiv:2604.21277, 2026)

## What this evaluates

Evaluates Multimodal Large Language Models' ability to reconstruct masked text from visual context without explicit prompts. It probes layout understanding, visual grounding, and world knowledge integration by requiring models to infer missing content from surrounding text, charts, and multi-page evidence.

## Datasets

- **MMTR-Bench** — total 2771; splits: test (2771)

## Metrics

- `exact-match / semantic-similarity` **(primary)** — range: [0, 1]
  - Level-aware evaluation protocol: exact matching is applied to short targets (Level 1), while semantic similarity combined with a factuality gate is used for longer, complex reconstructions (Levels 2–4). An LLM-as-Judge performs automated assessment to determine the final score.

## Input / output format

**Input**: One or more images containing text with locally masked regions. No explicit question or prompt is provided.

**Output**: Recovered target text corresponding to the masked region(s).

## Scoring recipe

```python
def evaluate(predictions, gold, level):
    if level == 1:
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        sim = compute_semantic_similarity(predictions, gold)
        factual = factuality_gate(predictions, gold)
        return sim if factual else 0.0
# Aggregate scores across all 2771 test samples; LLM-as-Judge used for final automated assessment
```

## Common pitfalls

- Assuming the task is standard VQA; there is no explicit question, requiring the model to autonomously identify relevant visual regions.
- Expecting random masking; masking is carefully curated by humans to ensure deterministic answers and clear contextual relationships.
- Overlooking the level-aware scoring; short targets use exact match while longer ones use semantic similarity + factuality, not a single uniform metric.

## Evidence (verbatim from paper)

> exact matching for short targets and semantic similarity with factuality gates for longer, complex reconstructions, revealing significant performance gaps between closed-source and open-source models, especially in sentence- and paragraph-level recovery.

## Citation

```bibtex
@misc{guo2026canmllmsread,
  title={Can MLLMs "Read" What is Missing?},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.21277}
}
```

- arXiv: 2604.21277