Mmdeepresearch Bench Eval

---
name: mmdeepresearch-bench-eval
description: Evaluates multimodal deep research agents on iterative retrieval, citation-grounded reasoning, and long-form report synthesis. It probes how well models align textual claims with visual evidence, maintain citation discipline, and produce high-quality structured reports under multimodal constraints. Use when the user wants to benchmark on MMDeepResearch-Bench, or asks about evaluating this task. Reports Overall MMDR-Bench Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.12346
  bibtex_key: [REDACTED]
  confidence: high
---

# mmdeepresearch-bench-eval

> MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents — Huang et al. (2026) (arXiv:2601.12346, 2026)

## What this evaluates

Evaluates multimodal deep research agents on iterative retrieval, citation-grounded reasoning, and long-form report synthesis. It probes how well models align textual claims with visual evidence, maintain citation discipline, and produce high-quality structured reports under multimodal constraints.

## Datasets

- **MMDeepResearch-Bench** — total 140; splits: test (140)

## Metrics

- `Overall MMDR-Bench Score` **(primary)** — range: [0, 100]
  - Weighted combination of three modules: 0.2 × FLAE + 0.5 × TRACE + 0.3 × MOSAIC. MOSAIC is gated with thresholds τ_F = τ_T = 0. Each module score is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru)).
- `FLAE` — range: [0, 100]
  - Report quality module scoring Readability, Instruction Following, and Structure on a 0–100 scale.
- `TRACE` — range: [0, 100]
  - Citation alignment module scoring Visual Evidence Fidelity (Vef.), Consistency, Coverage, and Fidelity on a 0–100 scale.
- `MOSAIC` — range: [0, 100]
  - Text–visual consistency module scoring Semantic alignment, Accuracy, and VQA on a 0–100 scale.

## Input / output format

**Input**: Multimodal research task prompts containing textual queries and supporting visual evidence (images, charts, screenshots), often requiring iterative web search and retrieval.

**Output**: Long-form multimodal research report containing synthesized text, embedded figures/images, and citation-grounded claims with source URLs/snippets.

## Scoring recipe

```python
def compute_mmdr(flae, trace, mosaic):
    # Each module is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru))
    # MOSAIC is gated with thresholds tau_F = tau_T = 0
    overall = 0.2 * flae + 0.5 * trace + 0.3 * max(mosaic, 0)
    return overall
# Unscorable cases receive a reason-aware validity penalty per Appendix A.5.2
```

## Common pitfalls

- Assuming multimodal inputs always improve performance; the benchmark shows vision can introduce noise and degrade citation discipline if visual grounding is unreliable.
- Equating high writing quality (FLAE) with faithful evidence use; strong prose does not guarantee accurate citation alignment or visual consistency (TRACE/MOSAIC).
- Overinterpreting absolute judge scores; different judge LLMs yield different absolute scores due to inductive biases, though relative rankings remain stable.

## Evidence (verbatim from paper)

> The overall MMDR-Bench score is a weighted combination of the three modules: FLAE (20%), TRACE (50%), and MOSAIC (30%). We assign the largest weight to TRACE because citation-grounded evidence quality is the most central requirement for deep research, while MOSAIC evaluates the additional report-quality constraints introduced by visual evidence.

## Citation

```bibtex
@misc{huang2026mmdeepresearchbench,
  title={MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2601.12346}
}
```

- arXiv: 2601.12346