Mmmg Eval

---
name: mmmg-eval
description: This benchmark evaluates text-to-image reasoning capabilities by requiring models to generate domain-specific diagrams, charts, and mindmaps from vague prompts. It probes factual fidelity against annotated knowledge graphs and visual clarity across six educational tiers, revealing deficits in compositional planning and abstract reasoning. Use when the user wants to benchmark on MMMG, or asks about evaluating this task. Reports MMMG-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10963
  bibtex_key: luo2025mmmg
  confidence: high
---

# mmmg-eval

> MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning — Luo et al. (2025) (arXiv:2506.10963, 2025)

## What this evaluates

This benchmark evaluates text-to-image reasoning capabilities by requiring models to generate domain-specific diagrams, charts, and mindmaps from vague prompts. It probes factual fidelity against annotated knowledge graphs and visual clarity across six educational tiers, revealing deficits in compositional planning and abstract reasoning.

## Datasets

- **MMMG** — total 4456; splits: test (4456)

## Metrics

- `MMMG-Score` **(primary)** — range: [0, 100]
  - A composite metric combining graph-edit distance (measuring factual accuracy against expert-annotated knowledge graphs) and SAM-2-based visual clarity metrics. Scores are scaled by 100 for reporting.

## Input / output format

**Input**: A text prompt describing a domain-specific concept at a given educational tier, optionally accompanied by a reasoning trace (for reasoning-augmented models).

**Output**: A generated image (e.g., diagram, chart, mind map, or infographic) representing the prompt's content.

## Scoring recipe

```python
def compute_mmmg_score(generated_image, gold_kg):
    # Extract knowledge graph from generated image
    gen_kg = extract_knowledge_graph(generated_image)
    # Compute factual fidelity via graph-edit distance
    factual_score = 1.0 - (graph_edit_distance(gen_kg, gold_kg) / max_nodes)
    # Compute visual clarity using SAM-2 segmentation
    clarity_score = compute_sam2_clarity(generated_image)
    # Combine and scale to 0-100
    return (factual_score * 0.5 + clarity_score * 0.5) * 100
```

## Common pitfalls

- LLM-as-a-judge (WIScore) and aesthetic metrics (AES-2.5) show poor correlation with human judgments for knowledge-dense visuals.
- FID is negatively correlated with human scores, making it an unreliable proxy for this task.
- Models may achieve high visual clarity but still fail on dependency structure/relationship accuracy, which the metric penalizes.

## Evidence (verbatim from paper)

> MMMG-Score combines graph-edit distance (for factual accuracy) and SAM-2-based visual clarity metrics, revealing severe reasoning deficits in state-of-the-art models (e.g., GPT-4o achieves only 50.20)... To assess alignment with human perception, we collected over 1,200 expert ratings (0–10 on clarity, correctness, accuracy and faithfulness) across six educational levels. We compared four metrics—MMMG-Score; an LLM-as-a-judge WIScore [27] with OpenAI-o3 evaluator; FID computed over 3,452 ground-truth images; and AES-2.5 [1]. Figure 8 reports their Pearson correlations against human scores: MMMG-Score leads with r = 0.876

## Citation

```bibtex
@misc{luo2025mmmg,
  title={MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2506.10963}
}
```

- arXiv: 2506.10963