Mme Unify Eval

---
name: mme-unify-eval
description: Evaluates unified multimodal large language models (U-MLLMs) on their ability to handle mixed-modality tasks that combine visual understanding, text generation, and sequential reasoning. It probes capabilities such as interleaved image-text generation, visual chain-of-thought reasoning, and image editing with explanations. Use when the user wants to benchmark on MME-Unify, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.03641
  bibtex_key: xie2025mmeunify
  confidence: high
---

# mme-unify-eval

> MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models — Xie et al. (2025) (arXiv:2504.03641, 2025)

## What this evaluates

Evaluates unified multimodal large language models (U-MLLMs) on their ability to handle mixed-modality tasks that combine visual understanding, text generation, and sequential reasoning. It probes capabilities such as interleaved image-text generation, visual chain-of-thought reasoning, and image editing with explanations.

## Datasets

- **MME-Unify** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Standard accuracy calculated as the percentage of instances where the model's output exactly matches the ground truth answer or generated image.
- `Acc+` — range: percent
  - Stricter accuracy metric that requires both the text reasoning component and the image generation component to be correct simultaneously.
- `Unify Score` — range: percent
  - Overall benchmark score aggregating performance across comprehension, generation, and unified task subdomains.

## Input / output format

**Input**: Multimodal prompts consisting of images and text instructions/questions. Unified tasks may include reference images and complex, multi-step instructions requiring sequential reasoning.

**Output**: Text responses and/or generated images. Unified tasks require interleaved text and image outputs, or specific actions and coordinates.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc_list = []
    acc_plus_list = []
    for pred, gold in zip(predictions, golds):
        text_ok = (pred.text == gold.text)
        img_ok = (pred.image == gold.image)
        acc_list.append(1.0 if text_ok or img_ok else 0.0)
        acc_plus_list.append(1.0 if text_ok and img_ok else 0.0)
    return {
        'Acc': sum(acc_list) / len(acc_list) * 100,
        'Acc+': sum(acc_plus_list) / len(acc_plus_list) * 100
    }
```

## Common pitfalls

- Evaluating only text or only image accuracy can mask severe deficiencies in the other modality, as models often excel at one while failing at the other.
- Multi-step Visual CoT tasks suffer from cascading errors; mistakes in early reasoning steps compound, causing final accuracy to drop to near zero even if intermediate outputs are partially correct.
- Complex instruction following for image generation (e.g., drawing auxiliary lines) is frequently ignored, with models producing style-biased or irrelevant images despite correct text reasoning.

## Evidence (verbatim from paper)

> For each unify task in Table[2], we require the models to generate the correct image and perform correct reasoning. Under these conditions, even for simple tasks such as answering common questions and generating images, the best open-sourced model (Anole) only achieves an accuracy of 59.65% and accuracy-plus of 38% (Table[3]).

## Citation

```bibtex
@misc{xie2025mmeunify,
  title={MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2504.03641}
}
```

- arXiv: 2504.03641