Emu35 T2i X2i Eval

---
name: emu35-t2i-x2i-eval
description: Evaluates a multimodal model's capability to generate images from text prompts and edit existing images based on natural language instructions. It probes semantic alignment, fine-grained text rendering accuracy, and instruction-following fidelity across diverse visual tasks. Use when the user wants to benchmark on GenEval, DPG-bench, OneIG-Bench, TIIF-Bench mini, LeX-Bench, CVTG-2K, LongText-Bench, ImgEdit, GEdit-Bench, OmniContext, ICE-Bench, or asks about evaluating this task. Reports Word Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26583
  bibtex_key: cui2025emu35
  confidence: high
---

# emu35-t2i-x2i-eval

> Emu3.5: Native Multimodal Models are World Learners — Yufeng Cui et al. (2025) (arXiv:2510.26583, 2025)

## What this evaluates

Evaluates a multimodal model's capability to generate images from text prompts and edit existing images based on natural language instructions. It probes semantic alignment, fine-grained text rendering accuracy, and instruction-following fidelity across diverse visual tasks.

## Datasets

- **GenEval** — total ?; splits: test (-1)
- **DPG-bench** — total ?; splits: test (-1)
- **OneIG-Bench** — total ?; splits: test (-1)
- **TIIF-Bench mini** — total ?; splits: test (-1)
- **LeX-Bench** — total 1310; splits: test (1310)
- **CVTG-2K** — total 2000; splits: test (2000)
- **LongText-Bench** — total ?; splits: test (-1)
- **ImgEdit** — total 737; splits: test (737)
- **GEdit-Bench** — total 606; splits: test (606)
- **OmniContext** — total ?; splits: test (-1)
- **ICE-Bench** — total ?; splits: test (-1)

## Metrics

- `Word Accuracy` **(primary)** — range: percent
  - Exact match rate of rendered text characters in the generated image compared to the prompt. Higher is better.
- `Normalized Edit Distance (NED)` — range: [0, 1]
  - 1 - (Edit Distance / max(len(pred), len(gold))). Measures character-level similarity; higher is better.
- `G_O` — range: [0, 5]
  - Overall score computed as the mean of GPT-4o/GPT-4.1 judge scores across all samples in the benchmark. Not the mean of sub-metrics.
- `Overall (OneIG-Bench)` — range: [0, 1]
  - Average of five dimensions: Alignment, Text, Reasoning, Style, and Diversity.

## Input / output format

**Input**: Text prompt (for T2I) or image(s) + natural language instruction (for X2I/editing).

**Output**: Generated image (T2I) or edited image (X2I). Evaluated at ~1024x1024 resolution for X2I, up to 2048px for T2I.

## Scoring recipe

```python
def compute_metrics(predictions, golds, benchmark):
    if benchmark == 'CVTG-2K':
        word_acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
        ned = [1 - edit_distance(p, g) / max(len(p), len(g)) for p, g in zip(predictions, golds)]
        return {'Word Accuracy': word_acc, 'NED': mean(ned)}
    elif benchmark in ['ImgEdit', 'GEdit-Bench']:
        # LLM-as-judge scoring (GPT-4.1 or GPT-4o)
        scores = [llm_judge_score(p, g, benchmark) for p, g in zip(predictions, golds)]
        return {'G_O': mean(scores)}
    elif benchmark == 'OneIG-Bench':
        dims = ['Alignment', 'Text', 'Reasoning', 'Style', 'Diversity']
        dim_scores = [llm_judge_score(p, g, dim) for dim in dims]
        return {'Overall': mean(dim_scores)}
    return {'score': -1}
```

## Common pitfalls

- G_O in GEdit-Bench is the mean score across all samples, not the arithmetic mean of G_SC and G_PQ.
- X2I evaluations must use ~1024x1024 resolution; T2I supports up to 2048px. Mismatched resolutions invalidate comparisons.
- Different benchmarks use different LLM judges (GPT-4o vs GPT-4.1); mixing them or assuming identical scoring rubrics leads to incorrect rankings.

## Evidence (verbatim from paper)

> For evaluation, two specific metrics are employed: Word Accuracy and Normalized Edit Distance (NED). As illustrated in the table, Emu3.5 outperforms state-of-the-art T2I models by a large margin, further demonstrating its strong capability in English text rendering.

## Citation

```bibtex
@misc{cui2025emu35,
  title={Emu3.5: Native Multimodal Models are World Learners},
  author={Yufeng Cui et al. (2025)},
  year={2025},
  note={arXiv:2510.26583}
}
```

- arXiv: 2510.26583