Omnigen2 Eval

---
name: omnigen2-eval
description: Evaluates a unified multimodal model's capabilities across visual understanding, text-to-image generation, instruction-based image editing, and in-context generation. It probes compositional prompt following, long-prompt adherence, edit accuracy versus preservation, and subject consistency across single, multiple, and scene contexts. Use when the user wants to benchmark on MMBench, MMMU, MM-Vet, GenEval, DPG-Bench, Emu-Edit, GEdit-Bench-EN, ImgEdit-Bench, OmniContext, or asks about evaluating this task. Reports GenEval Overall, DPG-Bench Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.18871
  bibtex_key: wu2025omnigen2
  confidence: high
---

# omnigen2-eval

> OmniGen2: Exploration to Advanced Multimodal Generation — Chenyuan Wu et al. (2025) (arXiv:2506.18871, 2025)

## What this evaluates

Evaluates a unified multimodal model's capabilities across visual understanding, text-to-image generation, instruction-based image editing, and in-context generation. It probes compositional prompt following, long-prompt adherence, edit accuracy versus preservation, and subject consistency across single, multiple, and scene contexts.

## Datasets

- **MMBench** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)
- **GenEval** — total ?; splits: test (-1)
- **DPG-Bench** — total ?; splits: test (-1)
- **Emu-Edit** — total ?; splits: test (-1)
- **GEdit-Bench-EN** — total ?; splits: test (-1)
- **ImgEdit-Bench** — total ?; splits: test (-1)
- **OmniContext** — total ?; splits: test (-1)

## Metrics

- `GenEval Overall` **(primary)** — range: [0, 1]
  - Average score across six compositional categories: single object, two objects, counting, colors, position, and color attribution. Scores range from 0 to 1 per category.
- `DPG-Bench Overall` **(primary)** — range: [0, 100]
  - Average score across five dimensions: global, entity, attribute, relation, and other. Measures long prompt following capability.
- `Emu-Edit CLIP-Out` — range: [0, 1]
  - CLIP feature similarity between the generated image and the target caption. Higher values indicate better alignment with the edit instruction.
- `GEdit-Bench-EN SC` — range: [0, 10]
  - Semantic Consistency score evaluating instruction following. Assessed via human or automated scoring on a scale up to 10.
- `ImgEdit-Bench Overall` — range: [0, 10]
  - Average score across nine editing task types (Add, Adjust, Extract, Replace, Remove, Background, Style, Hybrid, Action).
- `OmniContext Overall` — range: [0, 10]
  - Average of Prompt Following (PF) and Subject Consistency (SC) scores across task splits (Single, Multiple, Scene). Higher is better.

## Input / output format

**Input**: Text prompts, reference images (for editing and in-context generation), and optional editing instructions or context images.

**Output**: Generated images corresponding to the input prompts and instructions.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    scores = {}
    # GenEval: average category scores
    scores['GenEval_Overall'] = np.mean([cat_score for cat_score in predictions['gen_eval_categories']])
    # DPG-Bench: average dimension scores
    scores['DPG_Overall'] = np.mean([dim_score for dim_score in predictions['dpg_dimensions']])
    # Emu-Edit: CLIP/DINO similarity
    scores['EmuCLIP_Out'] = clip_similarity(predictions['image'], gold['target_caption'])
    scores['EmuCLIP_I'] = clip_similarity(predictions['image'], gold['source_image'])
    scores['EmuDINO'] = dino_similarity(predictions['image'], gold['source_image'])
    # GEdit/ImgEdit/OmniContext: average task/subtask scores
    scores['GEdit_SC'] = predictions['gedit_sc']
    scores['ImgEdit_Overall'] = np.mean(predictions['imgedit_tasks'])
    scores['OmniContext_Overall'] = np.mean(predictions['omni_context_pf_sc'])
    return scores
```

## Common pitfalls

- Different benchmarks use different scoring scales (0-1, 0-100, 0-10), which can cause confusion when comparing overall model performance across tasks.
- GenEval and DPG-Bench report both sub-category scores and an overall average; readers must specify which is being reported to avoid misinterpretation.
- The OmniContext benchmark is newly proposed and uses task-specific splits (Single, Multiple, Scene) that are not standardized in external repositories, making direct reproduction difficult without the paper's code.
- Image editing metrics like CLIP-I and DINO measure preservation of unedited regions, while CLIP-Out measures alignment with the edit instruction; optimizing one often trades off against the other.

## Evidence (verbatim from paper)

> We assess OmniGen2’s T2I generation capabilities on two standard benchmarks: GenEval, which evaluates compositional understanding, and DPG-Bench, which measures long prompt following. Our model demonstrates highly competitive performance, particularly when considering its resource efficiency. ... On Emu-Edit, our model achieves the highest CLIP-Out score (0.309), indicating it most effectively applies the requested edits among all compared models. Concurrently, it secures the second-best scores for CLIP-I (0.876) and DINO (0.822), which measure the preservation of unedited regions.

## Citation

```bibtex
@misc{wu2025omnigen2,
  title={OmniGen2: Exploration to Advanced Multimodal Generation},
  author={Chenyuan Wu et al. (2025)},
  year={2025},
  note={arXiv:2506.18871}
}
```

- arXiv: 2506.18871