Emu Edit Eval

---
name: emu-edit-eval
description: Evaluates image editing capability by measuring instruction following (text similarity) and source image preservation (image similarity). It tests the model's ability to modify images based on textual instructions while maintaining relevant visual elements. Use when the user wants to benchmark on EMU-Edit, or asks about evaluating this task. Reports CLIP-T.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.11340
  bibtex_key: xiao2024omnigen
  confidence: high
---

# emu-edit-eval

> OmniGen: Unified Image Generation — Shitao Xiao et al. (2024) (arXiv:2409.11340, 2024)

## What this evaluates

Evaluates image editing capability by measuring instruction following (text similarity) and source image preservation (image similarity). It tests the model's ability to modify images based on textual instructions while maintaining relevant visual elements.

## Datasets

- **EMU-Edit** — total ?; splits: test (-1)

## Metrics

- `CLIP-T` **(primary)** — range: [0, 1]
  - Cosine similarity between the CLIP text encoder embeddings of the editing instruction and the generated image.
- `CLIP-I` — range: [0, 1]
  - Cosine similarity between the CLIP image encoder embeddings of the source image and the generated image.

## Input / output format

**Input**: Source image, editing instruction (text), and optionally a target image or mask.

**Output**: Edited RGB image.

## Scoring recipe

```python
clip_model = load_clip()
clip_t = cosine_similarity(clip_model.encode_text(instruction), clip_model.encode_image(generated_img))
clip_i = cosine_similarity(clip_model.encode_image(source_img), clip_model.encode_image(generated_img))
```

## Common pitfalls

- CLIP similarity scores can be inflated by background preservation rather than actual editing quality.
- Different baselines may use different preprocessing or guidance scales, affecting direct comparison.

## Evidence (verbatim from paper)

> We evaluate the image editing on EMU-Edit[[58]] dataset and subject-driven generation capability on DreamBench[[56]]. We use CLIP-T to measure how well the model followed the instructions, while CLIP-I similarity scores measure the model’s ability to preserve elements from the source image.

## Citation

```bibtex
@misc{xiao2024omnigen,
  title={OmniGen: Unified Image Generation},
  author={Shitao Xiao et al. (2024)},
  year={2024},
  note={arXiv:2409.11340}
}
```

- arXiv: 2409.11340