Dreambench Eval

---
name: dreambench-eval
description: Evaluates subject-driven image generation by measuring how well the model follows text instructions and preserves the reference subject from the source image. It tests the model's ability to extract and reuse specific objects without fine-tuning. Use when the user wants to benchmark on DreamBench, or asks about evaluating this task. Reports CLIP-T.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.11340
  bibtex_key: xiao2024omnigen
  confidence: high
---

# dreambench-eval

> OmniGen: Unified Image Generation — Shitao Xiao et al. (2024) (arXiv:2409.11340, 2024)

## What this evaluates

Evaluates subject-driven image generation by measuring how well the model follows text instructions and preserves the reference subject from the source image. It tests the model's ability to extract and reuse specific objects without fine-tuning.

## Datasets

- **DreamBench** — total ?; splits: test (-1)

## Metrics

- `CLIP-T` **(primary)** — range: [0, 1]
  - Cosine similarity between CLIP text embeddings of the instruction and the generated image.
- `CLIP-I` — range: [0, 1]
  - Cosine similarity between CLIP image embeddings of the reference subject image and the generated image.

## Input / output format

**Input**: Reference image containing a subject, text instruction specifying the desired generation.

**Output**: Generated RGB image featuring the subject.

## Scoring recipe

```python
clip_model = load_clip()
clip_t = cosine_similarity(clip_model.encode_text(instruction), clip_model.encode_image(generated_img))
clip_i = cosine_similarity(clip_model.encode_image(reference_img), clip_model.encode_image(generated_img))
```

## Common pitfalls

- CLIP-I may not accurately capture fine-grained subject identity preservation compared to face/object recognition models.
- The paper notes using a single reference image per object instead of fine-tuning, which differs from standard DreamBooth evaluation.

## Evidence (verbatim from paper)

> We evaluate the image editing on EMU-Edit[[58]] dataset and subject-driven generation capability on DreamBench[[56]]. We use CLIP-T to measure how well the model followed the instructions, while CLIP-I similarity scores measure the model’s ability to preserve elements from the source image.

## Citation

```bibtex
@misc{xiao2024omnigen,
  title={OmniGen: Unified Image Generation},
  author={Shitao Xiao et al. (2024)},
  year={2024},
  note={arXiv:2409.11340}
}
```

- arXiv: 2409.11340