Dreambench Subject Control Eval

---
name: dreambench-subject-control-eval
description: Evaluates a diffusion model's ability to generate images that preserve subject identity from reference images while adhering to text prompts, covering both single-subject and multi-subject scenarios. Use when the user wants to benchmark on DreamBench, or asks about evaluating this task. Reports DINO.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.02160
  bibtex_key: wu2025lesstomore
  confidence: high
---

# dreambench-subject-control-eval

> Less-to-More Generalization: Unlocking More Controllability by In-Context Generation — Shaojin Wu et al. (2025) (arXiv:2504.02160, 2025)

## What this evaluates

Evaluates a diffusion model's ability to generate images that preserve subject identity from reference images while adhering to text prompts, covering both single-subject and multi-subject scenarios.

## Datasets

- **DreamBench** — total ?; splits: test (-1)

## Metrics

- `DINO` **(primary)** — range: [0, 1]
  - Cosine similarity between DINO embeddings of the generated image and the reference image. Higher values indicate better subject preservation.
- `CLIP-I` — range: [0, 1]
  - Cosine similarity between CLIP image embeddings of the generated image and the reference image. Used as a secondary measure of subject similarity.
- `CLIP-T` — range: [0, 1]
  - Cosine similarity between CLIP text embeddings of the prompt and CLIP image embeddings of the generated image. Measures text fidelity/adherence.

## Input / output format

**Input**: One or more reference images (subject(s)) and a text prompt describing the desired scene.

**Output**: A single generated image per prompt.

## Scoring recipe

```python
def score(generated_img, ref_img, prompt):
    gen_emb = get_embeddings(generated_img)
    ref_emb = get_embeddings(ref_img)
    txt_emb = get_text_embeddings(prompt)
    dino = cosine_sim(gen_emb['dino'], ref_emb['dino'])
    clip_i = cosine_sim(gen_emb['clip_img'], ref_emb['clip_img'])
    clip_t = cosine_sim(gen_emb['clip_img'], txt_emb['clip_txt'])
    return {'DINO': dino, 'CLIP-I': clip_i, 'CLIP-T': clip_t}
```

## Common pitfalls

- CLIP-I measures image-to-image similarity (subject fidelity), while CLIP-T measures text-to-image similarity (prompt adherence); confusing them leads to incorrect capability assessment.
- DreamBench requires generating 6 images per prompt and averaging scores; reporting single samples or different counts breaks comparability with the benchmark.
- Multi-subject evaluation uses 30 specific subject combinations from DreamBench; using arbitrary subjects invalidates the benchmark protocol.

## Evidence (verbatim from paper)

> Following previous works, we use standard automatic metrics to evaluate both subject similarity and text fidelity. Specifically, we employ cosine similarity measures between generated images and reference images within CLIP*[[27]]* and DINO*[[22]]* spaces, referred to as CLIP-I and DINO scores, respectively, to assess subject similarity. Additionally, we calculate the cosine similarity between the prompt and the image CLIP embeddings (CLIP-T) to evaluate text fidelity.

## Citation

```bibtex
@misc{wu2025lesstomore,
  title={Less-to-More Generalization: Unlocking More Controllability by In-Context Generation},
  author={Shaojin Wu et al. (2025)},
  year={2025},
  note={arXiv:2504.02160}
}
```

- arXiv: 2504.02160