Imagen Coco Drawbench Eval

---
name: imagen-coco-drawbench-eval
description: Evaluates text-to-image generation models on photorealism, image-text alignment, and compositional reasoning using standard dataset metrics and human preference studies. Use when the user wants to benchmark on MS-COCO, DrawBench, or asks about evaluating this task. Reports FID-30K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.11487
  bibtex_key: saharia2022imagen
  confidence: high
---

# imagen-coco-drawbench-eval

> Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding — Saharia et al. (2022) (arXiv:2205.11487, 2022)

## What this evaluates

Evaluates text-to-image generation models on photorealism, image-text alignment, and compositional reasoning using standard dataset metrics and human preference studies.

## Datasets

- **MS-COCO** — total ?; splits: val (-1)
- **DrawBench** — total ?; splits: test (-1)

## Metrics

- `FID-30K` **(primary)** — range: other
  - Fréchet Inception Distance computed between the feature distributions of 30,000 generated images and 30,000 real images from the validation set.
- `Human preference rate` — range: percent
  - Percentage of pairwise comparisons where raters prefer the model-generated image over the reference image for photorealism or alignment.

## Input / output format

**Input**: Text prompts (captions for COCO, structured prompts for DrawBench)

**Output**: RGB images (256x256 for COCO evaluation, varying resolutions for DrawBench)

## Scoring recipe

```python
def compute_fid(gen_imgs, ref_imgs):
    return frechet_inception_distance(gen_imgs[:30000], ref_imgs[:30000])

def compute_human_pref(gen_imgs, ref_imgs, criterion):
    wins = sum(1 for g, r in zip(gen_imgs, ref_imgs) if rater_prefers(g, r, criterion))
    return (wins / len(gen_imgs)) * 100
```

## Common pitfalls

- FID is computed on a fixed subset of 30,000 samples, not the full validation set.
- Human evaluation filters out prompts containing people for the 'no people' subset, which significantly affects photorealism scores.
- Guidance weights differ between base (1.35) and super-resolution (8.0) models, affecting sample quality and alignment scores.

## Evidence (verbatim from paper)

> Table 1: MS-COCO 256x256 FID-30K. We use a guidance weight of 1.35 for our 64x64 model, and a guidance weight of 8.0 for our super-resolution model. ... Imagen achieves state of the art zero-shot FID on COCO at 7.27... Table 2 reports the human evaluation to test image quality and alignment on the COCO validation set. ... We report the percentage of time raters prefer Model A, Model B, or are indifferent for both image fidelity and image-text alignment.

## Citation

```bibtex
@misc{saharia2022imagen,
  title={Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding},
  author={Saharia et al. (2022)},
  year={2022},
  note={arXiv:2205.11487}
}
```

- arXiv: 2205.11487