Image Synthesis Eval

---
name: image-synthesis-eval
description: Evaluates the visual fidelity and text-image alignment of generated images. It measures realism and distribution matching using FID, and semantic alignment using CLIP scores. Use when the user wants to benchmark on COCO-2014, or asks about evaluating this task. Reports FID (CLIP features).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.03206
  bibtex_key: esser2024scalingrectifiedflow
  confidence: high
---

# image-synthesis-eval

> Scaling Rectified Flow Transformers for High-Resolution Image Synthesis — Esser et al. (2024) (arXiv:2403.03206, 2024)

## What this evaluates

Evaluates the visual fidelity and text-image alignment of generated images. It measures realism and distribution matching using FID, and semantic alignment using CLIP scores.

## Datasets

- **COCO-2014** — total ?; splits: val (-1)

## Metrics

- `FID (CLIP features)` **(primary)** — range: [0, ∞) (lower is better)
  - Fréchet Inception Distance computed on CLIP image features to measure the distance between real and generated image distributions. Lower values indicate better quality.
- `CLIP score` — range: [0, 1] (higher is better)
  - Cosine similarity between CLIP image and text embeddings. Higher values indicate better text-image alignment.

## Input / output format

**Input**: Text prompt (for generation) and real images (for FID calculation).

**Output**: Generated RGB image.

## Scoring recipe

```python
def compute_metrics(gen_images, real_images, prompts):
    clip_gen = clip_model.encode(gen_images)
    clip_real = clip_model.encode(real_images)
    fid = frechet_distance(clip_gen, clip_real)
    clip_scores = cosine_similarity(clip_gen, clip_model.encode(prompts))
    return fid, clip_scores.mean()
```

## Common pitfalls

- FID is calculated on CLIP features, not the standard Inception v3 features.
- Lower FID is better, while higher CLIP score is better.
- Evaluation is strictly performed on the COCO-2014 validation split, not the training data.

## Evidence (verbatim from paper)

> We calculate the FID on CLIP features as proposed by (Sauer et al., 2021). All metrics are evaluated on the COCO-2014 validation split (Lin et al., 2014).

## Citation

```bibtex
@misc{esser2024scalingrectifiedflow,
  title={Scaling Rectified Flow Transformers for High-Resolution Image Synthesis},
  author={Esser et al. (2024)},
  year={2024},
  note={arXiv:2403.03206}
}
```

- arXiv: 2403.03206