Echo 4o Eval

---
name: echo-4o-eval
description: Evaluates text-to-image generation models on instruction-following accuracy, surreal/fantasy creativity, and multi-reference composition. It probes the model's ability to align complex textual prompts with visual outputs, handle long-tail attributes, and integrate multiple reference images. Use when the user wants to benchmark on GenEval, DPG-Bench, GenEval++, Imagine-Bench, OmniContext, or asks about evaluating this task. Reports GenEval Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.09987
  bibtex_key: ye2025echo4o
  confidence: high
---

# echo-4o-eval

> Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation — Junyan Ye et al. (arXiv:2508.09987, 2025)

## What this evaluates

Evaluates text-to-image generation models on instruction-following accuracy, surreal/fantasy creativity, and multi-reference composition. It probes the model's ability to align complex textual prompts with visual outputs, handle long-tail attributes, and integrate multiple reference images.

## Datasets

- **GenEval** — total ?; splits: test (-1)
- **DPG-Bench** — total ?; splits: test (-1)
- **GenEval++** — total ?; splits: test (-1)
- **Imagine-Bench** — total ?; splits: test (-1)
- **OmniContext** — total ?; splits: test (-1)

## Metrics

- `GenEval Overall` **(primary)** — range: [0, 1]
  - Proportion of prompts where all specified objects, attributes, colors, and spatial relationships are correctly generated, measured via object detection and attribute matching.
- `DPG-Bench Overall` — range: [0, 100]
  - LLM-based scoring across five categories (Global, Entity, Attribute, Relation, Other), averaged to produce a score out of 100.
- `GenEval++ Overall` — range: [0, 1]
  - Extended version of GenEval with more complex, long-text instructions and additional objects/attributes; scored as the proportion of fully correct generations.
- `Imagine-Bench Overall` — range: [0, 10]
  - LLM-based evaluation of surreal and imaginative generation across four categories (Attribute shift, Spatiotemporal, Hybridization, Multi-Object), averaged on a 0-10 scale.
- `OmniContext Overall` — range: [0, 10]
  - LLM-based scoring of multi-reference image generation fidelity across MULTIPLE and SCENE settings, averaged on a 0-10 scale.

## Input / output format

**Input**: Text prompt describing the target image (and optionally reference images for OmniContext multi-reference tasks).

**Output**: A single RGB image generated by the model.

## Scoring recipe

```python
def evaluate(predictions, gold, benchmark):
    if benchmark == 'GenEval':
        correct = sum(1 for p, g in zip(predictions, gold) if check_attributes(p, g))
        return correct / len(gold)
    elif benchmark == 'DPG-Bench':
        scores = llm_score(predictions, gold, categories=['Global', 'Entity', 'Attribute', 'Relation', 'Other'])
        return mean(scores) * 100
    elif benchmark == 'GenEval++':
        return complex_attribute_match(predictions, gold)
    elif benchmark == 'Imagine-Bench':
        return llm_score_imagination(predictions, gold) / len(gold) * 10
    elif benchmark == 'OmniContext':
        return llm_score_multi_ref(predictions, gold, ref_images) / len(gold) * 10
    return 0
```

## Common pitfalls

- GenEval++ tasks differ from GenEval by adding only 1-2 objects or attributes, but this minor change substantially raises difficulty, causing many models to fail completely.
- OmniContext requires native multi-reference support; many models lack this capability entirely or are incompatible with multi-reference scenarios.
- Metrics rely on automated LLM or detection-based scoring, which may not perfectly correlate with human aesthetic judgments or subtle instruction adherence.

## Evidence (verbatim from paper)

> We evaluate the instruction-following capability of our model on two widely used benchmarks—GenEval and DPG-Bench, as well as our newly proposed benchmark, GenEval++. As shown in Table[1], Echo-4o achieves a score of 0.89 on GenEval, outperforming prior state-of-the-art unified models such as Bagel and OmniGen2. On DPG-Bench (Table[2]), Echo-4o attains an overall score of 86.07, outperforming strong competitors including SD3 and UniWorld.

## Citation

```bibtex
@misc{ye2025echo4o,
  title={Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation},
  author={Junyan Ye et al.},
  year={2025},
  note={arXiv:2508.09987}
}
```

- arXiv: 2508.09987