Echo Eval

---
name: echo-eval
description: This benchmark probes an image generation model's ability to follow complex, real-world user prompts and produce high-quality outputs that preserve specific attributes like identity and color. It evaluates how well models handle non-standard, community-driven inputs and context-dependent instructions often found in social media discussions. Use when the user wants to benchmark on ECHO, or asks about evaluating this task. Reports quality_label.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.15021
  bibtex_key: ge2025echo
  confidence: medium
---

# echo-eval

> Constantly Improving Image Models Need Constantly Improving Benchmarks — Ge et al. (2025) (arXiv:2510.15021, 2025)

## What this evaluates

This benchmark probes an image generation model's ability to follow complex, real-world user prompts and produce high-quality outputs that preserve specific attributes like identity and color. It evaluates how well models handle non-standard, community-driven inputs and context-dependent instructions often found in social media discussions.

## Datasets

- **ECHO** — total 31000; splits: benchmark (-1), analysis (-1), trash (-1)

## Metrics

- `quality_label` **(primary)** — range: categorical
  - LLM-assigned categorical label ("Benchmark", "Analysis", or "Trash") based on prompt coherence, clear user intent, and data quality.
- `identity preservation` — range: percent
  - Assessed via VLM analysis to determine if the generated image maintains the subject's identity from the input or prompt context.
- `color shift` — range: percent
  - Measured by VLM to detect unintended color changes between the input/reference and the generated output.

## Input / output format

**Input**: Text prompt (optionally reconstructed from social media reply trees), optional input image(s), and optional community feedback/comments.

**Output**: A single generated image corresponding to the provided prompt and context.

## Scoring recipe

```python
def score_echo(predictions, gold):
    results = []
    for pred, g in zip(predictions, gold):
        label = g.get('quality_label') or llm_classify_quality(g['prompt'], pred)
        id_pres = vlm_check_identity(g.get('input_images'), pred)
        color_shift = vlm_check_color(g.get('input_images'), pred)
        results.append({'label': label, 'id_pres': id_pres, 'color_shift': color_shift})
    return {
        'benchmark_rate': sum(1 for r in results if r['label'] == 'Benchmark') / len(results),
        'avg_id_pres': mean([r['id_pres'] for r in results]),
        'avg_color_shift': mean([r['color_shift'] for r in results])
    }
```

## Common pitfalls

- Assuming input and output images are always explicitly labeled or appear in a fixed order; users often embed them in screenshots or collages without standard markers.
- Treating social media posts as isolated instances; prompts and context are frequently split across parent posts and reply trees, requiring recursive reconstruction.
- Relying solely on automated metrics without accounting for qualitative community feedback, which captures nuanced failures like identity drift or color shifts.

## Evidence (verbatim from paper)

> operationalizes qualitative community insights into measurable metrics like identity preservation and color shift—enabling adaptive, real-time evaluation of model capabilities.

## Citation

```bibtex
@misc{ge2025echo,
  title={Constantly Improving Image Models Need Constantly Improving Benchmarks},
  author={Ge et al. (2025)},
  year={2025},
  note={arXiv:2510.15021}
}
```

- arXiv: 2510.15021