Ideabench Eval

---
name: ideabench-eval
description: Evaluates the professional design capabilities of generative models across text-to-image, image-to-image, and multi-image generation tasks. It probes aesthetic quality, contextual relevance, multimodal alignment, and adherence to complex, real-world design requirements that go beyond basic generation. Use when the user wants to benchmark on IDEA-Bench, or asks about evaluating this task. Reports Avg. Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.11767
  bibtex_key: liang2024ideabench
  confidence: high
---

# ideabench-eval

> IDEA-Bench: How Far are Generative Models from Professional Designing? — Chen Liang et al. (2024) (arXiv:2412.11767, 2024)

## What this evaluates

Evaluates the professional design capabilities of generative models across text-to-image, image-to-image, and multi-image generation tasks. It probes aesthetic quality, contextual relevance, multimodal alignment, and adherence to complex, real-world design requirements that go beyond basic generation.

## Datasets

- **IDEA-Bench** — total 275; splits: test (275); repo https://github.com/ali-vilab/IDEA-Bench

## Metrics

- `Avg. Score` **(primary)** — range: [0, 100]
  - The average score across all subtasks and categories. Each subtask score is computed by averaging hierarchical evaluation question scores (0-100 scale), and the final metric is the mean of these subtask scores.

## Input / output format

**Input**: Text prompts for generation tasks; input images plus text instructions for editing tasks; multiple input images and text for multi-image tasks.

**Output**: Generated image(s) corresponding to the provided prompt or instruction.

## Scoring recipe

```python
# Pseudo-code for IDEA-Bench scoring

total_score = 0
task_count = 0

for task in benchmark_tasks:
    subtask_scores = []
    for question in task.hierarchical_questions:
        # MLLM (e.g., GPT-4o) evaluates generated image against gold/reference
        score = mllm_evaluate(generated_image, question, gold_reference)
        subtask_scores.append(score)
    task_score = mean(subtask_scores)
    total_score += task_score
    task_count += 1

final_score = total_score / task_count
return final_score
```

## Common pitfalls

- Relying on traditional metrics like FID or CLIPScore, which fail to capture professional design nuances and contextual relevance.
- Assuming basic T2I models can handle multi-image or complex editing tasks without MLLM-assisted prompt rephrasing.
- Overlooking the hierarchical evaluation structure (1,650 questions across 275 cases), which requires granular assessment rather than single-pass generation.

## Evidence (verbatim from paper)

> | Method | Subtasks Score | | | | | | | | | | | Avg. Score |

Each task category is averaged across all its subtasks, with the top-ranked model scores for each subtask highlighted in bold.

## Citation

```bibtex
@misc{liang2024ideabench,
  title={IDEA-Bench: How Far are Generative Models from Professional Designing?},
  author={Chen Liang et al. (2024)},
  year={2024},
  note={arXiv:2412.11767}
}
```

- arXiv: 2412.11767