Conceptmix Eval

---
name: conceptmix-eval
description: Evaluates the compositional generalization capability of text-to-image models by testing their ability to generate images that satisfy multiple, simultaneously specified visual concepts (objects, colors, shapes, spatial relationships, etc.) within a single prompt. The benchmark probes model robustness to increasing compositional complexity (k) and reveals limitations in handling less frequent concept combinations. Use when the user wants to benchmark on ConceptMix, or asks about evaluating this task. Reports Full-mark score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14339
  bibtex_key: wu2024conceptmix
  confidence: high
---

# conceptmix-eval

> ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty — Wu et al. (2024) (arXiv:2408.14339, 2024)

## What this evaluates

Evaluates the compositional generalization capability of text-to-image models by testing their ability to generate images that satisfy multiple, simultaneously specified visual concepts (objects, colors, shapes, spatial relationships, etc.) within a single prompt. The benchmark probes model robustness to increasing compositional complexity (k) and reveals limitations in handling less frequent concept combinations.

## Datasets

- **ConceptMix** — total 2100; splits: test (2100)

## Metrics

- `Full-mark score` **(primary)** — range: [0, 1]
  - The proportion of generated images that correctly satisfy all k+1 required visual concepts specified in the prompt. Calculated as (number of fully correct images) / (total number of prompts).
- `Concept fraction score` — range: [0, 1]
  - The average proportion of visual concepts satisfied across all generated images for a given prompt set. Calculated as (sum of satisfied concepts per image) / (total concepts across all images).

## Input / output format

**Input**: Text prompt containing at least one object and k additional visual concept categories (e.g., color, shape, spatial relationship, style, texture, size, number).

**Output**: Generated image corresponding to the input prompt.

## Scoring recipe

```python
def compute_full_mark_score(prompts, images, k_values):
    correct_count = 0
    for prompt, image, k in zip(prompts, images, k_values):
        # GPT-4o grades the image against the k+1 required concepts
        points_awarded = gpt4o_grade(image, prompt)  # returns int in [0, k+1]
        if points_awarded == k + 1:
            correct_count += 1
    return correct_count / len(prompts)

def compute_concept_fraction_score(prompts, images, k_values):
    total_satisfied = 0
    total_concepts = 0
    for prompt, image, k in zip(prompts, images, k_values):
        points_awarded = gpt4o_grade(image, prompt)
        total_satisfied += points_awarded
        total_concepts += (k + 1)
    return total_satisfied / total_concepts
```

## Common pitfalls

- Pixel-level similarity metrics (e.g., CLIP score, FID) are inadequate for this benchmark because they fail to accurately capture spatial, shape, and size concepts, leading to misleading performance estimates.
- Evaluating models only at low complexity (k=1) masks the sharp performance degradation that occurs as compositional difficulty increases, particularly for open-source models.
- Model performance is heavily biased by concept frequency in training data (e.g., LAION-5B), causing uneven capabilities across concept categories that are not inherent to the model's architecture.

## Evidence (verbatim from paper)

> We consider two grading metrics: 1) Full-mark score, which measures the proportion of generated images where the image correctly satisfies *all* $k+1$ required concepts, and 2) Concept fraction score, which measures the average proportion of visual concepts satisfied by the generated images.

## Citation

```bibtex
@misc{wu2024conceptmix,
  title={ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2408.14339}
}
```

- arXiv: 2408.14339