Evalmuse 40k Eval

---
name: evalmuse-40k-eval
description: Evaluates the fine-grained alignment and structural fidelity between generated images and their corresponding text prompts. It probes a model's ability to match specific visual elements (e.g., objects, colors, counts) and overall composition against human-annotated ground truth. Use when the user wants to benchmark on EvalMuse-40K, or asks about evaluating this task. Reports SRCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.18150
  bibtex_key: han2024evalmuse40k
  confidence: high
---

# evalmuse-40k-eval

> EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation — Han et al. (2024) (arXiv:2412.18150, 2024)

## What this evaluates

Evaluates the fine-grained alignment and structural fidelity between generated images and their corresponding text prompts. It probes a model's ability to match specific visual elements (e.g., objects, colors, counts) and overall composition against human-annotated ground truth.

## Datasets

- **EvalMuse-40K** — total 40000; splits: train (30000), test (1000)

## Metrics

- `SRCC` **(primary)** — range: [-1, 1]
  - Spearman Rank Correlation Coefficient measuring the monotonic relationship between predicted alignment scores and human annotations.
- `Accuracy` — range: percent
  - Percentage of correctly predicted element-wise matches (or mismatches) in fine-grained evaluation.

## Input / output format

**Input**: Image-text pair, optionally accompanied by element-level annotations or structural labels for fine-grained evaluation.

**Output**: Continuous alignment score (0–1) for overall evaluation, or element-wise match probability/label for fine-grained evaluation.

## Scoring recipe

```python
# Overall alignment (SRCC)
preds = [model.predict(img, txt) for img, txt in test_set]
gold = [human_score for _, human_score in test_set]
srcc = spearmanr(preds, gold).correlation

# Fine-grained evaluation (Accuracy)
preds_elem = [model.predict_element(img, elem_prompt) for img, elem_prompt in test_set]
gold_elem = [elem_label for _, elem_label in test_set]
acc = sum(p == g for p, g in zip(preds_elem, gold_elem)) / len(gold_elem)
```

## Common pitfalls

- Confusing overall alignment scoring with fine-grained element-wise scoring; the benchmark requires separate evaluation pipelines and metrics for each.
- Failing to apply the fixed-step (0.01) threshold search for binary classification methods, which is explicitly required to maximize overall accuracy.
- Using prompts that overlap between training and test splits, violating the no-overlap constraint and artificially inflating correlation metrics.

## Evidence (verbatim from paper)

> For overall alignment scores, we compare FGA-BLIP2 with the state-of-the-art models and report the Spearman Rank Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC) to measure the correlation between model predictions and human annotations. For fine-grained evaluation, we compare PN-VQA with TIFA and VQ2, and use several advanced MLLMs for the VQA task. On one hand, we average the fine-grained scores of the image-text pairs and compare them with the overall alignment scores from human annotations. On the other hand, we conduct fine-grained evaluation by reporting the accuracy of the method’s element-wise predictions.

## Citation

```bibtex
@misc{han2024evalmuse40k,
  title={EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2412.18150}
}
```

- arXiv: 2412.18150