Image Reward Eval

---
name: image-reward-eval
description: Evaluates a model's ability to predict human preferences for text-to-image generation by ranking pairs of images generated from the same text prompt. It measures alignment with human judgment on coherence, fidelity, and aesthetic quality. Use when the user wants to benchmark on ImageReward Test Set, or asks about evaluating this task. Reports Preference Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.05977
  bibtex_key: xu2023imagereward
  confidence: high
---

# image-reward-eval

> ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — Xu et al. (2023) (arXiv:2304.05977, 2023)

## What this evaluates

Evaluates a model's ability to predict human preferences for text-to-image generation by ranking pairs of images generated from the same text prompt. It measures alignment with human judgment on coherence, fidelity, and aesthetic quality.

## Datasets

- **ImageReward Test Set** — total 6399; splits: test (6399); repo https://github.com/THUDM/ImageReward

## Metrics

- `Preference Accuracy` **(primary)** — range: percent
  - Percentage of image pairs where the model's predicted preferred image matches the human-annotated preferred image for a given prompt. Calculated as correct predictions divided by total pairs.
- `Recall@k / Filter@k` — range: percent
  - Recall@k: proportion of prompts where the human-preferred image is ranked in the top-k by the model. Filter@k: proportion of prompts where the human-preferred image is among the top-k selected images from a set of 8. All scores are averaged per prompt.

## Input / output format

**Input**: A text prompt and two (or k) images generated from that prompt.

**Output**: A scalar preference score for each image, or a ranked list of images.

## Scoring recipe

```python
def compute_preference_accuracy(predictions, gold):
    correct = 0
    for pred, gold_label in zip(predictions, gold):
        pred_label = 1 if pred[0] > pred[1] else 0
        if pred_label == gold_label:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- The test set is curated by selecting prompts annotated by annotators with high agreement to researchers, which may not represent the full distribution of human preferences.
- Baseline models (CLIP, Aesthetic, BLIP) use fundamentally different scoring mechanisms (cosine similarity vs. learned predictors), so direct score comparison requires careful calibration or thresholding.
- The 'Filter' metric is mentioned but not explicitly defined in the text, leaving ambiguity about whether it refers to top-k selection or filtering based on a score threshold.

## Evidence (verbatim from paper)

> Preference accuracy is the correctness of a scorer choosing the same one from two different images of one prompt with a human. As Table 3 shows, our model outperforms all the baselines. The preference accuracy of ImageReward reaches up to 65.14%, which is 15.14% more than 50% (random), about twice as much as 7.76% (that of BLIP score).

## Citation

```bibtex
@misc{xu2023imagereward,
  title={ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2304.05977}
}
```

- arXiv: 2304.05977