Tifa 100 Eval

---
name: tifa-100-eval
description: Evaluates how well an automatic multimodal transformer predicts fine-grained human feedback (quality scores, region-level misalignment heatmaps, and text misalignment annotations) on text-to-image generation tasks. Use when the user wants to benchmark on TIFA, or asks about evaluating this task. Reports correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.10240
  bibtex_key: liang2023richhumanfeedback
  confidence: medium
---

# tifa-100-eval

> Rich Human Feedback for Text-to-Image Generation — Liang et al. (2023) (arXiv:2312.10240, 2023)

## What this evaluates

Evaluates how well an automatic multimodal transformer predicts fine-grained human feedback (quality scores, region-level misalignment heatmaps, and text misalignment annotations) on text-to-image generation tasks.

## Datasets

- **TIFA** — total 100; splits: test (100)

## Metrics

- `correlation` **(primary)** — range: [-1, 1]
  - Measures the statistical correlation between the model's predicted scores, heatmaps, and misalignment annotations and the corresponding human-annotated ground truth labels.

## Input / output format

**Input**: A text prompt and its corresponding generated image.

**Output**: A scalar quality score, a region-level implausibility/misalignment heatmap, and a list of misaligned prompt keywords.

## Scoring recipe

```python
def compute_correlation(predictions, human_labels):
    # predictions and human_labels are arrays of floats
    # Compute Pearson correlation coefficient
    corr = np.corrcoef(predictions, human_labels)[0, 1]
    return corr
```

## Common pitfalls

- Evaluation is strictly limited to 100 prompts due to the high cost of human annotation.
- Relies entirely on human feedback as ground truth, which may introduce annotator bias or variability.

## Evidence (verbatim from paper)

> For the 100 prompts for our human evaluation, they are sampled from the existing benchmark: TIFA [24]. We only did our evaluation on 100 prompts due to the high cost of the human annotation. The model predicts these annotations with high correlation to human labels.

## Citation

```bibtex
@misc{liang2023richhumanfeedback,
  title={Rich Human Feedback for Text-to-Image Generation},
  author={Liang et al. (2023)},
  year={2023},
  note={arXiv:2312.10240}
}
```

- arXiv: 2312.10240