Pamela Eval

---
name: pamela-eval
description: Probes a model's ability to predict individual user aesthetic preferences for AI-generated images based on prompt, image, and user demographics. It specifically tests both interpolation for known users and zero-shot few-shot generalization to novel users. Use when the user wants to benchmark on PAM∃LA, or asks about evaluating this task. Reports SROCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.07427
  bibtex_key: [REDACTED]
  confidence: high
---

# pamela-eval

> Personalizing Text-to-Image Generation to Individual Taste — Maerten et al. (2026) (arXiv:2604.07427, 2026)

## What this evaluates

Probes a model's ability to predict individual user aesthetic preferences for AI-generated images based on prompt, image, and user demographics. It specifically tests both interpolation for known users and zero-shot few-shot generalization to novel users.

## Datasets

- **PAM∃LA** — total 70000; splits: train (50222), val_seen (6551), test_seen (9735), val_unseen (926), test_unseen (2470); repo https://github.com/PAMELA-bench/PAMELA_Predictor

## Metrics

- `SROCC` **(primary)** — range: [-1, 1]
  - Spearman Rank Correlation Coefficient; measures the rank-order correlation between predicted and actual preference ratings.
- `PLCC` — range: [-1, 1]
  - Pearson Linear Correlation Coefficient; measures the linear correlation between predicted and actual preference ratings.
- `pairwise accuracy` — range: [0, 1]
  - The proportion of correctly ordered image pairs based on predicted versus actual ratings.

## Input / output format

**Input**: A text prompt, a generated image, and user demographic metadata (age, gender, education, art experience) or a few-shot context of k image-rating pairs for unseen users.

**Output**: A continuous scalar aesthetic preference rating score.

## Scoring recipe

```python
import scipy.stats as stats
def score(preds, golds):
    srocc = stats.spearmanr(preds, golds).correlation
    plcc = stats.pearsonr(preds, golds).correlation
    correct, total = 0, 0
    for i in range(len(preds)):
        for j in range(i+1, len(preds)):
            if golds[i] != golds[j]:
                total += 1
                if (preds[i] > preds[j]) == (golds[i] > golds[j]):
                    correct += 1
    return {'SROCC': srocc, 'PLCC': plcc, 'pairwise_accuracy': correct / total if total else 0}
```

## Common pitfalls

- Confusing 'seen users' (interpolation) with 'unseen users' (zero-shot few-shot generalization), which require fundamentally different evaluation protocols.
- Averaging ratings across users before evaluation, which defeats the purpose of personalized preference prediction and masks individual taste variations.
- Relying solely on population-level metrics (AvgSROCC) instead of user-level metrics (UserSROCC), which fails to capture the model's ability to capture idiosyncratic preferences.

## Evidence (verbatim from paper)

> Our model outperforms all baselines across both evaluation regimes (user level vs population average) and all three metrics (SROCC, PLCC, pairwise accuracy).

## Citation

```bibtex
@misc{maerten2026pamela,
  title={Personalizing Text-to-Image Generation to Individual Taste},
  author={Maerten et al. (2026)},
  year={2026},
  note={arXiv:2604.07427}
}
```

- arXiv: 2604.07427