Livs T2i Alignment Eval

---
name: livs-t2i-alignment-eval
description: Evaluates how well text-to-image models align with pluralistic, intersectional community preferences for urban public space design. Probes whether multi-criteria preference optimization (DPO) improves alignment over a baseline, and how prompt origin and annotator demographics influence preference consistency and rating distributions. Use when the user wants to benchmark on LIVS, or asks about evaluating this task. Reports preference_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.01894
  bibtex_key: mushkani2025livs
  confidence: high
---

# livs-t2i-alignment-eval

> LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces — Mushkani et al. (2025) (arXiv:2503.01894, 2025)

## What this evaluates

Evaluates how well text-to-image models align with pluralistic, intersectional community preferences for urban public space design. Probes whether multi-criteria preference optimization (DPO) improves alignment over a baseline, and how prompt origin and annotator demographics influence preference consistency and rating distributions.

## Datasets

- **LIVS** — total 37710; splits: train (-1), test (2200)

## Metrics

- `preference_rate` **(primary)** — range: percent
  - Percentage of pairwise comparisons where annotators preferred the DPO-aligned model over the baseline, alongside the percentage preferring the baseline and the percentage rating both as neutral. Calculated per criterion and overall.

## Input / output format

**Input**: Text prompts describing urban public space designs (human-authored or AI-generated).

**Output**: Generated images from text-to-image models (baseline SDXL or DPO-finetuned SDXL).

## Scoring recipe

```python
# For each of the 2,200 held-out comparisons:
# 1. Generate images I_base and I_dpo from the same prompt.
# 2. Annotators rate I_base vs I_dpo on 3 criteria.
# 3. Record preference: 'base', 'dpo', or 'neutral' per criterion.
# 4. Collapse split ratings via majority vote if needed (for training, but eval reports per-criterion).
# 5. Compute:
#    dpo_rate = count(prefs == 'dpo') / total_comparisons
#    base_rate = count(prefs == 'base') / total_comparisons
#    neutral_rate = count(prefs == 'neutral') / total_comparisons
# Return rates per criterion and overall.
```

## Common pitfalls

- Collapsing multi-criteria split ratings into a single binary label via majority vote obscures which specific dimensions drive preference.
- High neutral ratings (up to 50%) indicate ambiguity in community values rather than model failure, requiring careful interpretation of alignment success.
- Prompt origin (human vs GPT-4o) significantly affects neutrality rates, confounding model evaluation if prompts are not controlled.

## Evidence (verbatim from paper)

> Out of 2,200 new comparisons, annotators chose the DPO-aligned model in 700 (32%) instances and the baseline in 300 (14%), marking the remaining 1,100 (50%) as neutral (See Appendix [I]). Criteria with more training annotations (e.g., Comfort, Invitingness) showed stronger improvements under DPO, whereas Inclusivity and Diversity had higher neutral ratings

## Citation

```bibtex
@misc{mushkani2025livs,
  title={LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces},
  author={Mushkani et al. (2025)},
  year={2025},
  note={arXiv:2503.01894}
}
```

- arXiv: 2503.01894