Genspace Alignment Eval

---
name: genspace-alignment-eval
description: Evaluates how well automated metrics and VLMs align with human judgments on spatially-aware image generation tasks across nine sub-domains. Use when the user wants to benchmark on GenSpace Human Alignment Test Set, or asks about evaluating this task. Reports agreement.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24870
  bibtex_key: wang2025genspace
  confidence: medium
---

# genspace-alignment-eval

> GenSpace: Benchmarking Spatially-Aware Image Generation — Wang et al. (2025) (arXiv:2505.24870, 2025)

## What this evaluates

Evaluates how well automated metrics and VLMs align with human judgments on spatially-aware image generation tasks across nine sub-domains.

## Datasets

- **GenSpace Human Alignment Test Set** — total 900; splits: test (900); repo https://github.com/SpatialVision/GenSpace

## Metrics

- `agreement` **(primary)** — range: percent
  - Proportion of samples where the predicted category (or mapped VLM score) matches the human-annotated category ('Correct', 'Partially Correct', 'Incorrect'). Calculated as (matches / total samples) × 100.

## Input / output format

**Input**: Generated images paired with their original text-to-image or image-editing prompts.

**Output**: Human annotations: 'Correct', 'Partially Correct', or 'Incorrect'. VLM baselines output continuous scores 0–100, mapped to categories for comparison.

## Scoring recipe

```python
def compute_agreement(predictions, human_labels):
    matches = sum(1 for p, h in zip(predictions, human_labels) if p == h)
    return (matches / len(human_labels)) * 100
```

## Common pitfalls

- VLMs struggle with allocentric perspective reasoning and quantitative spatial measurement, causing low alignment scores.
- This subsection only covers the human alignment validation subset (900 samples), not the full benchmark protocol.
- Mapping continuous VLM scores to discrete human categories can obscure fine-grained spatial errors.

## Evidence (verbatim from paper)

> Finally, we measure how well each method aligns with human perception by comparing its accuracy against manual human labels. To align the fine-grained continuous scores with this categorical system for comparison, we map scores as follows: 0 to "Incorrect," (0, 100) to "Partially Correct," and 100 to "Correct." Results. Tab.[3] presents the comparative results of different evaluators’ alignment with human judgment. Overall, our spatial expertise pipeline and corresponding metric demonstrate a stronger correlation with human recognition. Across sub-domains, our method achieves 76.22% average agreement with manual labels, while the most advanced VLM, Gemini-2.5-Pro, achieves only 56.44%.

## Citation

```bibtex
@misc{wang2025genspace,
  title={GenSpace: Benchmarking Spatially-Aware Image Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.24870}
}
```

- arXiv: 2505.24870