Rovi Instance Grounding Eval

---
name: rovi-instance-grounding-eval
description: Evaluates the ability of text-to-image models to accurately render specific objects at requested bounding box locations while maintaining prompt fidelity, attribute correctness, and overall aesthetic quality. Use when the user wants to benchmark on ROVI validation set, or asks about evaluating this task. Reports Gen Inst..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01008
  bibtex_key: peng2025rovi
  confidence: high
---

# rovi-instance-grounding-eval

> ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation — Peng et al. (2025) (arXiv:2508.01008, 2025)

## What this evaluates

Evaluates the ability of text-to-image models to accurately render specific objects at requested bounding box locations while maintaining prompt fidelity, attribute correctness, and overall aesthetic quality.

## Datasets

- **ROVI validation set** — total ?; splits: val (-1); repo https://github.com/CihangPeng/ROVI

## Metrics

- `Gen Inst.` **(primary)** — range: [0, 1]
  - Per-box cross-checking pass rate using VLM-based evaluation. Each expected instance is cropped from its specified bounding box on the generated image, and a VLM (Qwen2VL) computes the likelihood that the cropped window matches its assigned label. The metric is the fraction of boxes that pass this check.
- `FID` — range: [0, ∞)
  - Fréchet Inception Distance measuring the distance between feature distributions of real and generated images. Lower values indicate higher quality.
- `Aes.` — range: [0, 10]
  - Aesthetic score predicting human preference for image quality. Higher values indicate better aesthetics.
- `CLIP Sim.` — range: [0, 1]
  - CLIP-based image-text similarity measuring alignment between the generated image and the input prompt. Higher values indicate better prompt fidelity.

## Input / output format

**Input**: Text prompt with associated bounding box coordinates and instance labels (box-label pairs).

**Output**: Generated image.

## Scoring recipe

```python
def score_gen_inst(predictions, gold):
    passes = 0
    total = 0
    for img, (_, box_labels) in zip(predictions, gold):
        for box, label in box_labels:
            crop = img.crop(box)
            if vlm_check(crop, label):  # Qwen2VL returns match
                passes += 1
            total += 1
    return passes / total if total > 0 else 0.0
```

## Common pitfalls

- Using standard detection datasets like Open Images for evaluation yields artificially low scores due to limited category vocabularies (e.g., 600 categories), inconsistent captions, and poor aesthetic quality for generation tasks.
- Image-level metrics like FID and CLIP similarity often disadvantage fine-tuned models; improvements in grounding and prompt quality are required to offset this baseline penalty.

## Evidence (verbatim from paper)

> *Gen Inst.*: per-box cross-checking pass rate using VLM-based evaluation. *Our G.*: GLIGEN trained on ROVI. *SD v1.4*: Stable Diffusion baseline*[[Rombach et al.(2022a)Rombach, Blattmann, Lorenz, Esser, and Ommer]]*.

## Citation

```bibtex
@misc{peng2025rovi,
  title={ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2508.01008}
}
```

- arXiv: 2508.01008