R Precision Sgg Eval

---
name: r-precision-sgg-eval
description: Evaluates the semantic alignment and retrieval capability of scene graph generation models and image-scene graph similarity frameworks. It measures how well generated or ground-truth scene graphs can retrieve corresponding images compared to other images in a dataset. Use when the user wants to benchmark on Visual Genome, Open Images V6, or asks about evaluating this task. Reports R-Precision@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.00590
  bibtex_key: cong2023span
  confidence: high
---

# r-precision-sgg-eval

> SPAN: Learning Similarity between Scene Graphs and Images with Transformers — Cong et al. (2023) (arXiv:2304.00590, 2023)

## What this evaluates

Evaluates the semantic alignment and retrieval capability of scene graph generation models and image-scene graph similarity frameworks. It measures how well generated or ground-truth scene graphs can retrieve corresponding images compared to other images in a dataset.

## Datasets

- **Visual Genome** — total 108000; splits: train (75600), test (32400)
- **Open Images V6** — total 131300; splits: train (126000), test (5300)

## Metrics

- `R-Precision@K` **(primary)** — range: [0, 1]
  - Precision at rank K for image retrieval using scene graph similarity scores. Calculated as the number of ground-truth matching images in the top K retrieved results divided by K.

## Input / output format

**Input**: A query scene graph (serialized with structural encodings) and a candidate set of images.

**Output**: Similarity scores between the query scene graph and each candidate image, used to produce a ranked retrieval list.

## Scoring recipe

```python
def compute_r_precision(sim_scores, gt_idx, k):
    top_k_indices = np.argsort(sim_scores)[::-1][:k]
    hit = 1 if gt_idx in top_k_indices else 0
    return hit / k
```

## Common pitfalls

- R-Precision measures retrieval performance based on semantic similarity, not triplet-level IoU or bounding box overlap.
- The metric is sensitive to the number of retrieved items (K=10/50/100); scores are not directly comparable to traditional Recall@K or mR@K metrics.
- Location-free vs location-bound scene graphs yield different scores; bounding box constraints significantly impact retrieval accuracy.

## Evidence (verbatim from paper)

> Considering that the triplet-oriented metrics cannot directly reflect the consistency between generated scene graphs and images, we re-evaluate the prior scene graph generation models with the graph-oriented metric R-Precision (K=10/50/100) based on the similarity computed by GICON.

## Citation

```bibtex
@misc{cong2023span,
  title={SPAN: Learning Similarity between Scene Graphs and Images with Transformers},
  author={Cong et al. (2023)},
  year={2023},
  note={arXiv:2304.00590}
}
```

- arXiv: 2304.00590