Scene Graph Generation Eval

---
name: scene-graph-generation-eval
description: Evaluates scene graph generation models on predicting subject-predicate-object triplets while mitigating long-tailed training biases. It probes zero-shot generalization and graph-level semantic coherence through sentence-to-graph retrieval. Use when the user wants to benchmark on Visual Genome (VG), MS-COCO Caption (VG Overlap), or asks about evaluating this task. Reports mR@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2002.11949
  bibtex_key: [REDACTED]
  confidence: high
---

# scene-graph-generation-eval

> Unbiased Scene Graph Generation from Biased Training — Tang et al. (2020) (arXiv:2002.11949, 2020)

## What this evaluates

Evaluates scene graph generation models on predicting subject-predicate-object triplets while mitigating long-tailed training biases. It probes zero-shot generalization and graph-level semantic coherence through sentence-to-graph retrieval.

## Datasets

- **Visual Genome (VG)** — total 108000; splits: train (-1), val (5000), test (-1); repo https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch
- **MS-COCO Caption (VG Overlap)** — total 41859; splits: train (35859), test-1k (1000), test-5k (5000)

## Metrics

- `mR@K` **(primary)** — range: [0, 100] percent
  - Mean Recall@K. Computes Recall@K for each predicate category separately, then averages across all 50 predicate categories. mR@K = (1/|P|) * Σ_p Recall@K(p).
- `R@K / Med` — range: [0, 100] percent | rank
  - Recall@K for image retrieval and Median Ranking (Med) of retrieved results. R@K is the fraction of queries where the correct image appears in the top K retrieved results. Med is the median rank of the ground truth image in the ranked list.

## Input / output format

**Input**: Images with ground-truth bounding boxes and labels (PredCls), images with ground-truth boxes but no labels (SGCls), or raw images (SGDet). For S2GR, image captions paired with detected scene graphs.

**Output**: Predicted subject-predicate-object triplets per image, ranked by confidence score. For S2GR, a ranked list of retrieved images.

## Scoring recipe

```python
def compute_mR_at_K(predictions, ground_truth, K, predicates):
    recalls = []
    for p in predicates:
        gt_p = [g for g in ground_truth if g.predicate == p]
        pred_p = [pred for pred in predictions if pred.predicate == p]
        correct = sum(1 for g in gt_p if g.predicate in [pred.label for pred in pred_p[:K]])
        recalls.append(correct / len(gt_p) if gt_p else 0)
    return (sum(recalls) / len(predicates)) * 100
```

## Common pitfalls

- Standard Recall@K heavily favors head predicates and masks tail predicate failures; mR@K must be used to fairly evaluate long-tailed bias mitigation.
- Zero-shot evaluation strictly excludes any triplet observed during training, requiring careful dataset splitting and filtering.
- S2GR requires filtering low-confidence RoIs and removing background predicates to align heterogeneous image and text graphs before retrieval.

## Evidence (verbatim from paper)

> The conventional metric of RR is Recall@K (R@K), which was abandoned in this paper due to the reporting bias[[36]]. As illustrated in Figure[3](b), previous methods like [[72]] with good performance on R@K unfairly cater to “head” predicates, e.g., on, while neglect the “tail” ones, e.g., predicates like parked on, laying on have embarrassingly 0.0 Recall@100. To speak for the valuable “tail” rather than the trivial “head”, we adopted a recent replacement, mean Recall@K (mR@K), proposed by Tang et al.[[56]] and Chen et al.[[6]]. mR@K retrieves each predicate separately and then averages R@K for all predicates.

## Citation

```bibtex
@misc{tang2020unbiased,
  title={Unbiased Scene Graph Generation from Biased Training},
  author={Tang et al. (2020)},
  year={2020},
  note={arXiv:2002.11949}
}
```

- arXiv: 2002.11949