Panoptic Scene Graph Generation Eval

---
name: panoptic-scene-graph-generation-eval
description: Evaluates a model's ability to jointly perform panoptic segmentation and scene graph generation by predicting object-background masks and relational triplets from a single image. It probes comprehensive scene understanding, accurate object grounding, and context-aware relation prediction without relying on separate detection heads. Use when the user wants to benchmark on PSG dataset, or asks about evaluating this task. Reports Mean Recall (mR)@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.11247
  bibtex_key: yang2022panoptic
  confidence: high
---

# panoptic-scene-graph-generation-eval

> Panoptic Scene Graph Generation — Yang et al. (2022) (arXiv:2207.11247, 2022)

## What this evaluates

Evaluates a model's ability to jointly perform panoptic segmentation and scene graph generation by predicting object-background masks and relational triplets from a single image. It probes comprehensive scene understanding, accurate object grounding, and context-aware relation prediction without relying on separate detection heads.

## Datasets

- **PSG dataset** — total 49000; splits: train (-1), val (-1), test (-1); repo https://github.com/Jingkang50/OpenPSG

## Metrics

- `Recall (R)@K` — range: percent
  - Percentage of ground-truth triplets correctly matched by the top-K predicted triplets. K is typically 20, 50, or 100.
- `Mean Recall (mR)@K` **(primary)** — range: percent
  - Mean of recall scores across all predicate classes for the top-K predictions. Used to measure unbiased relation prediction performance, as emphasized in the paper for evaluating one-stage models.
- `Panoptic Quality (PQ)` — range: [0, 1]
  - Standard panoptic segmentation metric combining segmentation quality and recognition quality, reported for visualization and segmentation evaluation.

## Input / output format

**Input**: RGB image with associated panoptic segmentation masks (instance masks + stuff masks) and ground-truth relational triplets.

**Output**: Predicted panoptic segmentation masks and a ranked list of relational triplets (subject, predicate, object) with confidence scores.

## Scoring recipe

```python
def compute_recall(preds, gold, k=20):
    top_k = preds[:k]
    matched = sum(1 for g in gold if g in top_k)
    return (matched / len(gold)) * 100 if gold else 0

def compute_mr(preds, gold, k=20):
    recalls = []
    for pred_class in unique_predicates(gold):
        recalls.append(compute_recall(
            [t for t in preds if t.predicate == pred_class],
            [t for t in gold if t.predicate == pred_class], k))
    return sum(recalls) / len(recalls) if recalls else 0
```

## Common pitfalls

- Predicate classification (PredCls) metrics use ground-truth segmentation, artificially inflating performance compared to detection-based (SGDet) settings.
- One-stage models lack a separate predicate classification head, so predicate metrics are inapplicable and marked as '-', only SGG metrics are reported.
- Training schedule heavily impacts one-stage models; PSGTR requires 60 epochs to reach SOTA, while two-stage models are evaluated at 12 epochs by default.

## Evidence (verbatim from paper)

> Table 2 reports the scene graph generation performance of all the methods mentioned in Sec. 4.1, Sec. 4.2, and Sec. 4.3 under the PSG dataset. Table 2: Comparison between all baselines and PSGFormer. Recall (R) and mean recall (mR) are reported. IMP [64] (CVPR'17), MOTIFS [71] (CVPR'18), VC-Tree [58] (CVPR'19), and GPSNet [44] (CVPR'20) all originate from the SGG task and are adapted for the PSG task. ... Models are trained using 12 epochs by default.

## Citation

```bibtex
@misc{yang2022panoptic,
  title={Panoptic Scene Graph Generation},
  author={Yang et al. (2022)},
  year={2022},
  note={arXiv:2207.11247}
}
```

- arXiv: 2207.11247