Psgg Metrics Eval

---
name: psgg-metrics-eval
description: Evaluates panoptic scene graph generation models on their ability to predict object triplets with correct masks and relations. It probes recall-based performance at different top-k limits, mean recall across predicates, pair recall, and predicate ranking accuracy. Use when the user wants to benchmark on PSG, or asks about evaluating this task. Reports Mean Recall@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.09616
  bibtex_key: lorenz2024sgbench
  confidence: high
---

# psgg-metrics-eval

> A Review and Efficient Implementation of Scene Graph Generation Metrics — Lorenz et al. (2024) (arXiv:2404.09616, 2024)

## What this evaluates

Evaluates panoptic scene graph generation models on their ability to predict object triplets with correct masks and relations. It probes recall-based performance at different top-k limits, mean recall across predicates, pair recall, and predicate ranking accuracy.

## Datasets

- **PSG** — total ?; splits: test (-1)

## Metrics

- `Recall@k` — range: [0, 1]
  - Fraction of ground truth triplets matched within the top-k predicted triplets. k can be absolute (e.g., 20, 50, 100) or relative to the number of ground truth annotations (e.g., ×1, ×10).
- `Mean Recall@k` **(primary)** — range: [0, 1]
  - Average of Recall@k computed separately for each predicate class, then averaged across all predicates. Higher values indicate better overall triplet prediction quality.
- `Pair Recall@k` — range: [0, 1]
  - Fraction of ground truth subject-predicate pairs matched in the top-k predictions, ignoring the object entity. Probes relation prediction quality independently of object detection.
- `Predicate Rank` — range: other
  - Average rank of the correct predicate for each subject-object pair in the predicted list. Lower values indicate better predicate ranking.

## Input / output format

**Input**: Image, ground truth panoptic scene graph annotations (objects, masks, predicates, relations), and model-predicted scene graph triplets with associated masks and predicates.

**Output**: Model outputs a list of predicted triplets (subject, predicate, object) with corresponding masks/bounding boxes. Evaluation converts these to a standardized format for metric computation.

## Scoring recipe

```python
def compute_recall(preds, gold, k):
    top_k = preds[:k]
    matches = sum(1 for p in top_k if p in gold)
    return matches / len(gold)

def compute_mean_recall(preds, gold, k):
    recalls = []
    for pred_type in unique_predicates(gold):
        recalls.append(compute_recall(
            [t for t in preds if t.predicate == pred_type],
            [t for t in gold if t.predicate == pred_type], k))
    return sum(recalls) / len(recalls)

# k can be absolute int or relative multiplier (e.g., len(gold) * 10)
```

## Common pitfalls

- Confusing absolute k (fixed number like 20, 50, 100) with relative k (multiplier of ground truth count), which breaks cross-dataset comparability.
- Assuming high Pair Recall@k automatically implies good Predicate Rank; the paper shows they can be independent or inversely correlated.
- Failing to support overlapping masks in output formats, which breaks evaluation for modern panoptic scene graph models that require specific pickle formats.

## Evidence (verbatim from paper)

> Table 1 shows Recall@k values for different k. Apart from the common k ∈ {20,50,100}, we include two relative values for k. For R@×10, it is allowed to select 10 triplets per ground truth annotation. For R@×1, the number of output triplets has to be the same as the number ground truth triplets.

## Citation

```bibtex
@misc{lorenz2024sgbench,
  title={A Review and Efficient Implementation of Scene Graph Generation Metrics},
  author={Lorenz et al. (2024)},
  year={2024},
  note={arXiv:2404.09616}
}
```

- arXiv: 2404.09616