Egtr Sgg Eval

---
name: egtr-sgg-eval
description: Evaluates a model's ability to detect objects and predict relational triplets (subject-predicate-object) in natural images. It probes both object detection accuracy and scene graph generation quality under graph constraints and standard recall/mAP metrics. Use when the user wants to benchmark on Visual Genome, Open Image V6, or asks about evaluating this task. Reports Recall@k (R@k), micro-R@50.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.02072
  bibtex_key: im2024egtr
  confidence: high
---

# egtr-sgg-eval

> EGTR: Extracting Graph from Transformer for Scene Graph Generation — Im et al. (2024) (arXiv:2404.02072, 2024)

## What this evaluates

Evaluates a model's ability to detect objects and predict relational triplets (subject-predicate-object) in natural images. It probes both object detection accuracy and scene graph generation quality under graph constraints and standard recall/mAP metrics.

## Datasets

- **Visual Genome** — total 88000; splits: train (57000), val (5000), test (26000)
- **Open Image V6** — total 133000; splits: train (126000), val (2000), test (5000)

## Metrics

- `Recall@k (R@k)` **(primary)** — range: percent
  - Class-agnostic recall of predicted predicates at top-k per object pair under a graph constraint (single predicate allowed per object pair).
- `mean Recall@k (mR@k)` — range: percent
  - Mean of R@k aggregated across each predicate category, emphasizing tail predicates.
- `AP50` — range: percent
  - Average Precision at IoU threshold 0.5 for object detection performance across all objects in the scene.
- `micro-R@50` **(primary)** — range: percent
  - Micro-averaged recall at top-50 predictions for Open Image V6.
- `wmAP_rel` — range: percent
  - Weighted mean AP for predicting subject and object bounding boxes separately.
- `wmAP_phr` — range: percent
  - Weighted mean AP for predicting a union bounding box of subject and object pairs.
- `Final Score` — range: other
  - Weighted combination: 0.2 × micro-R@50 + 0.4 × wmAP_rel + 0.4 × wmAP_phr.

## Input / output format

**Input**: RGB images (standard resolution; FPS measured on images resized to min 600px shortest side and max 1000px longest side).

**Output**: Set of predicted scene graph triplets (subject bounding box, predicate class, object bounding box) and object bounding boxes.

## Scoring recipe

```python
def score_vg(preds, gold, k):
    recalls = []
    for pair in gold:
        top_k = sorted(preds[pair], key=lambda x: x.score, reverse=True)[:k]
        recalls.append(1.0 if any(p.rel == gold[pair].rel for p in top_k) else 0.0)
    R_k = sum(recalls) / len(recalls)
    mR_k = mean(mean_recall_per_predicate_class(recalls, gold))
    return R_k, mR_k

def score_oiv6(preds, gold):
    micro_R50 = recall_at_k(preds, gold, k=50, average='micro')
    wmAP_rel = weighted_mean_AP(preds.sub_boxes, preds.rel, gold)
    wmAP_phr = weighted_mean_AP(preds.union_boxes, preds.rel, gold)
    final = 0.2 * micro_R50 + 0.4 * wmAP_rel + 0.4 * wmAP_phr
    return micro_R50, wmAP_rel, wmAP_phr, final
```

## Common pitfalls

- Graph constraint: Evaluation enforces a single predicate per object pair, which differs from standard multi-label triplet detection and can penalize models that predict multiple valid relations.
- AP50 calculation varies by architecture: For explicit object detectors, AP50 is computed directly on object boxes. For triplet detectors without explicit detectors, AP50 requires applying NMS to the union of predicted subjects and objects first.
- Logit adjustment trade-off: Applying logit adjustment significantly improves mR@k (tail predicates) but drops R@k (head predicates), requiring careful threshold tuning to balance performance.

## Evidence (verbatim from paper)

> We adopt Scene Graph Detection (SGDet) evaluation settings and report Recall@k (R@k) which is class agnostic and mean Recall@k (mR@k) that aggregates the recalls for each predicate category. Following Motifs [45], these metrics are measures with graph constraint, which means each object pair can have a single predicate category. ... For recall evaluation, micro-R@50 is adopted. The WMAP is evaluated with two settings: wmAP_rel for predicting boxes of subject entity and object entity separately and wmAP_phr for predicting a union box of them. The final score is calculated by 0.2 × micro-R@50 + 0.4 × wmAP_rel + 0.4 × wmAP_phr.

## Citation

```bibtex
@misc{im2024egtr,
  title={EGTR: Extracting Graph from Transformer for Scene Graph Generation},
  author={Im et al. (2024)},
  year={2024},
  note={arXiv:2404.02072}
}
```

- arXiv: 2404.02072