Visual Genome Sgg Eval

---
name: visual-genome-sgg-eval
description: Evaluates fine-grained scene graph generation by predicting subject-predicate-object triplets from images. It measures recall and F1 scores across head, body, and tail predicate classes to assess performance on long-tailed distributions and missing annotations. Use when the user wants to benchmark on Visual Genome, or asks about evaluating this task. Reports mR@K, F@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.09786
  bibtex_key: kim2024adaptive
  confidence: high
---

# visual-genome-sgg-eval

> Adaptive Self-training Framework for Fine-grained Scene Graph Generation — Kim et al. (2024) (arXiv:2401.09786, 2024)

## What this evaluates

Evaluates fine-grained scene graph generation by predicting subject-predicate-object triplets from images. It measures recall and F1 scores across head, body, and tail predicate classes to assess performance on long-tailed distributions and missing annotations.

## Datasets

- **Visual Genome** — total ?; splits: test (-1)

## Metrics

- `R@K` — range: percent
  - Recall@K: percentage of ground-truth predicates among the top-K predicted predicates per image.
- `mR@K` **(primary)** — range: percent
  - Mean Recall@K: macro-average of recall across all predicate classes at top-K predictions.
- `F@K` **(primary)** — range: percent
  - F1@K: harmonic mean of precision and recall at top-K predictions.

## Input / output format

**Input**: Image, bounding boxes for subjects and objects, and ground-truth predicate labels (for PredCls/SGCls) or only subject/object boxes (for SGDet).

**Output**: Top-K predicted predicate labels for each subject-object pair.

## Scoring recipe

```python
def compute_recall_at_k(preds, gold, k):
    correct = sum(1 for p, g in zip(preds[:k], gold) if p == g)
    return correct / len(gold) * 100

def compute_mR_at_k(preds_list, gold_list, k):
    recalls = [compute_recall_at_k(p, g, k) for p, g in zip(preds_list, gold_list)]
    return sum(recalls) / len(recalls)

def compute_F1_at_k(preds, gold, k):
    tp = sum(1 for p, g in zip(preds[:k], gold) if p == g)
    fp = k - tp
    fn = len(gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) * 100 if (prec + rec) > 0 else 0
```

## Common pitfalls

- mR@K is a macro-average, heavily penalizing models on tail predicates.
- F@K balances precision and recall, so high recall alone does not guarantee a high F-score.
- Metrics are reported at K=50 and K=100; results are not directly comparable across different K values.

## Evidence (verbatim from paper)

> We compare ST-SGG with state-of-the-arts methods that alleviate the long-tailed problem on commonly used benchmark datasets, VG and OI-V6. ... Motif+ST-SGG and VCTree+ST-SGG improve their performance in terms of mR@K and F@K, implying that ST-SGG greatly increases the performance on tail predicates while retaining that of head predicates.

## Citation

```bibtex
@misc{kim2024adaptive,
  title={Adaptive Self-training Framework for Fine-grained Scene Graph Generation},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2401.09786}
}
```

- arXiv: 2401.09786