Cyclo Sgg Eval

---
name: cyclo-sgg-eval
description: Evaluates a model's ability to generate scene graphs from aerial video sequences by predicting object relationships and interactions. It specifically probes long-range temporal dependency modeling and periodic interaction recognition in drone-captured footage across predicate classification, scene graph classification, and scene graph detection tasks. Use when the user wants to benchmark on AeroEye, PVSG, ASPIRe, or asks about evaluating this task. Reports mean Recall@K (mR@K).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.01029
  bibtex_key: nguyen2024cyclo
  confidence: high
---

# cyclo-sgg-eval

> CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos — Nguyen et al. (2024) (arXiv:2406.01029, 2024)

## What this evaluates

Evaluates a model's ability to generate scene graphs from aerial video sequences by predicting object relationships and interactions. It specifically probes long-range temporal dependency modeling and periodic interaction recognition in drone-captured footage across predicate classification, scene graph classification, and scene graph detection tasks.

## Datasets

- **AeroEye** — total 2260; splits: train (1797), test (463)
- **PVSG** — total ?; splits: (unstated)
- **ASPIRe** — total ?; splits: (unstated)

## Metrics

- `Recall@K (R@K)` — range: percent
  - The percentage of ground-truth predicates correctly predicted within the top K predictions for a video sequence.
- `mean Recall@K (mR@K)` **(primary)** — range: percent
  - The average of Recall@K across all predicate categories. This metric is used to handle long-tail class distributions common in scene graph generation.

## Input / output format

**Input**: Video sequences of aerial/drone footage. Visual features are extracted using DINO (ResNet-50 backbone). For PredCls and SGCls, ground-truth object bounding boxes are provided; for SGDet, raw frames are processed.

**Output**: Scene graphs comprising predicted object classes, bounding boxes (SGDet only), and predicate relationships linking detected objects.

## Scoring recipe

```python
def compute_recall(preds, gold, k):
    top_k = preds[:k]
    return sum(1 for p in top_k if p in gold) / len(gold) * 100

def compute_mR(preds, gold, k):
    cat_correct, cat_total = {}, {}
    for p, g in zip(preds, gold):
        cat = g.category
        cat_total[cat] = cat_total.get(cat, 0) + 1
        if p in g: cat_correct[cat] = cat_correct.get(cat, 0) + 1
    recalls = [cat_correct[c]/cat_total[c] for c in cat_total]
    return sum(recalls) / len(recalls) * 100
```

## Common pitfalls

- Failing to distinguish between PredCls (uses GT objects), SGCls (uses GT objects), and SGDet (requires object detection) evaluation protocols.
- Reporting raw Recall instead of mean Recall (mR), which masks performance on long-tail predicate categories.
- Using inconsistent K thresholds (20, 50, 100) without standardizing the reporting format across papers.

## Evidence (verbatim from paper)

> Evaluation Metrics. We evaluate models on two standard tasks in image-based scene graph generation followed by previous work[[74], [7]]* that are predicate classification (PredCls), scene graph classification (SGCls), and scene graph detection (SGDet). While SGCls predicts relationships given ground truth objects, SGDet involves detecting objects and predicting relationships. These tasks are evaluated using Recall (R@K) and mean Recall (mR@$K$), where $K\in{20,50,100}$.

## Citation

```bibtex
@misc{nguyen2024cyclo,
  title={CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2406.01029}
}
```

- arXiv: 2406.01029