Thyme Scene Graph Eval

---
name: thyme-scene-graph-eval
description: Evaluates a model's ability to generate dynamic video scene graphs by predicting inter-object relationships and attributes across multiple temporal frames. It specifically probes temporal consistency, handling of occlusions, and modeling of long-range dependencies in both ground-level and aerial video footage. Use when the user wants to benchmark on ASPIRe, AeroEye-v1.0, or asks about evaluating this task. Reports R@20, mR@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.09200
  bibtex_key: nguyen2025thyme
  confidence: high
---

# thyme-scene-graph-eval

> THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage — Trong-Thuan Nguyen et al. (2025) (arXiv:2507.09200, 2025)

## What this evaluates

Evaluates a model's ability to generate dynamic video scene graphs by predicting inter-object relationships and attributes across multiple temporal frames. It specifically probes temporal consistency, handling of occlusions, and modeling of long-range dependencies in both ground-level and aerial video footage.

## Datasets

- **ASPIRe** — total ?; splits: test (-1)
- **AeroEye-v1.0** — total ?; splits: test (-1)

## Metrics

- `R@20` **(primary)** — range: percent
  - Recall@K measures the fraction of ground-truth predicates correctly predicted within the top-K ranked predictions. Evaluated at K=20, 50, and 100.
- `mR@20` **(primary)** — range: percent
  - Mean Recall@K computes the average recall across all predicate classes to mitigate class imbalance. Evaluated at K=20, 50, and 100.

## Input / output format

**Input**: Video sequences (ground-level or aerial) with annotated object bounding boxes, attributes, and ground-truth scene graph edges categorized into five interactivity types: Appearance, Situation, Position, Interaction, and Relation.

**Output**: Top-K predicted scene graph edges (subject-predicate-object triples) per video frame/clip, ranked by confidence score.

## Scoring recipe

```python
def compute_recall(preds, gold, k=20):
    top_k = set(preds[:k])
    correct = len(top_k.intersection(gold))
    return (correct / len(gold)) * 100 if gold else 0

def compute_mean_recall(preds_list, gold_list, k=20):
    recalls = [compute_recall(p, g, k) for p, g in zip(preds_list, gold_list)]
    return sum(recalls) / len(recalls) if recalls else 0
```

## Common pitfalls

- Evaluating only at top-20 without checking top-50/100 may overstate performance on long-tail predicate classes.
- Failing to account for temporal occlusions can cause models to incorrectly penalize valid predictions where objects temporarily disappear from the frame.
- Reporting only standard Recall without Mean Recall hides severe class imbalance issues in rare interactivity types like Interaction or Relation.

## Evidence (verbatim from paper)

> Following[[52], [32], [33]], we evaluate methods using Recall (R) and mean Recall (mR) at multiple thresholds, specifically, predictions from the top-20, top-50, and top-100.

## Citation

```bibtex
@misc{nguyen2025thyme,
  title={THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage},
  author={Trong-Thuan Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2507.09200}
}
```

- arXiv: 2507.09200