Pvsg Eval

---
name: pvsg-eval
description: Evaluates a model's ability to generate temporal scene graphs where nodes are grounded with pixel-level panoptic segmentation masks instead of bounding boxes, capturing non-rigid objects, backgrounds, and fine-grained interactions in dynamic videos. Use when the user wants to benchmark on PVSG, or asks about evaluating this task. Reports R/mR@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.17058
  bibtex_key: yang2023panoptic
  confidence: high
---

# pvsg-eval

> Panoptic Video Scene Graph Generation — Yang et al. (2023) (arXiv:2311.17058, 2023)

## What this evaluates

Evaluates a model's ability to generate temporal scene graphs where nodes are grounded with pixel-level panoptic segmentation masks instead of bounding boxes, capturing non-rigid objects, backgrounds, and fine-grained interactions in dynamic videos.

## Datasets

- **PVSG** — total 400; splits: train (338), test (62)

## Metrics

- `R/mR@20` **(primary)** — range: [0, 100] percent
  - Recall (R) and mean Recall (mR) at top-20 predicted relations per query. A prediction is counted as correct if the predicted mask tube overlaps with the ground-truth mask tube above a specified IoU threshold (e.g., 0.1 or 0.5).

## Input / output format

**Input**: Video frames with ground-truth panoptic segmentation masks and temporal scene graph annotations (subject-predicate-object triplets with mask tubes).

**Output**: Predicted scene graph triplets with corresponding mask tubes for each frame in the video.

## Scoring recipe

```python
def compute_recall(preds, golds, k=20, iou_thresh=0.1):
    correct = 0
    for gt in golds:
        top_k = sorted(preds[gt.subject], key=lambda x: x.score, reverse=True)[:k]
        for pred in top_k:
            if compute_mask_iou(pred.mask_tube, gt.mask_tube) >= iou_thresh:
                correct += 1
                break
    return (correct / len(golds)) * 100
```

## Common pitfalls

- The task uses mask tubes instead of bounding boxes, requiring temporal consistency across frames rather than single-frame detection.
- Recall is evaluated at a strict IoU threshold (e.g., 0.5) but the paper notes a looser 0.1 threshold yields significantly higher scores, which can mislead comparisons if not specified.
- Performance heavily depends on the first-stage mask tracking quality, as poor tracking directly degrades relation prediction regardless of the second-stage model.

## Evidence (verbatim from paper)

> We split the dataset with 338 videos for training and 62 videos for testing. When examining Table 2, it is crucial to prioritize the R/mR@20 as it represents our most significant indicator. The highest value for R@20 currently stands at 3.88, meaning that roughly for every 25 ground-truth triplets, one meets the criteria for a successful recall, indicating a relatively low efficiency. However, when setting the threshold to 0.1, the score improves to around 10, meaning the model can predict one in every 10 triplets with a looser requirement of recall.

## Citation

```bibtex
@misc{yang2023panoptic,
  title={Panoptic Video Scene Graph Generation},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2311.17058}
}
```

- arXiv: 2311.17058