Stvg Eval

---
name: stvg-eval
description: Evaluates a model's ability to localize a specific object or event in a video based on a natural language query. It measures both temporal localization (identifying the correct start and end timestamps) and spatial localization (predicting accurate bounding box trajectories across the video frames). Use when the user wants to benchmark on VidSTG, HCSTVG-v1&v2, or asks about evaluating this task. Reports m_vIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08014
  bibtex_key: tu2026bridging
  confidence: high
---

# stvg-eval

> Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding — Tu et al. (2026) (arXiv:2604.08014, 2026)

## What this evaluates

Evaluates a model's ability to localize a specific object or event in a video based on a natural language query. It measures both temporal localization (identifying the correct start and end timestamps) and spatial localization (predicting accurate bounding box trajectories across the video frames).

## Datasets

- **VidSTG** — total 10000; splits: train (-1), val (-1), test (-1)
- **HCSTVG-v1&v2** — total 107000; splits: train (-1), val (-1), test (-1)

## Metrics

- `m_tIoU` — range: [0, 1]
  - Mean temporal Intersection-over-Union (tIoU) computed as the overlap between predicted and ground-truth temporal intervals divided by their union, averaged across all queries.
- `m_vIoU` **(primary)** — range: [0, 1]
  - Mean 3D Intersection-over-Union (vIoU) computed as the average spatial overlap between predicted and ground-tr bounding box trajectories across all frames in the temporal interval, averaged across all queries.
- `vIoU@0.3` — range: [0, 1]
  - Proportion of samples where the predicted vIoU exceeds a threshold of 0.3, indicating performance under moderate localization demands.
- `vIoU@0.5` — range: [0, 1]
  - Proportion of samples where the predicted vIoU exceeds a threshold of 0.5, indicating performance under strict localization demands.

## Input / output format

**Input**: Video frames uniformly sampled at 2 FPS, paired with a natural language query (declarative or interrogative sentence).

**Output**: Predicted temporal interval (start and end timestamps) and a sequence of spatial bounding boxes (x, y, width, height) for each frame within the interval.

## Scoring recipe

```python
def compute_stvg_metrics(predictions, gold):
    tIoU_scores = []
    vIoU_scores = []
    for pred, gt in zip(predictions, gold):
        t_iou = intersection_union(pred.t_start, pred.t_end, gt.t_start, gt.t_end)
        tIoU_scores.append(t_iou)
        v_iou = mean_3d_iou(pred.boxes, gt.boxes)
        vIoU_scores.append(v_iou)
    m_tIoU = sum(tIoU_scores) / len(tIoU_scores)
    m_vIoU = sum(vIoU_scores) / len(vIoU_scores)
    vIoU_03 = sum(1 for v in vIoU_scores if v > 0.3) / len(vIoU_scores)
    vIoU_05 = sum(1 for v in vIoU_scores if v > 0.5) / len(vIoU_scores)
    return m_tIoU, m_vIoU, vIoU_03, vIoU_05
```

## Common pitfalls

- Failing to separate declarative and interrogative sentence subsets when reporting VidSTG results, as they require different reasoning strategies and yield significantly different scores.
- Ignoring the 2 FPS video sampling rate during inference, which disrupts the temporal alignment and spatial token processing pipeline.
- Mixing training data (HCSTVG-v1&v2, synthetic ReVOS data) with test sets without strict train-test separation, leading to inflated performance.

## Evidence (verbatim from paper)

> Following (Gu et al., 2024, 2025a; Jin et al., 2022; Su et al., 2021; Yang et al., 2022), m_tIoU, m_vIoU, vIoU@R are adopted as evaluation metrics for STVG. m_tIoU reports the mean temporal Intersection-over-Union (tIoU) of predicted versus ground-truth intervals, evaluating temporal grounding. m_vIoU computes the average 3D IoU of spatio-temporal tubes to assess spatial grounding. vIoU@R measures the proportion of samples with vIoU exceeding a threshold R (e.g., 0.3, 0.5), indicating performance under precise localization demands.

## Citation

```bibtex
@misc{tu2026bridging,
  title={Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding},
  author={Tu et al. (2026)},
  year={2026},
  note={arXiv:2604.08014}
}
```

- arXiv: 2604.08014