Track Any State Eval

---
name: track-any-state-eval
description: Evaluates a model's ability to track objects through appearance-changing state transformations and explicitly model those transformations as a state graph. It probes spatiotemporal continuity, zero-shot object recovery, and semantic reasoning about object interactions. Use when the user wants to benchmark on VOST, VSCOS, M3-VOS, DAVIS 2017, VOST-TAS, or asks about evaluating this task. Reports Jaccard (J).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.04678
  bibtex_key: sun2025tracking
  confidence: high
---

# track-any-state-eval

> Tracking and Understanding Object Transformations — Sun et al. (2025) (arXiv:2511.04678, 2025)

## What this evaluates

Evaluates a model's ability to track objects through appearance-changing state transformations and explicitly model those transformations as a state graph. It probes spatiotemporal continuity, zero-shot object recovery, and semantic reasoning about object interactions.

## Datasets

- **VOST** — total 70; splits: val (70)
- **VSCOS** — total 98; splits: val (98)
- **M3-VOS** — total 479; splits: test (479)
- **DAVIS 2017** — total ?; splits: val (-1)
- **VOST-TAS** — total 57; splits: test (57)

## Metrics

- `Jaccard (J)` **(primary)** — range: [0, 1]
  - Mean Intersection-over-Union (IoU) between predicted and ground-truth masks across all frames and objects.
- `J_tr` — range: [0, 1]
  - Jaccard score computed only over the last 25% of frames in each video.
- `Precision (P)` — range: [0, 1]
  - Mean IoU for predicted masks that achieve IoU > 0.5 with any ground-truth mask.
- `Recall (R)` — range: [0, 1]
  - Mean IoU for ground-truth masks that achieve IoU > 0.5 with any predicted mask.
- `Temporal Precision (T_P)` — range: [0, 1]
  - Precision of predicted transformation start/end frames against annotated temporal boundaries.
- `Temporal Recall (T_R)` — range: [0, 1]
  - Recall of predicted transformation start/end frames against annotated temporal boundaries.
- `Semantic Accuracy (S_V)` — range: [0, 1]
  - Accuracy of predicted action verbs, computed only on transformations that are correctly localized temporally.
- `Semantic Accuracy (S_O)` — range: [0, 1]
  - Accuracy of predicted resulting object descriptions, computed only on objects matched with IoU > 0.5.
- `Spatiotemporal Recall (H_ST)` — range: [0, 1]
  - Recall of transformations where temporal localization is correct AND all resulting objects are matched with IoU > 0.5.
- `Overall Recall (H)` — range: [0, 1]
  - Recall of transformations where temporal localization is correct, all resulting objects are matched with IoU > 0.5, AND all action/object descriptions are correct.

## Input / output format

**Input**: Video frames with an initial object mask or tracklet prompt.

**Output**: Per-frame segmentation masks for the tracked object(s), plus a structured state graph listing each transformation with start/end frames, an action verb, and resulting objects with masks and text descriptions.

## Scoring recipe

```python
def compute_tracking(pred_masks, gt_masks):
    ious = compute_iou(pred_masks, gt_masks) # per frame
    J = mean(ious)
    J_tr = mean(ious[last_25_frames])
    P = mean(max_iou_per_pred > 0.5)
    R = mean(max_iou_per_gt > 0.5)
    return J, J_tr, P, R

def compute_state_graph(pred_trans, gt_trans):
    T_P = precision(pred_trans.start_end, gt_trans.start_end)
    T_R = recall(pred_trans.start_end, gt_trans.start_end)
    S_V = accuracy(pred_verbs[correct_temporal], gt_verbs[correct_temporal])
    S_O = accuracy(pred_obj_descs[iou>0.5], gt_obj_descs[iou>0.5])
    H_ST = recall(trans where temporal_correct AND all_objects_iou>0.5)
    H = recall(trans where temporal_correct AND all_objects_iou>0.5 AND all_descs_correct)
    return T_P, T_R, S_V, S_O, H_ST, H
```

## Common pitfalls

- J_tr is evaluated exclusively on the final 25% of frames, not the full video duration.
- Semantic accuracy metrics (S_V, S_O) are conditional on successful temporal/spatial localization, masking overall system failure rates.
- Object matching in state graph evaluation strictly uses an IoU > 0.5 threshold, which may penalize slight segmentation drift.
- DAVIS 2017 serves as a non-transformation baseline but is evaluated with the same protocol, potentially inflating expectations for standard tracking.

## Evidence (verbatim from paper)

> To evaluate state graph quality, we report precision ${\mathcal{T}}_{P}$ and recall ${\mathcal{T}}_{R}$ for temporal localization within annotated transformation boundaries, and description accuracy for correctly localized action verbs (${\mathcal{A}}_{V}$) and resulting objects (${\mathcal{A}}_{O}$) with IoU $>0.5$.

## Citation

```bibtex
@misc{sun2025tracking,
  title={Tracking and Understanding Object Transformations},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2511.04678}
}
```

- arXiv: 2511.04678