Synthverse Eval

---
name: synthverse-eval
description: This benchmark evaluates 2D and 3D point tracking capabilities across diverse synthetic domains, including rapid camera motion, articulated objects, and occlusions. It probes a model's ability to maintain spatio-temporal correspondence, handle depth-adaptive spatial errors, and correctly classify occlusion or out-of-frame status under significant distribution shifts. Use when the user wants to benchmark on SynthVerse, or asks about evaluating this task. Reports AJ_3D.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.04441
  bibtex_key: zhao2026synthverse
  confidence: high
---

# synthverse-eval

> SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking — Zhao et al. (2026) (arXiv:2602.04441, 2026)

## What this evaluates

This benchmark evaluates 2D and 3D point tracking capabilities across diverse synthetic domains, including rapid camera motion, articulated objects, and occlusions. It probes a model's ability to maintain spatio-temporal correspondence, handle depth-adaptive spatial errors, and correctly classify occlusion or out-of-frame status under significant distribution shifts.

## Datasets

- **SynthVerse** — total 5800000; splits: train (5800000), test (-1)

## Metrics

- `AJ_3D` **(primary)** — range: percent
  - Calculates the proportion of points whose predicted 3D positions fall within a depth-adaptive error tolerance relative to ground truth, assessing spatial and visibility overlap in 3D coordinates.
- `APD_3D` — range: other
  - Measures the average Euclidean distance between predicted and ground truth 3D trajectories.
- `AJ_2D` — range: percent
  - Measures the spatio-temporal alignment of pixels across video sequences using the Intersection-over-Union metric after mapping 3D results back to 2D images.
- `APD_2D` — range: other
  - Measures whether the average Euclidean distance between predicted and ground truth trajectories on the pixel plane remains within minimal error bounds.
- `OA` — range: percent
  - Evaluates the accuracy of the model’s binary classification regarding whether a tracked point is occluded or out of frame.

## Input / output format

**Input**: Video sequences with initial 2D/3D point coordinates or query points to track.

**Output**: Predicted 2D/3D trajectories and binary occlusion/out-of-frame labels per frame.

## Scoring recipe

```python
def compute_metrics(pred_2d, pred_3d, pred_occ, gt_2d, gt_3d, gt_occ):
    aj_3d = mean([1 if dist_3d(p, g) < depth_adaptive_tol(g) else 0 for p, g in zip(pred_3d, gt_3d)])
    apd_3d = mean([dist_3d(p, g) for p, g in zip(pred_3d, gt_3d)])
    aj_2d = iou(pred_2d, gt_2d)
    apd_2d = mean([dist_2d(p, g) for p, g in zip(pred_2d, gt_2d)])
    oa = accuracy(pred_occ, gt_occ)
    return {'AJ_3D': aj_3d, 'APD_3D': apd_3d, 'AJ_2D': aj_2d, 'APD_2D': apd_2d, 'OA': oa}
```

## Common pitfalls

- Depth-adaptive tolerance for AJ_3D is not a fixed threshold but scales with scene depth, requiring careful implementation to avoid over/under-counting valid points.
- OA requires strict binary alignment with ground truth occlusion masks; partial occlusions or out-of-frame states must be explicitly handled to match evaluation conventions.
- 3D tracking performance degrades significantly under rapid viewpoint changes and large camera motion compared to 2D, so evaluating only on static scenes will overestimate capability.

## Evidence (verbatim from paper)

> Following TAPIP3D*[[35]]*, we adopt AJ_3D, APD_3D, AJ_2D, APD_2D, OA as the main evaluation metric. AJ_3D assesses tracking rigor by calculating the spatial and visibility overlap between predicted points and ground truth in 3D coordinates. AJ_3D quantifies the proportion of points whose predicted 3D positions fall within a depth-adaptive error tolerance relative to the actual coordinates. In addition, OA specifically evaluates the accuracy of the model’s binary classification regarding whether a tracked point is occluded or out of frame. After mapping results back to 2D images, AJ_2D measures the spatio-temporal alignment of pixels across video sequences using the Intersection-over-Union metric. Moreover, APD_2D measures whether the average Euclidean distance between the predicted trajectories and ground truth on the pixel plane remains within minimal error bounds.

## Citation

```bibtex
@misc{zhao2026synthverse,
  title={SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2602.04441}
}
```

- arXiv: 2602.04441