Tapvid 3d Eval

---
name: tapvid-3d-eval
description: Evaluates a model's ability to track arbitrary points in 3D space over time from monocular video, assessing spatio-temporal consistency, depth accuracy, and occlusion handling. It probes whether models can reconstruct scene geometry up to scale or only maintain relative depth consistency for local interactions. Use when the user wants to benchmark on TAPVid-3D, or asks about evaluating this task. Reports 3D Average Jaccard (3D-AJ).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.05921
  bibtex_key: koppula2024tapvid3d
  confidence: high
---

# tapvid-3d-eval

> TAPVid-3D: A Benchmark for Tracking Any Point in 3D — Koppula et al. (2024) (arXiv:2407.05921, 2024)

## What this evaluates

Evaluates a model's ability to track arbitrary points in 3D space over time from monocular video, assessing spatio-temporal consistency, depth accuracy, and occlusion handling. It probes whether models can reconstruct scene geometry up to scale or only maintain relative depth consistency for local interactions.

## Datasets

- **TAPVid-3D** — total ?; splits: full_eval (-1)

## Metrics

- `3D Average Jaccard (3D-AJ)` **(primary)** — range: [0, 1]
  - Jaccard index computed on 3D point trajectories after rescaling, measuring the overlap between predicted and ground-truth 3D point sets over time while accounting for occlusions and spatio-temporal smoothness.
- `APD_3D` — range: meters
  - Average Point Distance in 3D, measuring the mean Euclidean distance between predicted and ground-truth 3D points along trajectories after rescaling.
- `OA` — range: [0, 1]
  - Occlusion Accuracy, measuring the fraction of correctly predicted visibility/occlusion states across all frames and points.

## Input / output format

**Input**: Monocular video frames, a query frame index, and a 2D point coordinate in that frame.

**Output**: A 3D trajectory (sequence of 3D points over time) and a binary visibility/occlusion mask for each frame.

## Scoring recipe

```python
def compute_metrics(pred_traj, gt_traj, pred_vis, gt_vis, t_query, rescale_mode='global'):
    if rescale_mode == 'global':
        scale = median(norm(gt_traj) / norm(pred_traj))
    elif rescale_mode == 'per_trajectory':
        scale = norm(gt_traj[t_query]) / norm(pred_traj[t_query])
    else: # local_neighborhood
        scale = compute_neighborhood_scale(gt_traj, pred_traj, tau=0.03)
    pred_rescaled = pred_traj * scale
    aj = jaccard_overlap(pred_rescaled, gt_traj, gt_vis)
    apd = mean_euclidean_dist(pred_rescaled, gt_traj, gt_vis)
    oa = accuracy(pred_vis, gt_vis)
    return aj, apd, oa
```

## Common pitfalls

- Depth ambiguity makes global scale estimation difficult without strong camera motion or known scene geometry, unfairly penalizing models that only need relative depth.
- Local vs. global scaling requirements vary by application; using the default global median rescaling may be overly stringent for tasks like robotic imitation.
- Occlusion states must be explicitly predicted and matched; ignoring visibility masks leads to inflated distance errors and incorrect Jaccard calculations.

## Evidence (verbatim from paper)

> In the results included in the main paper, we compute the 3D Average Jaccard and APD metrics using a global median rescaling procedure (L277). To get a good score, the entire scene must be reconstructed up to scale, and dynamic objects must be placed precisely.

## Citation

```bibtex
@misc{koppula2024tapvid3d,
  title={TAPVid-3D: A Benchmark for Tracking Any Point in 3D},
  author={Koppula et al. (2024)},
  year={2024},
  note={arXiv:2407.05921}
}
```

- arXiv: 2407.05921