Thinkjepa Ego Dex Eval

---
name: thinkjepa-ego-dex-eval
description: Evaluates a model's ability to forecast future 3D hand/joint trajectories and latent video representations from egocentric video inputs. It probes long-horizon temporal consistency and physical plausibility in dexterous manipulation scenarios. Use when the user wants to benchmark on EgoDex, EgoExo4D, or asks about evaluating this task. Reports ADE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.22281
  bibtex_key: [REDACTED]
  confidence: high
---

# thinkjepa-ego-dex-eval

> ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model — Zhang et al. (2026) (arXiv:2603.22281, 2026)

## What this evaluates

Evaluates a model's ability to forecast future 3D hand/joint trajectories and latent video representations from egocentric video inputs. It probes long-horizon temporal consistency and physical plausibility in dexterous manipulation scenarios.

## Datasets

- **EgoDex** — total ?; splits: train (-1), val (-1), test (-1)
- **EgoExo4D** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `ADE` **(primary)** — range: other
  - Mean Euclidean distance between predicted and ground-truth 3D joint positions across all future frames and joints, averaged over the batch.
- `FDE` — range: other
  - Mean Euclidean distance on the final future frame, averaged over joints and batch.
- `Accuracy` — range: percent
  - Fraction of predicted joint positions where the Euclidean error is below 0.05 m, aggregated over time and joints.
- `FD` — range: other
  - Feature L2 distance between predicted and target V-JEPA latent representations.
- `SL1` — range: other
  - SmoothL1 distance between predicted and target latents.
- `CD` — range: other
  - Cosine distance, defined as 1 - cosine similarity, between predicted and target latents.
- `A@H / F@H` — range: other
  - ADE@H and FDE@H computed after recursive autoregressive rollout to horizon H ∈ {4, 8, 16, 32}.

## Input / output format

**Input**: Sequence of past egocentric video frames (typically 32 frames) used to predict future 3D hand/joint trajectories and latent video representations.

**Output**: Predicted 3D joint coordinates Y_hat ∈ R^(B x T_f x J x 3) for future frames, and predicted latent feature vectors.

## Scoring recipe

```python
def compute_metrics(pred_traj, gt_traj, pred_lat, gt_lat, threshold=0.05):
    dist = np.linalg.norm(pred_traj - gt_traj, axis=-1)
    ade = np.mean(dist)
    fde = np.mean(dist[:, -1, :])
    acc = np.mean(dist < threshold)
    fd = np.mean(np.linalg.norm(pred_lat - gt_lat, axis=-1))
    sl1 = np.mean(smooth_l1_loss(pred_lat, gt_lat))
    cd = np.mean(1 - cosine_similarity(pred_lat, gt_lat))
    return ade, fde, acc, fd, sl1, cd
```

## Common pitfalls

- Latent metrics (FD/SL1/CD) are computed on V-JEPA feature space, not raw pixel space.
- Rollout metrics (A@H/F@H) require recursive autoregressive prediction, not single-step forecasting.
- Accuracy uses a strict 0.05 m Euclidean threshold per joint, not a global pose similarity metric.

## Evidence (verbatim from paper)

> We compute: ADE (Average Displacement Error): the mean Euclidean distance over all future frames and joints, averaged over the batch. FDE (Final Displacement Error): the mean Euclidean distance on the final future frame, averaged over joints and batch. Accuracy: the fraction of predicted joint positions with Euclidean error below 0.05 m, aggregated over time and joints.

## Citation

```bibtex
@misc{zhang2026thinkjepa,
  title={ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2603.22281}
}
```

- arXiv: 2603.22281