Vl Jepa Zero Shot Benchmarks

---
name: vl-jepa-zero-shot-benchmarks
description: Evaluates zero-shot video understanding capabilities on action recognition and text-to-video retrieval tasks across diverse benchmarks. Use when the user wants to benchmark on Something-something-v2 (SSv2), EPIC-KITCHENS-100 (EK-100), EgoExo4D Keysteps, Kinetics-400, COIN, CrossTask, MSR-VTT, ActivityNet, DiDeMo, MSVD, YouCook2, PVD-Bench, Dream-1K, VDC-1K, or asks about evaluating this task. Reports top-1 accuracy, recall@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.10592
  bibtex_key: chen2026action100m
  confidence: high
---

# vl-jepa-zero-shot-benchmarks

> Action100M: A Large-scale Video Action Dataset — Chen et al. (2026) (arXiv:2601.10592, 2026)

## What this evaluates

Evaluates zero-shot video understanding capabilities on action recognition and text-to-video retrieval tasks across diverse benchmarks.

## Datasets

- **Something-something-v2 (SSv2)** — total ?; splits: test (-1)
- **EPIC-KITCHENS-100 (EK-100)** — total ?; splits: test (-1)
- **EgoExo4D Keysteps** — total ?; splits: test (-1)
- **Kinetics-400** — total ?; splits: test (-1)
- **COIN** — total ?; splits: test (-1)
- **CrossTask** — total ?; splits: test (-1)
- **MSR-VTT** — total ?; splits: test (-1)
- **ActivityNet** — total ?; splits: test (-1)
- **DiDeMo** — total ?; splits: test (-1)
- **MSVD** — total ?; splits: test (-1)
- **YouCook2** — total ?; splits: test (-1)
- **PVD-Bench** — total ?; splits: test (-1)
- **Dream-1K** — total ?; splits: test (-1)
- **VDC-1K** — total ?; splits: test (-1)

## Metrics

- `top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of the total number of instances.
- `recall@1` **(primary)** — range: [0, 1]
  - Fraction of queries where the ground-truth video is ranked first in the retrieved list.

## Input / output format

**Input**: Video frames sampled at 1, 8, or 32 frames per input, paired with class labels or text queries.

**Output**: Predicted class label for action recognition; ranked list of videos for retrieval.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    r1 = sum(1 for p, g in zip(predictions, golds) if p[0] == g) / len(golds)
    return {'top-1_accuracy': acc, 'recall@1': r1}
```

## Common pitfalls

- COIN and CrossTask require separate evaluation for segment-level step recognition and global task recognition.
- Zero-shot evaluation implies no fine-tuning on target benchmarks, only inference.
- Frame sampling strategy (1, 8, or 32 frames) significantly impacts memory and performance, requiring consistent settings across models.

## Evidence (verbatim from paper)

> We evaluate on two tasks: Zero-shot action recognition (top-1 accuracy) on eight benchmarks: Something-something-v2 (SSv2) (Goyal et al., [2017]), EPIC-KITCHENS-100 (EK-100) (Damen et al., [2020]), EgoExo4D Keysteps (Grauman et al., [2024]), Kinetics-400 (Kay et al., [2017]), COIN (Tang et al., [2019]), and CrossTask (Zhukov et al., [2019]). For COIN and Crosstask, we evaluate both segment-level step recognition and global task recognition. Zero-shot text-to-video retrieval (recall@1) on eight benchmarks: MSR-VTT (Xu et al., [2016]), ActivityNet (Caba Heilbron et al., [2015]), DiDeMo (Anne Hendricks et al., [2017]), MSVD (Chen and Dolan, [2011]), YouCook2 (Zhou et al., [2018]), PVD-Bench (Bolya et al., [2025]), Dream-1K (Wang et al., [2024a]), and VDC-1K (Chai et al., [2024]).

## Citation

```bibtex
@misc{chen2026action100m,
  title={Action100M: A Large-scale Video Action Dataset},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.10592}
}
```

- arXiv: 2601.10592