Human Action Recognition Eval

---
name: human-action-recognition-eval
description: Evaluates a video foundation model's ability to recognize and classify human actions across diverse, real-world, and benchmark video datasets. It tests generalization from self-supervised pre-training on unstructured social media content to structured action recognition tasks. Use when the user wants to benchmark on Kinetics-400, Something-Something V2, UCF-101, HMDB51, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.08875
  bibtex_key: qian2024advancing
  confidence: high
---

# human-action-recognition-eval

> Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos — Yang Qian et al. (2024) (arXiv:2402.08875, 2024)

## What this evaluates

Evaluates a video foundation model's ability to recognize and classify human actions across diverse, real-world, and benchmark video datasets. It tests generalization from self-supervised pre-training on unstructured social media content to structured action recognition tasks.

## Datasets

- **Kinetics-400** — total ?; splits: test (-1)
- **Something-Something V2** — total ?; splits: test (-1)
- **UCF-101** — total ?; splits: test (-1)
- **HMDB51** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Fraction of correctly predicted action classes out of the total number of samples in the evaluation set, multiplied by 100.

## Input / output format

**Input**: Video clips sampled as a sequence of 16 frames from raw videos, resized to 224x224 pixels. Sampling stride varies by dataset (τ=4 for Kinetics-400, τ=2 for Something-Something V2).

**Output**: Predicted action class label from the dataset's predefined category set.

## Scoring recipe

```python
correct = 0
total = 0
for video, true_label in dataset:
    frames = sample_frames(video, num_frames=16, stride=dataset_stride)
    frames = resize(frames, 224, 224)
    pred_label = model(frames)
    if pred_label == true_label:
        correct += 1
    total += 1
return (correct / total) * 100
```

## Common pitfalls

- Confusing the pre-training dataset (TikTok, 283k unlabeled videos) with the evaluation benchmarks.
- Different fine-tuning protocols are used: direct fine-tuning for Kinetics-400 and Something-Something V2, versus transfer learning (K400 -> UCF101/HMDB51) for the latter two.
- Model capacity (ViT-Base vs ViT-Giant) drastically changes results; reporting without specifying backbone is misleading.
- Frame sampling stride differs across datasets (τ=4 vs τ=2), affecting temporal resolution and performance.

## Evidence (verbatim from paper)

> We report the fine-tuning accuracy of multiple view fusion $(5	imes 3)$ . Both models are pre-trained and fine-tuned at the input of $16	imes 224	imes 224$ and sampling stride $	au = 4$ .

## Citation

```bibtex
@misc{qian2024advancing,
  title={Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos},
  author={Yang Qian et al. (2024)},
  year={2024},
  note={arXiv:2402.08875}
}
```

- arXiv: 2402.08875