Severe Benchmark Eval

---
name: severe-benchmark-eval
description: Evaluates the generalization and sensitivity of video self-supervised learning models to domain shifts, downstream sample sizes, action similarity, and task shifts beyond action recognition. Use when the user wants to benchmark on UCF-101, NTU-60, FineGym (Gym-99), Something-Something-v2, EPIC-Kitchens-100, Charades, AVA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.14221
  bibtex_key: thoker2022severe
  confidence: high
---

# severe-benchmark-eval

> How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning? — Thoker et al. (2022) (arXiv:2203.14221, 2022)

## What this evaluates

Evaluates the generalization and sensitivity of video self-supervised learning models to domain shifts, downstream sample sizes, action similarity, and task shifts beyond action recognition.

## Datasets

- **UCF-101** — total ?; splits: test (-1)
- **NTU-60** — total ?; splits: test (-1)
- **FineGym (Gym-99)** — total ?; splits: test (-1)
- **Something-Something-v2** — total ?; splits: test (-1)
- **EPIC-Kitchens-100** — total ?; splits: test (-1)
- **Charades** — total ?; splits: test (-1)
- **AVA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Top-1 classification accuracy computed over the test split after finetuning a task-dependent head on the pre-trained R(2+1)D-18 backbone.

## Input / output format

**Input**: Video clips/frames from downstream datasets, processed through a pre-trained R(2+1)D-18 backbone.

**Output**: Class labels (or action categories) predicted by a task-dependent head attached to the backbone.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Assuming self-supervised pre-training always outperforms supervised pre-training across all shifts.
- Overlooking the impact of domain/task shifts by only evaluating on datasets similar to Kinetics-400.
- Not controlling for hyperparameters during finetuning across different pre-trained models.

## Evidence (verbatim from paper)

> To finetune for downstream tasks we simply attach a task-dependent head at the last layer of the pre-trained R(2+1)D-18 backbone to produce label predictions for the corresponding task. For a fair comparison, we use the same set of hyper-parameters, optimization and pre-processing during the downstream training of each model.

## Citation

```bibtex
@misc{thoker2022severe,
  title={How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?},
  author={Thoker et al. (2022)},
  year={2022},
  note={arXiv:2203.14221}
}
```

- arXiv: 2203.14221