Lightweight Action Recognition Eval

---
name: lightweight-action-recognition-eval
description: Evaluates the real-world efficiency (training and inference latency, VRAM footprint) of video action recognition models across desktop GPUs and mobile devices, alongside their classification accuracy on standard benchmarks. Use when the user wants to benchmark on EK100, SSV2, K400, or asks about evaluating this task. Reports relative latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.09641
  bibtex_key: [REDACTED]
  confidence: high
---

# lightweight-action-recognition-eval

> Evaluating Transformers for Lightweight Action Recognition — Koot et al. (2021) (arXiv:2111.09641, 2021)

## What this evaluates

Evaluates the real-world efficiency (training and inference latency, VRAM footprint) of video action recognition models across desktop GPUs and mobile devices, alongside their classification accuracy on standard benchmarks.

## Datasets

- **EK100** — total 67217; splits: train (67217), val (9668), test (3568)
- **SSV2** — total 168913; splits: train (168913), val (24777), test (174)
- **K400** — total 246535; splits: train (246535), val (19907), test (400)

## Metrics

- `relative latency` **(primary)** — range: other
  - Measured as the ratio of a model's average runtime to the fastest model's runtime on the same device, where the fastest model is normalized to 1.0. Measured for training steps, batch-mode inference, and single-instance inference.
- `VRAM requirement` — range: other
  - Peak video memory usage in gigabytes (GB) during training, batch-mode inference, and single-instance inference, measured on a representative set of desktop GPUs and mobile devices.

## Input / output format

**Input**: Video clips resized to 224x224 frames, processed in batches (size 8 for training, 32 for batch inference, 1 for single-instance) or as single instances.

**Output**: Predicted action class labels, plus per-model per-device latency (relative or absolute) and peak VRAM usage (GB).

## Scoring recipe

```python
# Measure raw latencies and VRAM
latencies = {m: measure_avg_latency(m, device, batch_size) for m in models}
vram = {m: measure_peak_vram(m, device, batch_size) for m in models}

# Normalize latency relative to fastest model on each device
fastest = min(latencies.values())
relative_latency = {m: l / fastest for m, l in latencies.items()}
```

## Common pitfalls

- Relative latency is normalized per device, making cross-device comparisons non-trivial.
- Mobile inference latency varies drastically across accelerators (CPU, GPU, ANE) and is less consistent than desktop.
- Swin and TSM are excluded from mobile benchmarks due to lack of mobile-compatible implementations.

## Evidence (verbatim from paper)

> Instead, we measure the latency of all models on 5 desktop GPUs and 5 mobile devices. ... We use mixed precision and take the mean latency over 100 runs, excluding 10 warm-up runs. Because each GPU yields different speeds, we report relative latency only, where the fastest model has a latency of 1.0. ... Besides latency, the VRAM requirement of a model is one of the biggest constraints during training... Table 2 gives an overview of how much VRAM each model requires.

## Citation

```bibtex
@misc{koot2021evaluatingtransformers,
  title={Evaluating Transformers for Lightweight Action Recognition},
  author={Koot et al. (2021)},
  year={2021},
  note={arXiv:2111.09641}
}
```

- arXiv: 2111.09641