Pushupbench Eval

---
name: pushupbench-eval
description: Evaluates video-language models on long-form repetition counting and temporal reasoning. It probes whether models can accurately track state changes and count actions across extended video clips, revealing weaknesses in spatio-temporal tracking compared to supervised baselines. Use when the user wants to benchmark on PushupBench, or asks about evaluating this task. Reports Exact Match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23407
  bibtex_key: li2026pushupbench
  confidence: high
---

# pushupbench-eval

> PushupBench: Your VLM is not good at counting pushups — Li et al. (2026) (arXiv:2604.23407, 2026)

## What this evaluates

Evaluates video-language models on long-form repetition counting and temporal reasoning. It probes whether models can accurately track state changes and count actions across extended video clips, revealing weaknesses in spatio-temporal tracking compared to supervised baselines.

## Datasets

- **PushupBench** — total 446; splits: test (446), train (391)

## Metrics

- `Exact Match` **(primary)** — range: percent
  - Percentage of predictions that exactly equal the ground truth count.
- `MAE` — range: other
  - Mean absolute error between predicted and ground truth counts. Computed only on samples where |prediction - ground truth| ≤ 50 to exclude extreme outliers.
- `R^2` — range: other
  - Coefficient of determination measuring variance explained beyond the mean baseline. Computed as 1 - (SS_res / SS_tot) on samples with |error| ≤ 50. Negative values indicate constant or random output.

## Input / output format

**Input**: Video clips sampled at 5 fps and capped at 112 frames (uniformly spaced for longer videos), resized to 360p. Each clip is paired with one of 10 diverse prompt templates requesting the repetition count.

**Output**: A single integer representing the predicted number of repetitions.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    exact = sum(1 for p, gt in zip(predictions, ground_truths) if p == gt) / len(predictions)
    valid = [(p, gt) for p, gt in zip(predictions, ground_truths) if abs(p - gt) <= 50]
    mae = sum(abs(p - gt) for p, gt in valid) / len(valid) if valid else 0
    y = [gt for _, gt in valid]
    y_hat = [p for p, _ in valid]
    y_mean = sum(y) / len(y)
    ss_res = sum((yi - yhati)**2 for yi, yhati in zip(y, y_hat))
    ss_tot = sum((yi - y_mean)**2 for yi in y)
    r2 = 1 - (ss_res / ss_tot) if ss_tot != 0 else 0
    return exact, mae, r2
```

## Common pitfalls

- Ambiguous action boundaries in ~6.1% of samples require accepting multiple valid ground truth counts rather than a single rigid label.
- Extreme outliers from parsing failures or hallucinations drastically skew MAE and R^2, so the protocol explicitly excludes predictions with |error| > 50.
- Fast repetitions (<3 frames/rep at 5fps) become temporally unresolvable, causing noise in training and evaluation if sampling rates are not adjusted.

## Evidence (verbatim from paper)

> We report three metrics: Exact Match (percentage of predictions equaling ground truth), MAE (mean absolute error), and $R^{2}$ (coefficient of determination). For 27 samples (6.1%) with ambiguous action boundaries, we accept multiple valid ground truth counts (see Appendix[B](#A2 "Appendix B Annotation Process ‣ Your VLM is not good at counting pushups")). MAE and $R^{2}$ are computed excluding predictions with $|\text{error}|>50$, as extreme outliers from parsing failures or hallucinations can drastically skew these metrics.

## Citation

```bibtex
@misc{li2026pushupbench,
  title={PushupBench: Your VLM is not good at counting pushups},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.23407}
}
```

- arXiv: 2604.23407