H2vu Benchmark Eval

---
name: h2vu-benchmark-eval
description: Evaluates multimodal large language models on hierarchical and holistic video understanding, specifically probing temporal reasoning, countercommonsense comprehension, trajectory state tracking, and first-person streaming video analysis. Use when the user wants to benchmark on H²VU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.24008
  bibtex_key: wu2025h2vubenchmark
  confidence: high
---

# h2vu-benchmark-eval

> H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding — Qi Wu et al. (2025) (arXiv:2503.24008, 2025)

## What this evaluates

Evaluates multimodal large language models on hierarchical and holistic video understanding, specifically probing temporal reasoning, countercommonsense comprehension, trajectory state tracking, and first-person streaming video analysis.

## Datasets

- **H²VU** — total ?; splits: test (-1); repo https://github.com/siriusrecco/H2VU-BenchMark

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed by comparing the model's generated answer against the ground truth label for each question. A match yields 1, otherwise 0. The final score is the average across all instances.

## Input / output format

**Input**: A video clip (processed via uniform frame sampling or fixed frame-rate sampling) paired with a natural language question or instruction.

**Output**: A natural language text response containing the model's predicted answer.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, ground_truths):
    if normalize(pred) == normalize(gold):
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Sparse frame sampling (e.g., 32 frames) often misses fine-grained motion dynamics and inter-frame dependencies, unfairly penalizing models on temporal tracking tasks.
- Models frequently over-rely on textual priors or real-world assumptions rather than observing actual visual cues, leading to systematic errors in countercommonsense tasks.
- Performance drops significantly in online/streaming settings compared to offline videos for models not explicitly optimized for continuous frame processing.

## Evidence (verbatim from paper)

> We utilized accuracy as the evaluation metric, determined by comparing the model outputs with the ground truth, without involving any third-party models.

## Citation

```bibtex
@misc{wu2025h2vubenchmark,
  title={H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding},
  author={Qi Wu et al. (2025)},
  year={2025},
  note={arXiv:2503.24008}
}
```

- arXiv: 2503.24008