Long Video Qa Eval

---
name: long-video-qa-eval
description: Evaluates long-form video understanding and multimodal reasoning capabilities across multiple-choice question answering tasks. It probes the model's ability to handle extended temporal dependencies, spatial-temporal reasoning, and tool-augmented retrieval in videos ranging from short clips to hour-long content. Use when the user wants to benchmark on LongVideoBench, VideoMME, LVBench, MLVU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15724
  bibtex_key: [REDACTED]
  confidence: high
---

# long-video-qa-eval

> VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning — Chenglin Li et al. (2026) (arXiv:2601.15724, 2026)

## What this evaluates

Evaluates long-form video understanding and multimodal reasoning capabilities across multiple-choice question answering tasks. It probes the model's ability to handle extended temporal dependencies, spatial-temporal reasoning, and tool-augmented retrieval in videos ranging from short clips to hour-long content.

## Datasets

- **LongVideoBench** — total 6678; splits: val (6678)
- **VideoMME** — total 2700; splits: test_long (-1)
- **LVBench** — total ?; splits: test (-1)
- **MLVU** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard multiple-choice accuracy: the proportion of questions where the model's predicted answer exactly matches the ground-truth option. Calculated as (correct predictions / total questions) * 100.

## Input / output format

**Input**: Video clips (up to 64 frames sampled, max 32,768 pixels per frame), accompanying subtitles, and audio. Paired with multiple-choice questions.

**Output**: Model generates a predicted answer option (or text) and a confidence score. For evaluation, the predicted option is compared against the ground-truth choice.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Performance heavily depends on the confidence threshold ($\tau$) and number of sampled frames ($n$); default $\tau=0.7$ is used but ablation shows optimal $\tau$ varies by dataset.
- Retrieval Top-$k$ settings are dataset-specific (e.g., Top-3 for LongVideoBench, Top-10 for VideoMME), so fixed retrieval parameters will hurt performance.
- Audio subtitles for LVBench and MLVU must be generated via FFmpeg and Whisper, as they are not natively provided in the benchmark splits.

## Evidence (verbatim from paper)

> As shown in Figure[6], VideoThinker performs on par with Qwen2.5-VL when handling short videos (under 2 minutes). However, its advantage becomes evident as the video length increases. For medium videos (between 2 and 15 minutes), VideoThinker achieves consistently higher accuracy on both VideoMME and LongVideoBench, showing stronger capability in modeling longer temporal dependencies.

## Citation

```bibtex
@misc{li2026videothinker,
  title={VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning},
  author={Chenglin Li et al. (2026)},
  year={2026},
  note={arXiv:2601.15724}
}
```

- arXiv: 2601.15724