Long Video Understanding Eval

---
name: long-video-understanding-eval
description: Evaluates a model's ability to perform temporal reasoning and question-answering on long-duration videos (several minutes to over an hour). It probes memory retention, attention allocation across extended sequences, and the capacity to filter irrelevant visual content while preserving critical frames. Use when the user wants to benchmark on LongVideoBench, MLVU, VideoMME (Long), LVBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14149
  bibtex_key: zhang2026xcomp
  confidence: high
---

# long-video-understanding-eval

> One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding — Zhang et al. (2026) (arXiv:2604.14149, 2026)

## What this evaluates

Evaluates a model's ability to perform temporal reasoning and question-answering on long-duration videos (several minutes to over an hour). It probes memory retention, attention allocation across extended sequences, and the capacity to filter irrelevant visual content while preserving critical frames.

## Datasets

- **LongVideoBench** — total ?; splits: test (-1)
- **MLVU** — total ?; splits: test (-1)
- **VideoMME (Long)** — total ?; splits: test (-1)
- **LVBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark. Calculated as (correct predictions / total questions) × 100.

## Input / output format

**Input**: Long video sequences (segmented into clips/frames) paired with a natural language question. For VideoMME, multiple-choice answer candidates are provided. Subtitles are excluded.

**Output**: A single predicted answer string or selected option corresponding to the question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Position bias ('lost-in-the-middle') in long sequences can cause models to ignore mid-video content unless segmented local attention is used.
- VideoMME evaluation explicitly excludes subtitles; including them inflates performance and violates the benchmark protocol.
- Frame count limits vary by video duration (e.g., 512 vs 2048 frames), which directly impacts token compression ratios and memory constraints during inference.

## Evidence (verbatim from paper)

> These benchmarks adopt a question-answering format, primarily using accuracy as the evaluation metric.

## Citation

```bibtex
@misc{zhang2026xcomp,
  title={One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.14149}
}
```

- arXiv: 2604.14149