Longvideo Bench Eval

---
name: longvideo-bench-eval
description: This benchmark evaluates long-context video-language understanding by testing a model's ability to retrieve specific moments from lengthy videos and reason over multimodal details. It distinguishes between single-moment visual perception and multi-moment relational reasoning across 17 fine-grained categories. Use when the user wants to benchmark on LongVideoBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.15754
  bibtex_key: wu2024longvideobench
  confidence: medium
---

# longvideo-bench-eval

> LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding — Wu et al. (2024) (arXiv:2407.15754, 2024)

## What this evaluates

This benchmark evaluates long-context video-language understanding by testing a model's ability to retrieve specific moments from lengthy videos and reason over multimodal details. It distinguishes between single-moment visual perception and multi-moment relational reasoning across 17 fine-grained categories.

## Datasets

- **LongVideoBench** — total 6678; splits: test (6678)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of questions where the model's predicted option exactly matches the ground-truth correct answer.

## Input / output format

**Input**: A long video (up to 60 minutes) with interleaved frames and subtitles, a referring query that identifies specific moments, and a multiple-choice question requiring reasoning over those moments.

**Output**: The model must select the correct answer option from the provided multiple-choice list.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)
```

## Common pitfalls

- Models may exploit single-frame bias or short-context shortcuts instead of truly reasoning over long sequences.
- Confusion between L1 (single-moment perception) and L2 (multi-moment relational) tasks can skew category-level performance analysis.
- Subtitle-based referring queries may be answered via text-only pathways, bypassing actual video understanding.

## Evidence (verbatim from paper)

> We employ the multiple-choice question format, where several distracting options are provided alongside the correct answer option.

## Citation

```bibtex
@misc{wu2024longvideobench,
  title={LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2407.15754}
}
```

- arXiv: 2407.15754