Lvbench Eval

---
name: lvbench-eval
description: This benchmark evaluates multimodal models' ability to comprehend extreme-length videos (averaging ~70 minutes) by testing six core temporal understanding capabilities. It probes long-term memory, multi-hop reasoning, and instruction-following across diverse video categories like sports, documentaries, and TV shows. Use when the user wants to benchmark on LVBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.08035
  bibtex_key: wang2024lvbench
  confidence: high
---

# lvbench-eval

> LVBench: An Extreme Long Video Understanding Benchmark — Wang et al. (2024) (arXiv:2406.08035, 2024)

## What this evaluates

This benchmark evaluates multimodal models' ability to comprehend extreme-length videos (averaging ~70 minutes) by testing six core temporal understanding capabilities. It probes long-term memory, multi-hop reasoning, and instruction-following across diverse video categories like sports, documentaries, and TV shows.

## Datasets

- **LVBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions, calculated as (correct predictions / total questions) × 100.

## Input / output format

**Input**: Video frames sampled at 1 FPS (or fixed counts like 32/96) concatenated with a text prompt containing the question and four multiple-choice options (A-D), followed by the instruction: 'Please select the best answer from the options above and directly provide the letter representing your choice without giving any explanation.'

**Output**: A single uppercase letter (A, B, C, or D) corresponding to the selected option.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_letter = extract_answer_letter(pred)
        if pred_letter == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Models frequently violate the strict output constraint, generating phrases like 'None of the above' instead of the required letter, necessitating robust extraction pipelines.
- Performance is highly sensitive to frame sampling density; sparse sampling yields near-random results, while 1 FPS is required to capture transient visual cues.
- Certain models exhibit strong response biases (e.g., defaulting to option A), which can artificially inflate accuracy if not accounted for during evaluation.

## Evidence (verbatim from paper)

> humans achieve a very high accuracy of 94.4% on average, setting a strong benchmark across all categories. In contrast, the overall performance of Gemini-2.5-Pro and Seed1.5-VL was considerably lower, at 67.4% and 64.0%, respectively.

## Citation

```bibtex
@misc{wang2024lvbench,
  title={LVBench: An Extreme Long Video Understanding Benchmark},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08035}
}
```

- arXiv: 2406.08035