Lvsum Eval

---
name: lvsum-eval
description: This benchmark evaluates multimodal large language models' ability to perform timestamp-aware summarization of long videos. It probes temporal grounding, instruction adherence regarding length constraints, and cross-modal consistency between visual/audio content and generated text descriptions. Use when the user wants to benchmark on LVSum, or asks about evaluating this task. Reports Kendall's tau & Spearman's rho.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10024
  bibtex_key: patel2026lvsum
  confidence: high
---

# lvsum-eval

> LVSum: A Benchmark for Timestamp-Aware Long Video Summarization — Patel et al. (2026) (arXiv:2604.10024, 2026)

## What this evaluates

This benchmark evaluates multimodal large language models' ability to perform timestamp-aware summarization of long videos. It probes temporal grounding, instruction adherence regarding length constraints, and cross-modal consistency between visual/audio content and generated text descriptions.

## Datasets

- **LVSum** — total 72; splits: test (72)

## Metrics

- `Kendall's tau & Spearman's rho` **(primary)** — range: [-1, 1]
  - Rank correlation coefficients measuring the correspondence between predicted and ground-truth importance rankings at second-level granularity.
- `Content Relevance (CR)` — range: other
  - LLM-as-judge metric scoring semantic alignment between the generated summary and ground-truth key moments on a 1-5 scale.
- `Modality Coherence (MC)` — range: other
  - LLM-as-judge metric scoring cross-modal consistency (visual, auditory, textual alignment) for each predicted segment on a 1-5 scale.

## Input / output format

**Input**: 96 uniformly sampled video frames, a timestamped transcript of spoken narration/dialogue, and a system prompt instructing the model to generate a summary covering ~15% of the video duration.

**Output**: A list of timestamped segments with textual descriptions, constrained to approximately 15% of the total video length.

## Scoring recipe

```python
# 1. Temporal Grounding
pred_ranks = rank_segments_by_importance(predictions)
gold_ranks = rank_segments_by_importance(gold)
tau = kendalltau(pred_ranks, gold_ranks)
rho = spearmanr(pred_ranks, gold_ranks)

# 2. LLM-as-Judge Metrics (CR & MC)
cr_scores, mc_scores = [], []
for seg in predictions:
    visual_audio = extract_media(seg.start, seg.end)
    judge_output = llm_judge_prompt(seg.text, visual_audio, gold)
    cr_scores.append(parse_score(judge_output, scale=5))
    mc_scores.append(parse_score(judge_output, scale=5))
avg_cr = mean(cr_scores)
avg_mc = mean(mc_scores)
```

## Common pitfalls

- Models frequently exceed the 15% duration budget despite explicit instructions, skewing temporal ranking metrics.
- Using frame-level granularity for importance scoring instead of second-level granularity misaligns with MLLM timestamp precision.
- LLM-as-judge metrics depend heavily on the specific judge model and prompt templates, which may introduce scoring bias.

## Evidence (verbatim from paper)

> In practice, current MLLMs are prone to producing summary segments whose accumulated duration exceeds the target 15% budget, even when explicitly instructed with a length constraint. Moreover,*[otani2019rethinking]* showed that Kendall’s $	au$ and Spearman’s $
ho$ are more robust than F1-based metrics when segment lengths vary. For these reasons, we use Kendall’s $	au$ and Spearman’s $
ho$ as our primary metrics and report the average predicted summary length to contextualize metric values.

## Citation

```bibtex
@misc{patel2026lvsum,
  title={LVSum: A Benchmark for Timestamp-Aware Long Video Summarization},
  author={Patel et al. (2026)},
  year={2026},
  note={arXiv:2604.10024}
}
```

- arXiv: 2604.10024