Lovr Eval

---
name: lovr-eval
description: Evaluates a model's ability to retrieve relevant long-form videos or fine-grained clips based on rich, narrative-driven text queries. It probes temporal reasoning, semantic alignment across extended durations, and robustness to long-context inputs and varying frame sampling strategies. Use when the user wants to benchmark on LoVR, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.13928
  bibtex_key: cai2025lovr
  confidence: high
---

# lovr-eval

> LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts — Cai et al. (2025) (arXiv:2505.13928, 2025)

## What this evaluates

Evaluates a model's ability to retrieve relevant long-form videos or fine-grained clips based on rich, narrative-driven text queries. It probes temporal reasoning, semantic alignment across extended durations, and robustness to long-context inputs and varying frame sampling strategies.

## Datasets

- **LoVR** — total 467; splits: test (-1); repo https://github.com/TechNomad-ds/LoVR-benchmark

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Proportion of true relevant items among the top-K retrieved results. Computed for K=1, 5, and 10. Averaged across all query-video/clip pairs.
- `Retrieval Time` — range: ms
  - End-to-end latency measured in milliseconds. Reported under two configurations: with I/O (including disk read/write and network transmission) and without I/O (pure computation like vector similarity calculation and ranking).

## Input / output format

**Input**: A video (full-length or fine-grained clip) represented as a sequence of sampled frames, paired with a text query/caption.

**Output**: A ranked list of video or clip identifiers sorted by similarity score to the text query.

## Scoring recipe

```python
def compute_recall_at_k(retrieved_ids, ground_truth_id, k):
    top_k = retrieved_ids[:k]
    return 1.0 if ground_truth_id in top_k else 0.0

def compute_recall_at_k_avg(retrieved_ids_list, ground_truth_ids_list, k):
    scores = [compute_recall_at_k(r, g, k) for r, g in zip(retrieved_ids_list, ground_truth_ids_list)]
    return sum(scores) / len(scores) if scores else 0.0
```

## Common pitfalls

- Using image encoders without averaging frame-level embeddings to produce a single video-level representation before similarity computation.
- Confusing 'Time w/ IO' (real-world deployment bottleneck) with 'Time w/o IO' (pure algorithmic computation potential).
- Assuming increasing sampled frames linearly improves accuracy; the benchmark shows diminishing returns or added noise beyond 10-20 frames.
- Failing to process long query contexts end-to-end, causing models to ignore critical details at the end of the prompt.

## Evidence (verbatim from paper)

> This metric is measured using Recall@K, which quantifies the proportion of true relevant items among the top-K retrieved results and reflects the recall capability of the retrieval model.

## Citation

```bibtex
@misc{cai2025lovr,
  title={LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2505.13928}
}
```

- arXiv: 2505.13928