Uvrb Eval

---
name: uvrb-eval
description: Evaluates zero-shot generalization of video embedding models across 16 diverse retrieval tasks and domains. It probes capabilities like spatial/temporal reasoning, compositional understanding, and partially relevant matching, revealing how well models generalize beyond standard benchmarks. Use when the user wants to benchmark on UVRB (Universal Video Retrieval Benchmark), or asks about evaluating this task. Reports Recall@1 (R@1).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.27571
  bibtex_key: guo2025towardsuniversalvideoretrieval
  confidence: high
---

# uvrb-eval

> Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum — Zhuoning Guo et al. (arXiv:2510.27571, 2025)

## What this evaluates

Evaluates zero-shot generalization of video embedding models across 16 diverse retrieval tasks and domains. It probes capabilities like spatial/temporal reasoning, compositional understanding, and partially relevant matching, revealing how well models generalize beyond standard benchmarks.

## Datasets

- **UVRB (Universal Video Retrieval Benchmark)** — total ?; splits: test (-1)

## Metrics

- `Recall@1 (R@1)` **(primary)** — range: [0, 1]
  - Fraction of queries where the ground-truth video ranks first among all candidates based on cosine similarity.
- `Recall@10 (R@10)` — range: [0, 1]
  - Fraction of queries where the ground-truth video ranks within the top 10 candidates. Used for challenging datasets with fuzzy queries.
- `Precision@1 (P@1)` — range: [0, 1]
  - Fraction of queries where at least one ground-truth video ranks first among candidates. Used for datasets with multiple positive matches.

## Input / output format

**Input**: Text query paired with a video uniformly sampled into exactly 8 raw visual frames. Audio, speech, and metadata are excluded.

**Output**: Normalized embedding vector (bf16 precision). Cosine similarity is computed between query and video embeddings for ranking.

## Scoring recipe

```python
def compute_recall_at_k(sim_matrix, labels, k=1):
    # sim_matrix: (N, N) cosine similarities (embeddings already L2-normalized)
    # labels: (N,) ground truth video indices for each query
    ranks = np.argsort(-sim_matrix, axis=1)
    top_k_indices = ranks[:, :k]
    hits = np.any(top_k_indices == labels, axis=1)
    return np.mean(hits)
```

## Common pitfalls

- Audio, speech, and metadata are strictly excluded; only raw visual frames are used.
- Each video is uniformly sampled to exactly 8 frames regardless of original length.
- Baseline models may have seen in-domain test data, creating an unfair advantage not present for the proposed GVE model.
- Frame resolution and token limits are adaptively adjusted per model to enforce a <200 visual tokens/frame constraint for MLLM-based models.

## Evidence (verbatim from paper)

> Our primary metric is Recall@1 (R@1), which measures if the most relevant item is the correct one. For more challenging datasets with fuzzy queries (e.g., CMRB and LoVR-TH), we choose to report Recall@10 (R@10). Additionally, we use Precision@1 (P@1) for the MS-TI and MS-TV with multiple positive candidates.

## Citation

```bibtex
@misc{guo2025towardsuniversalvideoretrieval,
  title={Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum},
  author={Zhuoning Guo et al.},
  year={2025},
  note={arXiv:2510.27571}
}
```

- arXiv: 2510.27571