Save Video Text Retrieval Eval

---
name: save-video-text-retrieval-eval
description: Evaluates a model's ability to retrieve relevant videos given a natural language query in an audio-visual setting. It specifically probes how well speech-aware representations and early vision-audio alignment improve cross-modal matching accuracy across diverse video-text benchmarks. Use when the user wants to benchmark on MSRVTT-9k, MSRVTT-7k, VATEX, Charades, LSMDC, or asks about evaluating this task. Reports SumR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.08224
  bibtex_key: zhao2026save
  confidence: high
---

# save-video-text-retrieval-eval

> SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval — Zhao et al. (2026) (arXiv:2603.08224, 2026)

## What this evaluates

Evaluates a model's ability to retrieve relevant videos given a natural language query in an audio-visual setting. It specifically probes how well speech-aware representations and early vision-audio alignment improve cross-modal matching accuracy across diverse video-text benchmarks.

## Datasets

- **MSRVTT-9k** — total 10000; splits: train (9000), test (1000)
- **MSRVTT-7k** — total 10000; splits: train (6513), val (497), test (2990)
- **VATEX** — total 28991; splits: train (25991), val (1500), test (1500)
- **Charades** — total 9848; splits: train (7985), test (1863)
- **LSMDC** — total 109487; splits: train (101079), val (7408), test (1000)

## Metrics

- `R@k (R1, R5, R10)` — range: percent
  - Recall at top-k: fraction of ground-truth videos retrieved within the top-k ranked results for each query.
- `SumR` **(primary)** — range: percent
  - Sum of R1, R5, and R10 scores (R1 + R5 + R10). Used as the overall performance score.
- `mR1` — range: percent
  - Mean R1 score averaged across all evaluated datasets.

## Input / output format

**Input**: A natural language query (caption) paired with a video containing visual frames and audio/speech transcripts.

**Output**: A ranked list of video candidates or similarity scores for each query, sorted by relevance.

## Scoring recipe

```python
def compute_metrics(predictions, gold_indices):
    recalls = {}
    for k in [1, 5, 10]:
        hits = sum(1 for pred in predictions if pred[:k] in gold_indices)
        recalls[f'R{k}'] = hits / len(gold_indices) * 100
    sumr = sum(recalls[f'R{k}'] for k in [1, 5, 10])
    return recalls, sumr
```

## Common pitfalls

- MSRVTT-9k and Charades lack validation splits, so the best checkpoint is selected based on peak test performance rather than validation R1.
- VATEX uses a specific partition from Chen et al. rather than the official default, requiring strict adherence to the cited split.
- SumR is reported as a sum of percentages (e.g., ~216), not an average, which can cause confusion when comparing against single R@k values.

## Evidence (verbatim from paper)

> Evaluation criteria. We report standard rank-based retrieval metrics, i.e. Recall at top $k$ (${1,5,10}$), and SumR (R1+R5+R10) as an overall score.

## Citation

```bibtex
@misc{zhao2026save,
  title={SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.08224}
}
```

- arXiv: 2603.08224