Textvr Retrieval Eval

---
name: textvr-retrieval-eval
description: Evaluates cross-modal video retrieval models that must jointly process visual context and scene text (OCR tokens) to match sentence queries with relevant videos. Probes the model's ability to read, comprehend, and align fine-grained text semantics with visual frames in real-world scenarios. Use when the user wants to benchmark on TextVR, or asks about evaluating this task. Reports R@K (Recall@K).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.03347
  bibtex_key: wu2023textvr
  confidence: high
---

# textvr-retrieval-eval

> A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension — Wu et al. (2023) (arXiv:2305.03347, 2023)

## What this evaluates

Evaluates cross-modal video retrieval models that must jointly process visual context and scene text (OCR tokens) to match sentence queries with relevant videos. Probes the model's ability to read, comprehend, and align fine-grained text semantics with visual frames in real-world scenarios.

## Datasets

- **TextVR** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/callsys/TextVR

## Metrics

- `R@K (Recall@K)` **(primary)** — range: [0, 1]
  - Proportion of queries where the ground-truth video appears in the top-K retrieved results (K=1, 5, 10). Higher is better.
- `MdR (Median Rank)` — range: rank
  - Median of the ranks of the ground-truth videos across all queries. Lower is better.
- `MnR (Mean Rank)` — range: rank
  - Arithmetic mean of the ranks of the ground-truth videos across all queries. Lower is better.

## Input / output format

**Input**: A sentence query, a video (represented as visual features), and scene text/OCR tokens extracted from the video frames.

**Output**: A ranked list of videos for each query (Language-to-Video) or a ranked list of queries for each video (Video-to-Language).

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    ranks = []
    for pred, gold in zip(predictions, golds):
        if gold in pred:
            ranks.append(pred.index(gold) + 1)
        else:
            ranks.append(len(pred) + 1)
    r1 = sum(1 for r in ranks if r <= 1) / len(ranks)
    r5 = sum(1 for r in ranks if r <= 5) / len(ranks)
    r10 = sum(1 for r in ranks if r <= 10) / len(ranks)
    md_r = sorted(ranks)[len(ranks)//2]
    mn_r = sum(ranks) / len(ranks)
    return {'R@1': r1, 'R@5': r5, 'R@10': r10, 'MdR': md_r, 'MnR': mn_r}
```

## Common pitfalls

- Confusing scene text (OCR tokens) with subtitles; the dataset specifically evaluates reading comprehension of in-video text, not closed captions.
- High false negatives in OCR detection can severely penalize retrieval performance, as missing relevant text tokens removes crucial semantic signals.
- Mean Rank (MnR) is less sensitive to top-1 improvements than R@1 or MdR due to its susceptibility to long-tail rank distributions.

## Evidence (verbatim from paper)

> Evaluation Metric Following previous video retrieval benchmarks, we adopt the average recall at K(R@K), median rank (MdR), and mean rank (MnR) over all queries as the metric. We consider a prediction correct if the predicted video matches the ground-truth video. Generally, the higher R@K and lower MdR, MnR show better performance.

## Citation

```bibtex
@misc{wu2023textvr,
  title={A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension},
  author={Wu et al. (2023)},
  year={2023},
  note={arXiv:2305.03347}
}
```

- arXiv: 2305.03347