Multivent2.0 Eval

---
name: multivent2.0-eval
description: Evaluates event-centric video retrieval across six languages, requiring models to match natural language queries about specific world events to relevant long-form videos using multimodal signals (vision, audio, OCR, metadata). It probes a model's ability to integrate cross-lingual, cross-modal information for complex event understanding rather than simple visual matching. Use when the user wants to benchmark on MultiVENT 2.0, or asks about evaluating this task. Reports Retrieval Performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.11619
  bibtex_key: [REDACTED]
  confidence: high
---

# multivent2.0-eval

> MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval — Kriz et al. (2024) (arXiv:2410.11619, 2024)

## What this evaluates

Evaluates event-centric video retrieval across six languages, requiring models to match natural language queries about specific world events to relevant long-form videos using multimodal signals (vision, audio, OCR, metadata). It probes a model's ability to integrate cross-lingual, cross-modal information for complex event understanding rather than simple visual matching.

## Datasets

- **MultiVENT 2.0** — total 218000; splits: test (-1); repo https://github.com/katesanders9/multiVENT

## Metrics

- `Retrieval Performance` **(primary)** — range: [0, 1]
  - Standard video retrieval recall metric. Computes the fraction of queries where the ground-truth video appears in the top-K ranked results. Values range from 0 to 1.

## Input / output format

**Input**: A video (optionally with human-written description) and a natural language query in one of six languages (Arabic, Chinese, English, Korean, Russian, Spanish).

**Output**: A ranked list of candidate videos or similarity scores for each query.

## Scoring recipe

```python
def compute_recall(predictions, gold, k_values=[1, 5, 10]):
    recalls = {}
    for k in k_values:
        hits = sum(1 for pred, gold_id in zip(predictions, gold) if gold_id in pred[:k])
        recalls[f'R@{k}'] = hits / len(predictions)
    return recalls
```

## Common pitfalls

- Videos are long-form, but most VLMs are trained on short segments, causing severe domain mismatch.
- Queries often target non-visual aspects (e.g., audio events, embedded text), making vision-only baselines fail.
- Performance varies significantly by language (Chinese/Korean are hardest) and video type (raw vs. professional).
- Human-written descriptions are not always available for raw/user-generated content, requiring models to handle missing metadata.

## Evidence (verbatim from paper)

> Table [3] presents the performance of these baselines on MultiVENT 2.0, as well as results on MSR-VTT, a standard benchmark video retrieval datasets. A key observation is that while VLMs excel on prior collections, most perform poorly on our task.

## Citation

```bibtex
@misc{kriz2024multivent2,
  title={MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval},
  author={Kriz et al. (2024)},
  year={2024},
  note={arXiv:2410.11619}
}
```

- arXiv: 2410.11619