Contextiq Retrieval Eval

---
name: contextiq-retrieval-eval
description: Evaluates zero-shot text-to-video retrieval for contextual advertising. It probes a model's ability to rank relevant video content based on natural language queries using multimodal signals (vision, audio, captions, metadata). Use when the user wants to benchmark on Val-1, Val-2, or asks about evaluating this task. Reports P@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.22233
  bibtex_key: chaubey2024contextiq
  confidence: high
---

# contextiq-retrieval-eval

> ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising — Chaubey et al. (2024) (arXiv:2410.22233, 2024)

## What this evaluates

Evaluates zero-shot text-to-video retrieval for contextual advertising. It probes a model's ability to rank relevant video content based on natural language queries using multimodal signals (vision, audio, captions, metadata).

## Datasets

- **Val-1** — total ?; splits: val (-1)
- **Val-2** — total 2000; splits: val (2000)

## Metrics

- `P@K` **(primary)** — range: [0, 1]
  - Precision at K: (Number of relevant videos in the top K retrieved results) / K. Evaluated at K ∈ {5, 10, 15, 20, 25, 30, 35, 40, 45, 50}.
- `Δ_avg` — range: [0, 1]
  - Average precision delta across K values: (1/|K|) * Σ_{K∈{5..30}} |P_{V,K} - P_{V+X,K}|, measuring the performance gain of adding a modality X to vision-only.

## Input / output format

**Input**: Natural language text query and video content (or video scenes/clips).

**Output**: Ranked list of retrieved videos/scenes.

## Scoring recipe

```python
# Automated P@K on Val-1
relevant = [v for v in top_k if v in ground_truth]
p_at_k = len(relevant) / k

# Manual validation for Val-2 / Condensed Movies / MSR-VTT
# 3 annotators independently label top-K results as relevant/irrelevant.
final_label = majority_vote(annotator_labels)
p_at_k = sum(final_label) / k
```

## Common pitfalls

- Manual validation relies on a 3-annotator voting system, which may introduce subjectivity or inconsistency for ambiguous queries.
- Internal dataset (Val-2) lacks ground truth, requiring manual annotation for every query, which limits scalability and reproducibility.
- Precision is only reported up to K=50, ignoring long-tail retrieval performance.

## Evidence (verbatim from paper)

> We use a voting-based system among the annotators to compile the results of our validation. ... Tab. [3] shows the performance of the proposed approach as compared to different approaches on our curated dataset (Val-1 as described in Sec. [4.3]). ... For each query, we search and retrieve the top 30 videos from the dataset. ... The top 30 videos from both methods are then annotated for correctness by 3 annotators. Table 3: Performance comparison on the curated dataset (Val-1) for ContextIQ... P@5 | P@10 | P@15 | P@20 | P@25 | P@30 | P@35 | P@40 | P@45 | P@50

## Citation

```bibtex
@misc{chaubey2024contextiq,
  title={ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising},
  author={Chaubey et al. (2024)},
  year={2024},
  note={arXiv:2410.22233}
}
```

- arXiv: 2410.22233