Clip Mteb Eval

---
name: clip-mteb-eval
description: Evaluates cross-modal (text-image) retrieval and text-only embedding performance. Probes zero-shot retrieval accuracy, semantic similarity, and overall text embedding capability across diverse benchmarks. Use when the user wants to benchmark on CLIP Benchmark, MTEB, or asks about evaluating this task. Reports r@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.20204
  bibtex_key: koukounas2024jinaclip
  confidence: high
---

# clip-mteb-eval

> Jina CLIP: Your CLIP Model Is Also Your Text Retriever — Koukounas et al. (2024) (arXiv:2405.20204, 2024)

## What this evaluates

Evaluates cross-modal (text-image) retrieval and text-only embedding performance. Probes zero-shot retrieval accuracy, semantic similarity, and overall text embedding capability across diverse benchmarks.

## Datasets

- **CLIP Benchmark** — total ?; splits: test (-1)
- **MTEB** — total ?; splits: test (-1)

## Metrics

- `r@5` **(primary)** — range: percent
  - Recall@5: fraction of queries where the ground-truth relevant item appears in the top-5 retrieved results. Averaged across all queries.
- `ndcg@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by weighting relevant items higher at top positions.
- `spearman` — range: [-1, 1]
  - Spearman rank correlation coefficient between predicted similarity scores and human-annotated relevance scores.
- `Avg MTEB Score` — range: percent
  - Macro-average of task-specific scores (normalized to 0-100) across the 8 MTEB tasks.

## Input / output format

**Input**: Text queries paired with image captions/descriptions for cross-modal retrieval; text queries and documents for text-only retrieval; text pairs for semantic textual similarity (STS).

**Output**: Ranked list of retrieved items (images or documents) or similarity scores for text pairs.

## Scoring recipe

```python
def recall_at_k(retrieved, relevant, k=5):
    if not relevant: return 0.0
    hits = sum(1 for item in retrieved[:k] if item in relevant)
    return hits / len(relevant)

def ndcg_at_k(retrieved, relevant, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(retrieved[:k]) if item in relevant)
    ideal = sorted([1.0 / math.log2(i + 2) for i in range(min(len(relevant), k))], reverse=True)
    idcg = sum(ideal)
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Recall@5 is computed per query and then averaged across the dataset, not calculated globally over all retrieved items.
- MTEB scores are task-specific averages normalized to [0,100] before computing the overall MTEB average; raw correlation values must be transformed first.
- Cross-modal retrieval is evaluated in a zero-shot setting without fine-tuning on the benchmark datasets.

## Evidence (verbatim from paper)

> txt-img r@5 : Text to Image Recall@5 [%] 
 img-txt r@5 : Image to Text Recall@5 [%] 
 r@5 : Recall@5 [%]  
 spearman: Spearman Correlation

For zero-shot image-text and text-image information retrieval, we evaluate using Flickr8k, Flickr30K and MSCOCO Captions, which are all included in CLIP Benchmark. [jina-clip-v1] achieves an average Recall@5 of 85.8% across all retrieval benchmarks, outperforming OpenAI’s CLIP model and performing on par with EVA-CLIP.

## Citation

```bibtex
@misc{koukounas2024jinaclip,
  title={Jina CLIP: Your CLIP Model Is Also Your Text Retriever},
  author={Koukounas et al. (2024)},
  year={2024},
  note={arXiv:2405.20204}
}
```

- arXiv: 2405.20204