Jina Embeddings V4 Eval

---
name: jina-embeddings-v4-eval
description: Evaluates a multimodal embedding model's ability to retrieve relevant documents, images, and code from large corpora, and to measure semantic similarity between text pairs across multiple languages and modalities. Use when the user wants to benchmark on J-VDR, ViDoRe, CLIPB, MMTEB, MTEB-en, COIR, LEMB, STS-m, STS-en, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.18902
  bibtex_key: gunther2025jinaembeddingsv4
  confidence: high
---

# jina-embeddings-v4-eval

> jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval — Günther et al. (2025) (arXiv:2506.18902, 2025)

## What this evaluates

Evaluates a multimodal embedding model's ability to retrieve relevant documents, images, and code from large corpora, and to measure semantic similarity between text pairs across multiple languages and modalities.

## Datasets

- **J-VDR** — total ?; splits: test (-1)
- **ViDoRe** — total ?; splits: test (-1)
- **CLIPB** — total ?; splits: test (-1); repo https://github.com/LAION-AI/CLIP_benchmark
- **MMTEB** — total ?; splits: test (-1)
- **MTEB-en** — total ?; splits: test (-1)
- **COIR** — total ?; splits: test (-1)
- **LEMB** — total ?; splits: test (-1)
- **STS-m** — total ?; splits: test (-1)
- **STS-en** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K=10. Computes the weighted sum of relevance scores for retrieved documents, normalized by the ideal DCG. Higher is better.
- `nDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K=5. Same as nDCG@10 but truncated at the top 5 retrieved results.
- `Spearman coefficient` — range: [-1, 1]
  - Spearman rank correlation coefficient measuring the monotonic relationship between predicted similarity scores and ground-truth human judgments.

## Input / output format

**Input**: Query and document pairs (text, images, or code) passed through the embedding model to generate vector representations.

**Output**: Ranked list of documents for retrieval tasks; cosine similarity scores for STS tasks.

## Scoring recipe

```python
def compute_ndcg_at_k(relevant_docs, retrieved_docs, k):
    dcg = sum(1 / math.log2(i + 2) for i, doc in enumerate(retrieved_docs[:k]) if doc in relevant_docs)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(k, len(relevant_docs))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_spearman(gold_scores, pred_scores):
    return scipy.stats.spearmanr(gold_scores, pred_scores).statistic

# For J-VDR/ViDoRE: average multilingual task scores first, then average across all tasks.
```

## Common pitfalls

- Different benchmarks use different cutoffs: nDCG@5 for J-VDR, ViDoRe, and CLIPB, but nDCG@10 for MMTEB, MTEB-en, COIR, and LEMB.
- J-VDR and ViDoRE require a two-step averaging process: calculate the average for multilingual tasks first, then average that result across all tasks.
- Specific tasks like ArguAna require a fixed query prefix ('Given a claim, find documents that refute the claim') to match the official evaluation protocol.

## Evidence (verbatim from paper)

> Scores are nDCG@5 for J-VDR, ViDoRe, and CLIPB, and nDCG@10 for MMTEB, MTEB-en, COIR, and LEMB, and Spearman coefficient for STS-m and STS-en. Average Calculation: For J-VDR and ViDoRE, we calculate the average for the multilingual tasks first and consider this as a single score before calculating the average across all tasks.

## Citation

```bibtex
@misc{gunther2025jinaembeddingsv4,
  title={jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval},
  author={Günther et al. (2025)},
  year={2025},
  note={arXiv:2506.18902}
}
```

- arXiv: 2506.18902