Mteb Loco Jina Eval

---
name: mteb-loco-jina-eval
description: Evaluates text embedding models on short and long-context retrieval, clustering, and semantic similarity tasks to measure representation quality across varying sequence lengths. Use when the user wants to benchmark on MTEB, Jina Long Context Benchmark, LoCo Benchmark, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.01613
  bibtex_key: nussbaum2024nomicembed
  confidence: high
---

# mteb-loco-jina-eval

> Nomic Embed: Training a Reproducible Long Context Text Embedder — Nussbaum et al. (2024) (arXiv:2402.01613, 2024)

## What this evaluates

Evaluates text embedding models on short and long-context retrieval, clustering, and semantic similarity tasks to measure representation quality across varying sequence lengths.

## Datasets

- **MTEB** — total ?; splits: test (-1)
- **Jina Long Context Benchmark** — total ?; splits: test (-1)
- **LoCo Benchmark** — total ?; splits: test (-1)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring retrieval effectiveness by weighting relevant documents by their graded relevance position.
- `V-score` — range: [0, 1]
  - Clustering performance metric reported for the Jina Long Context Benchmark, typically based on adjusted mutual information or similar clustering quality measures.
- `Spearman's Tau` — range: [-1, 1]
  - Rank correlation coefficient used to evaluate long-context retrieval tasks (Tau Scr., Tau Gov., Tau QMS.) and overall performance averages.

## Input / output format

**Input**: Text strings (queries and documents) or single texts for clustering/STS tasks, with task-specific prefixes added (e.g., 'search_query', 'clustering').

**Output**: L2-normalized embedding vectors (except for Classification tasks where raw embeddings are used without normalization).

## Scoring recipe

```python
def compute_metrics(embeddings, labels, task_type):
    if task_type == 'retrieval':
        scores = cosine_similarity(query_emb, doc_emb)
        return ndcg_at_k(scores, labels, k=10)
    elif task_type == 'clustering':
        return v_score(cluster_labels, true_labels)
    elif task_type == 'long_context':
        return spearman_tau(predicted_ranks, true_ranks)
    elif task_type == 'classification':
        return accuracy(predicted_classes, true_labels)
    return 0
```

## Common pitfalls

- Truncating all texts to 512 tokens for MTEB evaluation regardless of model context length.
- Adding task-specific prefixes (e.g., 'search_query', 'clustering') to inputs, which varies by task type.
- Skipping L2 normalization only for Classification tasks, while applying it to all other tasks.
- Using Dynamic NTK Interpolation (alpha=2) for sequences exceeding the 2048 training length.

## Evidence (verbatim from paper)

> Similar to Günther et al. ([2024]), we report the V-scores and NDCG@10 for the clustering and retrieval datasets respectively. We evaluate all models at sequence length 128, 512, and 8191.

## Citation

```bibtex
@misc{nussbaum2024nomicembed,
  title={Nomic Embed: Training a Reproducible Long Context Text Embedder},
  author={Nussbaum et al. (2024)},
  year={2024},
  note={arXiv:2402.01613}
}
```

- arXiv: 2402.01613