Gte Eval

---
name: gte-eval
description: Evaluates the cross-task generalization and retrieval quality of a general-purpose text embedding model across classification, retrieval, clustering, reranking, semantic similarity, summarization, and code search tasks. It measures how well zero-shot and unsupervised embeddings transfer to diverse downstream benchmarks without task-specific fine-tuning. Use when the user wants to benchmark on SST-2, BEIR, MTEB (English subset), CodeSearchNet, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.03281
  bibtex_key: li2023gte
  confidence: high
---

# gte-eval

> Towards General Text Embeddings with Multi-stage Contrastive Learning — Li et al. (2023) (arXiv:2308.03281, 2023)

## What this evaluates

Evaluates the cross-task generalization and retrieval quality of a general-purpose text embedding model across classification, retrieval, clustering, reranking, semantic similarity, summarization, and code search tasks. It measures how well zero-shot and unsupervised embeddings transfer to diverse downstream benchmarks without task-specific fine-tuning.

## Datasets

- **SST-2** — total ?; splits: test (-1)
- **BEIR** — total ?; splits: test (-1)
- **MTEB (English subset)** — total 56; splits: test (-1)
- **CodeSearchNet** — total ?; splits: dev+test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked retrieval results by penalizing relevant documents appearing lower in the list.
- `accuracy` — range: [0, 1]
  - Proportion of correctly classified instances in zero-shot text classification, computed by matching input embeddings to verbalized label embeddings.
- `v-measure` — range: [0, 1]
  - Harmonic mean of homogeneity and completeness, used for evaluating text clustering quality.
- `Spearman coefficient` — range: [-1, 1]
  - Rank correlation coefficient measuring the monotonic relationship between predicted similarity scores and human judgments for STS and reranking tasks.

## Input / output format

**Input**: Text pairs (query, document) or single texts for classification. For classification, inputs are paired with verbalized label texts (e.g., 'positive'/'negative' or fuzzy prompt templates). For retrieval, inputs are queries against a large candidate corpus.

**Output**: Embedding vectors (typically [CLS] token or mean pooling) used to compute similarity scores via dot product or cosine similarity. Final output is a ranked list of candidates or a classification label.

## Scoring recipe

```python
query_emb = model.encode(query)
candidate_embs = [model.encode(c) for c in candidates]
scores = [np.dot(query_emb, e) for e in candidate_embs]  # or cosine
ranked = np.argsort(scores)[::-1]
if task == 'classification':
  metric = 1.0 if ranked[0] == gold_idx else 0.0
else:
  metric = compute_ndcg(ranked, relevance_labels, k=10)
return mean(metric across instances)
```

## Common pitfalls

- Models use different similarity functions (dot product vs cosine) which drastically changes scores and breaks direct comparison.
- Comparing models of vastly different parameter sizes without normalizing for inference speed or computational cost.
- Confusing unsupervised pre-training results with supervised fine-tuning results when reading benchmark tables.

## Evidence (verbatim from paper)

> The evaluation metrics employed in MTEB are accuracy, v-measure, average precision, MAP, nDCG@10, and Spearman coefficients, respectively. ... We use BEIR (Thakur et al., 2021) as our evaluation benchmark for zero-shot unsupervised text retrieval.

## Citation

```bibtex
@misc{li2023gte,
  title={Towards General Text Embeddings with Multi-stage Contrastive Learning},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.03281}
}
```

- arXiv: 2308.03281