Mteb English Eval

---
name: mteb-english-eval
description: Evaluates the ability of decoder-only LLMs to generate universal text embeddings across diverse natural language processing tasks. It probes retrieval, reranking, clustering, classification, pair classification, semantic textual similarity, and summarization capabilities using standardized benchmark datasets. Use when the user wants to benchmark on MTEB (English subset), or asks about evaluating this task. Reports Average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.23386
  bibtex_key: lin2025causal2vec
  confidence: high
---

# mteb-english-eval

> Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models — Lin et al. (2025) (arXiv:2507.23386, 2025)

## What this evaluates

Evaluates the ability of decoder-only LLMs to generate universal text embeddings across diverse natural language processing tasks. It probes retrieval, reranking, clustering, classification, pair classification, semantic textual similarity, and summarization capabilities using standardized benchmark datasets.

## Datasets

- **MTEB (English subset)** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` — range: [0, 1]
  - Normalized discounted cumulative gain at rank 10, used for retrieval tasks.
- `MAP` — range: [0, 1]
  - Mean Average Precision, used for reranking tasks.
- `V-measure` — range: [0, 1]
  - V-measure for clustering tasks, measuring homogeneity and completeness.
- `AP` — range: [0, 1]
  - Average Precision for pair classification tasks.
- `Accuracy` — range: [0, 1]
  - Classification accuracy for classification tasks.
- `Spearman correlation` — range: [-1, 1]
  - Spearman rank correlation for STS and summarization tasks.
- `Average score` **(primary)** — range: [0, 1]
  - Macro-average of task-specific metrics across all 56 datasets.

## Input / output format

**Input**: Text inputs are first processed by a BERT-style encoder to generate a single Contextual token, which is prepended to the LLM input sequence. The model receives the concatenated sequence of the Contextual token and the original text tokens.

**Output**: The model outputs hidden states for all tokens. The final text embedding is constructed by concatenating the last hidden states of the Contextual token and the EOS token.

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type == 'Retrieval': return ndcg_at_k(gold, predictions, k=10)
    elif task_type == 'Reranking': return map_score(gold, predictions)
    elif task_type == 'Clustering': return v_measure(gold, predictions)
    elif task_type == 'PairClassification': return ap_score(gold, predictions)
    elif task_type == 'Classification': return accuracy(gold, predictions)
    elif task_type in ['STS', 'Summarization']: return spearman_corr(gold, predictions)
    return None
```

## Common pitfalls

- Training on non-public or synthetic data inflates MTEB scores but harms generalization to unseen non-retrieval tasks.
- Using standard last-token pooling introduces recency bias; the protocol explicitly requires concatenating Contextual and EOS token states to mitigate this.
- Comparing against methods that use bidirectional attention or in-context learning examples without accounting for increased sequence length and computational overhead.

## Evidence (verbatim from paper)

> The main evaluation metrics are nDCG@10, MAP, V-measure (V-meas.), average precision (AP), accuracy (Acc.), and Spearman correlation (Spear., both for STS and Summ.), respectively.

## Citation

```bibtex
@misc{lin2025causal2vec,
  title={Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2507.23386}
}
```

- arXiv: 2507.23386