Mteb Echo Eval

---
name: mteb-echo-eval
description: Evaluates the quality of text embeddings extracted from autoregressive language models in zero-shot and fine-tuned settings across a broad suite of downstream NLP tasks including classification, clustering, retrieval, and semantic textual similarity. Use when the user wants to benchmark on MTEB, or asks about evaluating this task. Reports Average MTEB Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.15449
  bibtex_key: springer2024repetition
  confidence: high
---

# mteb-echo-eval

> Repetition Improves Language Model Embeddings — Springer et al. (2024) (arXiv:2402.15449, 2024)

## What this evaluates

Evaluates the quality of text embeddings extracted from autoregressive language models in zero-shot and fine-tuned settings across a broad suite of downstream NLP tasks including classification, clustering, retrieval, and semantic textual similarity.

## Datasets

- **MTEB** — total 56; splits: test (-1)

## Metrics

- `Average MTEB Score` **(primary)** — range: [0, 100]
  - Task-specific metrics (e.g., accuracy, NDCG@10, Pearson correlation) computed per dataset, then averaged across all 56 datasets. Higher is better.

## Input / output format

**Input**: Text input $S$, optionally wrapped in a task-specific prompt (e.g., 'Rewrite the following paragraph: $S$. The rewritten paragraph: $S$' for zero-shot, or 'Query: $S$' / 'Document: $S$' for fine-tuning).

**Output**: Fixed-dimensional embedding vector extracted from the model's hidden states using mean pooling or last-token pooling.

## Scoring recipe

```python
def compute_mteb_score(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy(predictions, gold)
    elif task_type in ['reranking', 'retrieval']:
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'sts':
        return pearson_correlation(predictions, gold)
    # ... other task-specific metrics ...
    return average_across_datasets(task_scores)
```

## Common pitfalls

- Echo embeddings inherently require twice the inference compute of classical embeddings due to input repetition; failing to match compute (e.g., by halving input length) leads to unfair comparisons.
- Pooling strategy significantly impacts zero-shot performance (mean pooling is required), but last-token pooling often performs better in fine-tuned settings.
- Casting causal attention to bidirectional without fine-tuning typically harms performance, except for models like Mistral-7B with non-standard pretraining.

## Evidence (verbatim from paper)

> Our main evaluation dataset is the English-language subset of the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2022). MTEB is a collection of 56 datasets that are grouped into different embedding tasks: classification, clustering, pair classification, reranking, retrieval, sentence similarity (STS), and summarization, with the goal of evaluating embeddings broadly. ... Table 2: Role of scale and base model in the zero-shot setting: Average MTEB score (56 datasets) for LLaMA-7B-2-Instruct and S-LLaMA-1.3B.

## Citation

```bibtex
@misc{springer2024repetition,
  title={Repetition Improves Language Model Embeddings},
  author={Springer et al. (2024)},
  year={2024},
  note={arXiv:2402.15449}
}
```

- arXiv: 2402.15449