Mteb Eval

---
name: mteb-eval
description: This evaluation probes the ability of decoder-only LLMs to generate high-quality, fixed-dimensional text embeddings without fine-tuning. It measures semantic similarity, information retrieval, classification, clustering, and long-context comprehension across diverse tasks and sequence lengths. Use when the user wants to benchmark on MTEB, LoCoV1, or asks about evaluating this task. Reports MTEB Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.01046
  bibtex_key: [REDACTED]
  confidence: high
---

# mteb-eval

> KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs — Tang et al. (2026) (arXiv:2601.01046, 2026)

## What this evaluates

This evaluation probes the ability of decoder-only LLMs to generate high-quality, fixed-dimensional text embeddings without fine-tuning. It measures semantic similarity, information retrieval, classification, clustering, and long-context comprehension across diverse tasks and sequence lengths.

## Datasets

- **MTEB** — total ?; splits: test (-1)
- **LoCoV1** — total ?; splits: test (-1)

## Metrics

- `MTEB Average Score` **(primary)** — range: [0, 1]
  - Task-specific metric (e.g., cosine similarity for STS, accuracy for classification, NDCG@10 for retrieval) computed per dataset, averaged within each of the seven task categories, then averaged across categories for an overall score.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked document retrieval lists against ground-truth relevant documents.

## Input / output format

**Input**: Text sequences (sentences or documents) fed to decoder-only LLMs. Max length 512 for MTEB; truncated to 1024, 2048, or 4096 tokens for LoCoV1.

**Output**: Fixed-dimensional text embeddings extracted from internal Key-Value (KV) states of specified transformer layers, without modifying the input sequence.

## Scoring recipe

```python
def compute_mteb_avg(datasets_by_category):
    category_scores = []
    for cat, ds_list in datasets_by_category.items():
        task_scores = []
        for ds in ds_list:
            preds = extract_kv_embeddings(ds.texts)
            gold = ds.labels
            task_scores.append(compute_task_metric(preds, gold))
        category_scores.append(mean(task_scores))
    return mean(category_scores)

def compute_locov1_score(retrieval_preds, gold_docs, k=10):
    return ndcg_at_k(retrieval_preds, gold_docs, k)
```

## Common pitfalls

- Assuming the method requires gradient-based fine-tuning; it is explicitly training-free and zero-shot.
- Misinterpreting the 'w/o KV Re-routing' row as the main method; it is an ablation that disables the core re-routing mechanism while keeping the compression prompt.
- Overlooking that MTEB scores are category-averaged task-specific metrics, not a single unified score, and vary by backbone architecture.

## Evidence (verbatim from paper)

> We evaluate KV-Embedding on two complementary benchmarks. MTEB provides a multi-task assessment across seven categories: STS, Retrieval, Classification, Pair Classification, Clustering, Reranking, and Summarization. To evaluate robustness in long-context scenarios, we use LoCoV1, truncating documents to 1024, 2048, and 4096 tokens. Table 2: Retrieval performance (NDCG@10) on LoCoV1 across different context lengths.

## Citation

```bibtex
@misc{tang2026kvembedding,
  title={KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2601.01046}
}
```

- arXiv: 2601.01046