Mteb Clustering Eval

---
name: mteb-clustering-eval
description: Evaluates the quality of text embeddings for document clustering by measuring how well the embeddings group semantically related sentences together. It probes the model's ability to capture semantic similarity without task-specific fine-tuning or with lightweight adaptation. Use when the user wants to benchmark on MTEB v1.38.30 English Clustering, or asks about evaluating this task. Reports clustering accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.22729
  bibtex_key: roth2025resourceefficient
  confidence: high
---

# mteb-clustering-eval

> Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning — Benedikt Roth et al. (2025) (arXiv:2507.22729, 2025)

## What this evaluates

Evaluates the quality of text embeddings for document clustering by measuring how well the embeddings group semantically related sentences together. It probes the model's ability to capture semantic similarity without task-specific fine-tuning or with lightweight adaptation.

## Datasets

- **MTEB v1.38.30 English Clustering** — total ?; splits: test (-1)

## Metrics

- `clustering accuracy` **(primary)** — range: percent
  - Macro-average clustering accuracy (vmeasure) computed per dataset and averaged across the 21 English clustering datasets in MTEB v1.38.30. Scores are reported as single-run values without confidence intervals.

## Input / output format

**Input**: Raw text sentences, optionally wrapped with task-specific prompts (e.g., CCW, CCP, or classification prompts).

**Output**: Fixed-dimensional embedding vectors derived from the LLM's hidden states using either mean pooling or last-token pooling.

## Scoring recipe

```python
scores = []
for dataset in mteb_english_clustering_datasets:
    embeddings = model.encode(dataset.documents, pooling='mean')
    clusters = agglomerative_cluster(embeddings)
    scores.append(vmeasure_score(clusters, dataset.labels))
final_metric = mean(scores) * 100  # reported as percent
```

## Common pitfalls

- MTEB protocol mandates single-run scores without confidence intervals or variance estimates for direct comparability.
- Pooling strategy drastically changes results; mean pooling generally outperforms last-token in zero-shot, but last-token benefits more from prompt engineering and fine-tuning.
- Prompt formatting must be applied consistently; wrapping prompts around input text yields different results than appending them.

## Evidence (verbatim from paper)

> The zero-shot baseline was obtained by comparing three token aggregation strategies across Llama-3.2-1B and Qwen3-0.6B. As shown in Table 3, mean pooling delivers the highest average clustering accuracy, outperforming last-token pooling by approximately  $12\%$  and EOS-token pooling by around  $20\%$  across both models.

## Citation

```bibtex
@misc{roth2025resourceefficient,
  title={Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning},
  author={Benedikt Roth et al. (2025)},
  year={2025},
  note={arXiv:2507.22729}
}
```

- arXiv: 2507.22729