German Text Embedding Clustering Eval

---
name: german-text-embedding-clustering-eval
description: Evaluates the quality of text embeddings for German-language documents by measuring how well they cluster into predefined topical categories. It probes a model's ability to capture semantic similarity and domain-specific nuances across different text lengths (titles vs. full texts) and sources. Use when the user wants to benchmark on BlurbsClusteringS2S/P2P, TenKGnadClusteringS2S/P2P, SubredditClusteringS2S/P2P, or asks about evaluating this task. Reports V-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.02709
  bibtex_key: wehrli2024german
  confidence: high
---

# german-text-embedding-clustering-eval

> German Text Embedding Clustering Benchmark — Wehrli et al. (2024) (arXiv:2401.02709, 2024)

## What this evaluates

Evaluates the quality of text embeddings for German-language documents by measuring how well they cluster into predefined topical categories. It probes a model's ability to capture semantic similarity and domain-specific nuances across different text lengths (titles vs. full texts) and sources.

## Datasets

- **BlurbsClusteringS2S/P2P** — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de
- **TenKGnadClusteringS2S/P2P** — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de
- **SubredditClusteringS2S/P2P** — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de

## Metrics

- `V-measure` **(primary)** — range: [0, 1]
  - Measures clustering quality by combining homogeneity (each cluster contains only members of a single class) and completeness (all members of a given class are assigned to the same cluster). Outputs a score between 0 and 1, with 1 indicating perfect alignment with ground-truth topical categories.

## Input / output format

**Input**: Raw text strings (titles, blurbs, news articles, or Reddit submissions) provided for embedding generation.

**Output**: Cluster assignments for each text instance, compared against ground-truth topical categories.

## Scoring recipe

```python
def compute_v_measure(true_labels, pred_labels):
    homogeneity = compute_homogeneity(true_labels, pred_labels)
    completeness = compute_completeness(true_labels, pred_labels)
    if homogeneity + completeness == 0:
        return 0.0
    return 2 * (homogeneity * completeness) / (homogeneity + completeness)
```

## Common pitfalls

- The benchmark uses 10 random subsamples (splits) of varying sizes (10-100% of available data) per dataset, so reported scores can fluctuate significantly depending on the specific split drawn.
- V-measure evaluates clustering against ground-truth topical categories but does not inherently account for hierarchical granularity differences; researchers must explicitly configure splits for coarse vs. fine-grained genre clustering.
- German datasets are smaller and less standardized than English counterparts, requiring careful handling of data privacy constraints (e.g., providing submission IDs instead of raw text for Reddit) and domain coverage.

## Evidence (verbatim from paper)

> The evaluation is based on the V-measure Rosenberg and Hirschberg ([2007](#bib.bib34 "")). Given a ground truth, the V-measure outputs a score between 0 and 1, measuring homogeneity (clusters contain only one class) and completeness (clusters contain all class samples). MTEB uses topical categories derived from the data, such as the scientific discipline of a publication or newsgroup, as the ground truth.

## Citation

```bibtex
@misc{wehrli2024german,
  title={German Text Embedding Clustering Benchmark},
  author={Wehrli et al. (2024)},
  year={2024},
  note={arXiv:2401.02709}
}
```

- arXiv: 2401.02709