Geneol Eval

---
name: geneol-eval
description: Evaluates training-free sentence embedding quality by aggregating LLM-generated semantic variations. Probes semantic similarity preservation and cross-task robustness without model fine-tuning. Use when the user wants to benchmark on STS benchmark, MTEB, or asks about evaluating this task. Reports Spearman rank correlation (cosine similarity).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.14635
  bibtex_key: thirukovalluru2024geneol
  confidence: high
---

# geneol-eval

> GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings — Thirukovalluru et al. (2024) (arXiv:2410.14635, 2024)

## What this evaluates

Evaluates training-free sentence embedding quality by aggregating LLM-generated semantic variations. Probes semantic similarity preservation and cross-task robustness without model fine-tuning.

## Datasets

- **STS benchmark** — total ?; splits: test (-1)
- **MTEB** — total ?; splits: test (-1)

## Metrics

- `Spearman rank correlation (cosine similarity)` **(primary)** — range: [-1, 1]
  - Spearman rank correlation between cosine similarities of aggregated sentence embeddings and human-provided similarity scores.

## Input / output format

**Input**: Single sentences or sentence pairs. For each sentence, the method generates m semantically equivalent transformations using an LLM, then computes the mean of their embeddings.

**Output**: A single fixed-dimensional sentence embedding vector per input, obtained by averaging the embeddings of the original sentence and its m generated transformations.

## Scoring recipe

```python
preds = []
for s1, s2 in test_pairs:
    emb1 = mean(embed(s1), *[embed(t) for t in generate(s1, m)])
    emb2 = mean(embed(s2), *[embed(t) for t in generate(s2, m)])
    preds.append(cosine_similarity(emb1, emb2))
return spearman_corr(preds, gold_scores)
```

## Common pitfalls

- Using training splits for STS tasks, which the authors explicitly exclude.
- Unequally distributing the generation budget (m) across sentence pairs in similarity tasks drastically reduces performance compared to equal allocation.
- Relying on the final hidden layer for embedding extraction; penultimate layers often yield better results for training-free methods.

## Evidence (verbatim from paper)

> We evaluate GenEOLand other baselines on the STS benchmark Conneau and Kiela ([2018]). Spearman rank correlation (cosine similarity) is the main metric Muennighoff et al. ([2022]). Training sets of the STS tasks are not used. We additionally asses GenEOLon 10 MTEB tasks across 4 categories (Classification, Clustering, Reranking and Pair Classification) using the appropriate metrics described in Muennighoff et al. ([2022]).

## Citation

```bibtex
@misc{thirukovalluru2024geneol,
  title={GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings},
  author={Thirukovalluru et al. (2024)},
  year={2024},
  note={arXiv:2410.14635}
}
```

- arXiv: 2410.14635