Pteb Eval

---
name: pteb-eval
description: Evaluates the robustness of sentence embedding models to token-level variations by measuring performance degradation when test instances are stochastically paraphrased at evaluation time. It probes whether models maintain semantic invariance under LLM-generated paraphrases that preserve meaning but alter surface form. Use when the user wants to benchmark on MTEB (STS & Non-STS tasks), or asks about evaluating this task. Reports Spearman’s rank correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06730
  bibtex_key: frank2025pteb
  confidence: high
---

# pteb-eval

> PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs — Frank et al. (2025) (arXiv:2510.06730, 2025)

## What this evaluates

Evaluates the robustness of sentence embedding models to token-level variations by measuring performance degradation when test instances are stochastically paraphrased at evaluation time. It probes whether models maintain semantic invariance under LLM-generated paraphrases that preserve meaning but alter surface form.

## Datasets

- **MTEB (STS & Non-STS tasks)** — total ?; splits: test (-1)

## Metrics

- `Spearman’s rank correlation` **(primary)** — range: [0, 1]
  - Non-parametric measure of rank correlation between predicted similarity scores (cosine similarity of embeddings) and gold human ratings. Computed as ρ = 1 - (6Σd_i²)/(n(n²-1)) for tied ranks, or via standard rank correlation function.

## Input / output format

**Input**: Sentence pairs (for STS, pair classification, reranking, retrieval) or single texts/documents (for classification, clustering, summarization). For PTEB, each instance is dynamically paraphrased into 6 variants using an LLM (gemma3-27b) before encoding.

**Output**: Fixed-dimensional embedding vectors for each text. For STS, cosine similarity between the two embeddings is computed and compared to a gold rating on a 0–5 scale.

## Scoring recipe

```python
def score_pteb(embeddings, gold_ratings, n_runs=6):
    spearman_scores = []
    for run in range(n_runs):
        sim_scores = cosine_similarity(embeddings[run])
        rho = spearmanr(sim_scores, gold_ratings).correlation
        spearman_scores.append(rho)
    return np.mean(spearman_scores) * 100  # percentage

# Compute Δ = Original_Score - PTEB_Score
# Validate significance with Wilcoxon signed-rank test + Holm correction
```

## Common pitfalls

- Failing to use sufficient paraphrase samples (n=6 is recommended) leads to high variance in standard deviation estimates and unreliable Δ calculations.
- Ignoring statistical significance testing; performance drops must be validated with Wilcoxon signed-rank tests and Holm-corrected p-values rather than relying on raw Δ differences.
- Assuming paraphrases perfectly preserve semantics without LLM judge or human verification, which can introduce semantic drift and confound robustness measurements.

## Evidence (verbatim from paper)

> Table 2: Spearman’s rank correlation between LLM Judge similarity scores and gold ratings for STS datasets; best scores bold. (in %)

## Citation

```bibtex
@misc{frank2025pteb,
  title={PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs},
  author={Frank et al. (2025)},
  year={2025},
  note={arXiv:2510.06730}
}
```

- arXiv: 2510.06730