Sentence Representation Eval

---
name: sentence-representation-eval
description: Evaluates the quality of sentence-level representations learned by a transformer-based autoencoder across semantic similarity, single- and multi-sentence classification, and controlled text generation. It probes the model's ability to capture semantic meaning, classify sentiment/acceptability, and reconstruct or modify text via vector arithmetic. Use when the user wants to benchmark on Semantic Textual Similarity (STS), GLUE benchmark, Yelp reviews, or asks about evaluating this task. Reports Spearman's rank correlation, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.00055
  bibtex_key: [REDACTED]
  confidence: high
---

# sentence-representation-eval

> Sentence Bottleneck Autoencoders from Transformer Language Models — Montero et al. (2021) (arXiv:2109.00055, 2021)

## What this evaluates

Evaluates the quality of sentence-level representations learned by a transformer-based autoencoder across semantic similarity, single- and multi-sentence classification, and controlled text generation. It probes the model's ability to capture semantic meaning, classify sentiment/acceptability, and reconstruct or modify text via vector arithmetic.

## Datasets

- **Semantic Textual Similarity (STS)** — total ?; splits: test (-1), dev (-1)
- **GLUE benchmark** — total ?; splits: dev (-1)
- **Yelp reviews** — total ?; splits: train (-1)

## Metrics

- `Spearman's rank correlation` **(primary)** — range: [-1, 1]
  - Computes the rank correlation between predicted similarity scores and human-annotated gold scores. Higher values indicate better alignment with human judgments.
- `Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of the total number of instances. Reported as median over three random seeds.
- `Self-BLEU` — range: [0, 1]
  - Measures the n-gram overlap between the generated sentence and the original input sentence, computed using standard BLEU scoring.

## Input / output format

**Input**: Pairs of sentences for similarity tasks; single sentences for classification; single sentences for generation/style transfer.

**Output**: Similarity scores or ranks; class labels; generated sentences.

## Scoring recipe

```python
from scipy.stats import spearmanr

# STS: Spearman correlation
spearman_corr, _ = spearmanr(gold_scores, pred_scores)

# Classification: Accuracy
correct = sum(1 for g, p in zip(gold_labels, pred_labels) if g == p)
acc = correct / len(gold_labels)

# Generation: Self-BLEU & Classifier Accuracy
self_bleu = compute_self_bleu(input_text, generated_text)
gen_acc = sentiment_classifier.predict(generated_text) == target_sentiment
```

## Common pitfalls

- STS evaluation uses Spearman's rank correlation, not Pearson correlation or exact-match accuracy.
- Classification results report median accuracy over three random seeds, not a single run.
- Generation accuracy is measured by an external sentiment classifier, not human judgment or standard BLEU.

## Evidence (verbatim from paper)

> We report Spearman's rank correlation on the test set and the model sizes are reported in terms of trained parameter size.

## Citation

```bibtex
@misc{montero2021sentence,
  title={Sentence Bottleneck Autoencoders from Transformer Language Models},
  author={Montero et al. (2021)},
  year={2021},
  note={arXiv:2109.00055}
}
```

- arXiv: 2109.00055