Multilingual Sts B Eval

---
name: multilingual-sts-b-eval
description: Evaluates the semantic similarity and cross-lingual transfer capabilities of pixel-based sentence representations by measuring how well the model captures semantic continuity across 10 languages and handles out-of-distribution text perturbations. Use when the user wants to benchmark on multilingual STS-b, Natural Questions, or asks about evaluating this task. Reports STS-b correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.08183
  bibtex_key: xiao2024pixelsentencerepresentationlearning
  confidence: high
---

# multilingual-sts-b-eval

> Pixel Sentence Representation Learning — Xiao et al. (2024) (arXiv:2402.08183, 2024)

## What this evaluates

Evaluates the semantic similarity and cross-lingual transfer capabilities of pixel-based sentence representations by measuring how well the model captures semantic continuity across 10 languages and handles out-of-distribution text perturbations.

## Datasets

- **multilingual STS-b** — total ?; splits: test (-1)
- **Natural Questions** — total ?; splits: test (-1)

## Metrics

- `STS-b correlation` **(primary)** — range: [-1, 1]
  - Standard STS-b evaluation: compute cosine similarity between normalized mean-pooled embeddings of sentence pairs, then calculate Spearman rank correlation against human similarity scores.

## Input / output format

**Input**: Text rendered into images on-the-fly using PangoCairo, processed as pixel patches with a maximum sequence length of 64.

**Output**: Normalized mean-pooled embeddings used for cosine similarity computation.

## Scoring recipe

```python
for each pair in test_set:
  img1 = render_text_to_image(pair.s1)
  img2 = render_text_to_image(pair.s2)
  emb1 = normalize(model.mean_pool(img1))
  emb2 = normalize(model.mean_pool(img2))
  pred_scores.append(cosine_similarity(emb1, emb2))
metric_value = spearman_corr(pred_scores, pair.human_scores)
```

## Common pitfalls

- Early-phase overfitting to semantic tasks can degrade later supervised training performance.
- STS performance is orthogonal to retrieval performance; the best STS checkpoint may not be optimal for retrieval tasks.
- Using the [CLS] token instead of mean-pooling leads to under-training and poor convergence.

## Evidence (verbatim from paper)

> For evaluation, we use the test set of multilingual STS-b (Cer et al.,, 2017; May,, 2021), which include 10 languages from 4 language families as shown throughout the paper. ... Empirically, we find that checkpoints that display good semantic performance in earlier stages do not necessarily provide best potential in later supervised training, exhibiting certain early-phase overfitting.

## Citation

```bibtex
@misc{xiao2024pixelsentencerepresentationlearning,
  title={Pixel Sentence Representation Learning},
  author={Xiao et al. (2024)},
  year={2024},
  note={arXiv:2402.08183}
}
```

- arXiv: 2402.08183