Negation Embedding Eval

---
name: negation-embedding-eval
description: Probes whether text embedding models suffer from 'negation blindness' by testing their ability to correctly identify paraphrases over negated counterparts, and measures the trade-off with semantic similarity correlation. Use when the user wants to benchmark on STSB, SemAntoNeg, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.00584
  bibtex_key: cao2025semanticadapter
  confidence: high
---

# negation-embedding-eval

> Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality — Cao (2025) (arXiv:2504.00584, 2025)

## What this evaluates

Probes whether text embedding models suffer from 'negation blindness' by testing their ability to correctly identify paraphrases over negated counterparts, and measures the trade-off with semantic similarity correlation.

## Datasets

- **STSB** — total ?; splits: train (-1), test (-1); HF `glue/stsb`
- **SemAntoNeg** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly identified paraphrase candidates out of total test instances. For STSB, candidates are the positive paraphrase and its negated counterpart. For SemAntoNeg, candidates are three expressions with negation/antonym substitutions.
- `Correlation` — range: [-1, 1]
  - Pearson correlation coefficient between the model's computed embedding similarity scores and human-annotated semantic similarity values, following the standard STSB evaluation protocol.

## Input / output format

**Input**: Sentence pairs (query, positive paraphrase) for STSB; query plus three candidate paraphrases with negation/antonym substitutions for SemAntoNeg. Models compute cosine similarity between query embeddings and candidate embeddings.

**Output**: Discrete class label (selected paraphrase candidate) for accuracy; continuous similarity score for correlation.

## Scoring recipe

```python
# Accuracy
correct = sum(1 for q, pos, neg in test
              if sim(q, pos) > sim(q, neg))
accuracy = correct / len(test)

# Correlation (STSB)
preds = [sim(s1, s2) for s1, s2 in test]
gold = [score for _, _, score in test]
correlation = pearsonr(preds, gold).statistic
```

## Common pitfalls

- STSB test set merges official validation and test splits.
- SemAntoNeg requires 10-fold repeated random sampling (32/68 split) due to missing predefined splits.
- Adapter application typically reduces Pearson correlation by ~1% despite boosting accuracy.

## Evidence (verbatim from paper)

> In order to test the effectiveness of the proposed embedding re-weighting method, a paraphrase detection task is formulated using the STSB dataset: the Group 5 data from STSB [$sentence_1_i$, $sentence_2_i$] are used as paraphrase text pairs. The embedding models are then required to identify the accurate paraphrase of $sentence_1_i$ from [$sentence_2_i$, $neg_sentence_1_i$]. ... Two evaluation metrics are used in this experiment: Accuracy: measures if state-of-the-art text embeddings can correctly identify semantically similar text $sentence_2_i$ from negated text $neg_sentence_1_i$ which has similar surface form as $sentence_1_i$. Correlation: measures if the similarity score of state-of-the-art text embeddings correlates well with human annotated similarity values using Pearson correlation following the evaluation protocol of STSB.

## Citation

```bibtex
@misc{cao2025semanticadapter,
  title={Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality},
  author={Cao (2025)},
  year={2025},
  note={arXiv:2504.00584}
}
```

- arXiv: 2504.00584