Scientific Embedding Eval

---
name: scientific-embedding-eval
description: Evaluates static (Word2Vec, FastText) and transformer-based (SciBERT, RoBERTa) embedding models on scientific text using intrinsic (word/sentence similarity) and extrinsic (NER, document classification) tasks. Probes the impact of domain-specific pretraining and sub-word tokenization on representation quality and downstream performance. Use when the user wants to benchmark on UNMSRS, SemEval, Clinical STS 2018, Clinical STS 2019, Conll 2003, CHEMDNER, SciERC, Reuters 12, BioChem 8, or asks about evaluating this task. Reports Pearson, F-Beta.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06244
  bibtex_key: ahmed2025evaluatingembedding
  confidence: high
---

# scientific-embedding-eval

> Evaluating Embedding Frameworks for Scientific Domain — Ahmed et al. (2025) (arXiv:2510.06244, 2025)

## What this evaluates

Evaluates static (Word2Vec, FastText) and transformer-based (SciBERT, RoBERTa) embedding models on scientific text using intrinsic (word/sentence similarity) and extrinsic (NER, document classification) tasks. Probes the impact of domain-specific pretraining and sub-word tokenization on representation quality and downstream performance.

## Datasets

- **UNMSRS** — total ?; splits: test (-1)
- **SemEval** — total ?; splits: test (-1)
- **Clinical STS 2018** — total ?; splits: test (-1)
- **Clinical STS 2019** — total ?; splits: test (-1)
- **Conll 2003** — total ?; splits: test (-1)
- **CHEMDNER** — total ?; splits: test (-1)
- **SciERC** — total ?; splits: test (-1)
- **Reuters 12** — total ?; splits: test (-1)
- **BioChem 8** — total ?; splits: test (-1)

## Metrics

- `Pearson` **(primary)** — range: [-1, 1]
  - Pearson correlation coefficient measuring the linear relationship between predicted and gold similarity scores. Computed as the covariance of the two variables divided by the product of their standard deviations.
- `F-Beta` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for entity or class prediction, typically with β=1. Calculated as (1 + β²) * (precision * recall) / ((β² * precision) + recall).

## Input / output format

**Input**: Pairs of words or sentences for similarity tasks; raw text spans or documents for NER and classification tasks.

**Output**: Continuous similarity scores for pairs; discrete entity tags or document class labels for NER and classification.

## Scoring recipe

```python
# Pearson correlation for similarity tasks
pearson_r = np.corrcoef(gold_scores, pred_scores)[0, 1]

# F-Beta score for NER/Classification tasks
tp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == p and g != 0)
fp = sum(1 for g, p in zip(gold_labels, pred_labels) if g != p and p != 0)
fn = sum(1 for g, p in zip(gold_labels, pred_labels) if g != p and g != 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f_beta = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall)
```

## Common pitfalls

- Pearson correlation calculation ignores NaN values produced by word-based models on OOV tokens, artificially inflating their scores compared to sub-word models that handle OOVs.
- Sub-word tokenization does not universally improve performance; word-based tokenizers often outperform on sentence similarity and document classification due to large vocabulary sizes mitigating OOV issues.

## Evidence (verbatim from paper)

> It is able to generate a relatively good pearson score for in vocabulary tokens though.

## Citation

```bibtex
@misc{ahmed2025evaluatingembedding,
  title={Evaluating Embedding Frameworks for Scientific Domain},
  author={Ahmed et al. (2025)},
  year={2025},
  note={arXiv:2510.06244}
}
```

- arXiv: 2510.06244