Ontology Alignment Eval

---
name: ontology-alignment-eval
description: This benchmark evaluates the accuracy of mapping OpenAlex paper topics to terms across 13 scientific ontologies. It probes a method's ability to perform semantic and lexical alignment between informal topic labels and formal domain-specific vocabularies. Use when the user wants to benchmark on Ontology Alignment Gold Standard, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.03126
  bibtex_key: [REDACTED]
  confidence: high
---

# ontology-alignment-eval

> The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment — Wilinski (2026) (arXiv:2603.03126, 2026)

## What this evaluates

This benchmark evaluates the accuracy of mapping OpenAlex paper topics to terms across 13 scientific ontologies. It probes a method's ability to perform semantic and lexical alignment between informal topic labels and formal domain-specific vocabularies.

## Datasets

- **Ontology Alignment Gold Standard** — total 300; splits: test (300)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed from unrounded Precision and Recall values.
- `Precision` — range: [0, 1]
  - Proportion of predicted mappings that are correct in the gold standard.
- `Recall` — range: [0, 1]
  - Proportion of gold standard mappings that are correctly predicted.

## Input / output format

**Input**: OpenAlex topic labels (4,516 total) and ontology term labels (1.3M total across 13 ontologies).

**Output**: A single mapped ontology term for each OpenAlex topic, or null if no mapping is produced.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    correct = sum(1 for t, pred in predictions.items() if (t, pred) in gold)
    total_preds = len(predictions)
    total_gold = len(gold)
    precision = correct / total_preds if total_preds > 0 else 0.0
    recall = correct / total_gold if total_gold > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- "Partial" matches in the gold standard are strictly counted as false positives, not partial credit.
- F1 must be computed from unrounded Precision and Recall values; rounding them first yields slightly different F1 scores.
- Large ontologies (MeSH, ChEBI, NCIT) bypass the embedding search and use exact string matching, which affects the overall alignment distribution.

## Evidence (verbatim from paper)

> Precision (P), recall (R), and F1 are evaluated against a stratified 300-pair gold-standard annotation set (see Section 4); a mapping is counted as correct only if the annotator labelled it correct (strict; partial counts as false positive). F1 is computed from unrounded P and R; rounding the displayed values may yield slightly different results.

## Citation

```bibtex
@misc{wilinski2026sciencedatalake,
  title={The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment},
  author={Wilinski (2026)},
  year={2026},
  note={arXiv:2603.03126}
}
```

- arXiv: 2603.03126