Scientific Relation Classification Eval

---
name: scientific-relation-classification-eval
description: Evaluates the robustness and cross-dataset/domain generalization of relation extraction models on scientific abstracts. It probes how annotation discrepancies and domain shifts affect relation classification performance. Use when the user wants to benchmark on SemEval-2018, SciERC, or asks about evaluating this task. Reports Macro F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.13516
  bibtex_key: bassignana2022survey
  confidence: high
---

# scientific-relation-classification-eval

> What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification — Bassignana et al. (2022) (arXiv:2204.13516, 2022)

## What this evaluates

Evaluates the robustness and cross-dataset/domain generalization of relation extraction models on scientific abstracts. It probes how annotation discrepancies and domain shifts affect relation classification performance.

## Datasets

- **SemEval-2018** — total ?; splits: train (257), dev (50), test (50)
- **SciERC** — total ?; splits: train (257), test_NLP (50), test_AI-ML (52), test_CV (105), test_SPEECH (35)

## Metrics

- `Macro F1-score` **(primary)** — range: [0, 1]
  - Computes the unweighted mean of the F1-score for each relation class, then averages them across all classes. It treats all classes equally regardless of frequency.

## Input / output format

**Input**: Scientific abstract sentences containing two entity spans and their surrounding context.

**Output**: A relation label from the unified label set (e.g., METHOD, TASK, METRIC) assigned to the entity pair.

## Scoring recipe

```python
def compute_macro_f1(preds, gold):
    classes = sorted(set(preds + gold))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- Cross-dataset evaluation uses overlapping abstracts with divergent annotations, so models trained on one dataset may not generalize directly to the other without handling annotation conflicts.
- The paper reports results averaged over three different seeds, so single-run evaluations will not match the reported numbers.
- SciERC is split into sub-domains (NLP, AI-ML, CV, SPEECH); cross-domain tests specifically train on NLP and test on the others, which differs from standard in-domain evaluation.

## Evidence (verbatim from paper)

> We use macro F1-score as evaluation metric. All experiments were run over three different seeds and the results reported are the mean.

## Citation

```bibtex
@misc{bassignana2022survey,
  title={What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification},
  author={Bassignana et al. (2022)},
  year={2022},
  note={arXiv:2204.13516}
}
```

- arXiv: 2204.13516