Resel Scientific Ie Eval

---
name: resel-scientific-ie-eval
description: Evaluates a model's ability to perform N-ary relation extraction from scientific documents by first retrieving relevant text/table components and then selecting the correct entities within those components. Use when the user wants to benchmark on SciREX, PubMed, NLP-TDMS (Full), or asks about evaluating this task. Reports Accuracy (Acc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.14427
  bibtex_key: zhuang2022resel
  confidence: high
---

# resel-scientific-ie-eval

> ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select — Zhuang et al. (2022) (arXiv:2210.14427, 2022)

## What this evaluates

Evaluates a model's ability to perform N-ary relation extraction from scientific documents by first retrieving relevant text/table components and then selecting the correct entities within those components.

## Datasets

- **SciREX** — total 438; splits: test (-1)
- **PubMed** — total 5688; splits: test (-1)
- **NLP-TDMS (Full)** — total 332; splits: test (-1)

## Metrics

- `Accuracy (Acc)` **(primary)** — range: percent
  - Fraction of instances where the predicted component or entity exactly matches the ground truth.
- `Mean Reciprocal Rank (MRR)` — range: [0, 1]
  - Average of 1/rank for the correct answer across all queries, where rank is the position of the ground truth in the ranked list.
- `Top-k Hit Rate (Hit@K)` — range: percent
  - Fraction of queries where the correct answer appears within the top K ranked results (k=2, 3, 5).

## Input / output format

**Input**: A query specifying the relation to extract, paired with a full scientific document containing text and tables (extracted from LaTeX/PDF).

**Output**: A ranked list of retrieved document components (paragraphs/tables) and a set of selected target entities within the retrieved components.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    mrr = sum(1.0 / (rank + 1) for p, g in zip(predictions, gold) for rank, pred in enumerate(p) if pred == g) / len(gold)
    hit_rates = {k: sum(1 for p, g in zip(predictions, gold) if g in p[:k]) / len(gold) for k in [2, 3, 5]}
    return acc, mrr, hit_rates
```

## Common pitfalls

- Using general-domain BERT instead of domain-specific encoders (SciBERT/ClinicalBERT) significantly degrades performance.
- Graph-based methods are prone to over-smoothing if the number of GAT layers (L) is increased too much.
- BERT embeddings alone struggle to discriminate numeric values and adjacent cells in tables without structural graph features.

## Evidence (verbatim from paper)

> Following existing works (Karpukhin et al., 2020), we use (1) Accuracy (Acc), (2) Mean Reciprocal Rank (MRR), and (3) Top-k Hit Rate (Hit@K) with  $k = 2, 3, 5$  for evaluating both high- and low-level models (see Appendix D).

## Citation

```bibtex
@misc{zhuang2022resel,
  title={ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select},
  author={Zhuang et al. (2022)},
  year={2022},
  note={arXiv:2210.14427}
}
```

- arXiv: 2210.14427