Scienceie Keyphrase Relation Eval

---
name: scienceie-keyphrase-relation-eval
description: This benchmark evaluates systems on mention-level keyphrase extraction and semantic relation extraction from scientific publications. It probes the model's ability to identify, classify, and link keyphrases across different scientific domains using exact match criteria. Use when the user wants to benchmark on ScienceIE, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1704.02853
  bibtex_key: augenstein2017scienceie
  confidence: high
---

# scienceie-keyphrase-relation-eval

> SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications — Augenstein et al. (2017) (SemEval 2017, 2017)

## What this evaluates

This benchmark evaluates systems on mention-level keyphrase extraction and semantic relation extraction from scientific publications. It probes the model's ability to identify, classify, and link keyphrases across different scientific domains using exact match criteria.

## Datasets

- **ScienceIE** — total 500; splits: train (-1), dev (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall based on exact matches between predicted and gold keyphrases, types, and relations. Micro-averaged across publications of the three genres (Computer Science, Material Sciences, Physics).

## Input / output format

**Input**: Plain text of scientific publications, optionally accompanied by manually annotated keyphrase boundaries and/or keyphrase types depending on the evaluation scenario.

**Output**: A list of extracted keyphrase mentions, their assigned types (PROCESS, TASK, or MATERIAL), and the semantic relations (HYPONYM-OF or SYNONYM-OF) linking them.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
# Final score is micro-averaged across all publications in a genre
```

## Common pitfalls

- Evaluators must use exact match criteria rather than semantic similarity metrics, as explicitly stated in the protocol.
- The final reported score must be micro-averaged across publications within each of the three scientific genres, not macro-averaged or averaged globally without genre stratification.

## Evidence (verbatim from paper)

> The traditionally used metrics of precision, recall and F1-score are computed and the micro-average of those metrics across publications of the three genres are calculated. These metrics are also calculated for Subtasks B and C.

## Citation

```bibtex
@misc{augenstein2017scienceie,
  title={SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications},
  author={Augenstein et al. (2017)},
  year={2017},
  note={SemEval 2017}
}
```

- arXiv: 1704.02853