Openie Systems Eval

---
name: openie-systems-eval
description: Evaluates the performance and efficiency of state-of-the-art neural OpenIE models and training datasets across multiple standard benchmarks. It probes how model properties like N-ary relation support and inferred relation extraction capability align with benchmark characteristics and downstream task requirements. Use when the user wants to benchmark on OIE2016, WiRE57, ReOIE2016, CaRB, LSOIE, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.08228
  bibtex_key: pei2022whentousewhat
  confidence: high
---

# openie-systems-eval

> When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications — Pei et al. (2022) (arXiv:2211.08228, 2022)

## What this evaluates

Evaluates the performance and efficiency of state-of-the-art neural OpenIE models and training datasets across multiple standard benchmarks. It probes how model properties like N-ary relation support and inferred relation extraction capability align with benchmark characteristics and downstream task requirements.

## Datasets

- **OIE2016** — total ?; splits: test (-1)
- **WiRE57** — total ?; splits: test (-1)
- **ReOIE2016** — total ?; splits: test (-1)
- **CaRB** — total ?; splits: test (-1)
- **LSOIE** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, calculated as the maximum F1 value across the precision-recall curve.
- `Precision` — range: [0, 1]
  - Proportion of extracted relations that exactly match gold standard relations.
- `Recall` — range: [0, 1]
  - Proportion of gold standard relations successfully extracted by the model.
- `Sentences per second` — range: other
  - Throughput metric measuring the average number of sentences processed per second across batches.

## Input / output format

**Input**: Raw English sentences from benchmark datasets.

**Output**: Extracted relation tuples (subject, relation, object), with optional additional arguments for N-ary relations.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- Using AUC instead of F1 can artificially inflate scores for low-recall models due to the trapezoidal rule's assumption of a (recall=0, precision=1) anchor point.
- Training data preprocessing (e.g., stripping inferred relations or flattening N-ary relations) is applied to training sets but not test sets, potentially skewing performance expectations.
- Efficiency is measured as average sentences per second across batches, not per individual sentence, which can obscure latency variance during parallel execution.

## Evidence (verbatim from paper)

> When comparing OpenIE systems, we place a greater emphasis on F1 score than AUC. ... We compare performance using primarily F1 score to address HR and HP and sentences extracted per second to address FE.

## Citation

```bibtex
@misc{pei2022whentousewhat,
  title={When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications},
  author={Pei et al. (2022)},
  year={2022},
  note={arXiv:2211.08228}
}
```

- arXiv: 2211.08228