Oie Error Analysis Eval

---
name: oie-error-analysis-eval
description: Evaluates Open Information Extraction (OIE) systems on their ability to correctly extract relational tuples from text, measuring precision, recall, and F2 scores under both strict and relaxed containment matching strategies. It also qualitatively classifies extraction errors to identify systematic failure modes like boundary mismatches and annotation style conflicts. Use when the user wants to benchmark on NYT-222, WEB-500, PENN-100, OIE2016, or asks about evaluating this task. Reports F2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1707.07499
  bibtex_key: schneider2017analysing
  confidence: high
---

# oie-error-analysis-eval

> Analysing Errors of Open Information Extraction Systems — Rudolf Schneider et al. (2017) (arXiv:1707.07499, 2017)

## What this evaluates

Evaluates Open Information Extraction (OIE) systems on their ability to correctly extract relational tuples from text, measuring precision, recall, and F2 scores under both strict and relaxed containment matching strategies. It also qualitatively classifies extraction errors to identify systematic failure modes like boundary mismatches and annotation style conflicts.

## Datasets

- **NYT-222** — total ?; splits: (unstated)
- **WEB-500** — total ?; splits: (unstated)
- **PENN-100** — total ?; splits: (unstated)
- **OIE2016** — total 1768; splits: (unstated)

## Metrics

- `F2` **(primary)** — range: [0, 1]
  - F2 = (5 * P * R) / (4 * P + R), where P is precision and R is recall. Beta is set to 2 to weight recall twice as heavily as precision.

## Input / output format

**Input**: Raw sentences from the dataset.

**Output**: Extracted relations formatted as predicate(arg1, arg2, ...) with variable arity (unary to n-ary).

## Scoring recipe

```python
def containment_match(pred, gold, relaxed=False):
    if relaxed:
        return pred.predicate == gold.predicate and set(pred.args) <= set(gold.args)
    return pred == gold  # exact span/string match

tp = sum(1 for p in preds if any(containment_match(p, g, relaxed=True) for g in golds))
fp = len(preds) - tp
fn = len(golds) - tp
P = tp / (tp + fp) if (tp + fp) > 0 else 0
R = tp / (tp + fn) if (tp + fn) > 0 else 0
F2 = (5 * P * R) / (4 * P + R) if (4 * P + R) > 0 else 0
```

## Common pitfalls

- Strict containment matching penalizes systems heavily for minor boundary shifts or argument count differences, especially when comparing binary systems on n-ary gold standards.
- Annotation style mismatches between gold datasets and system outputs (e.g., verb-centric vs. nominal triggers) can cause false negatives that are not true extraction failures.
- Noisy text artifacts (HTML encodings, unfinished sentences) in datasets like WEB-500 disrupt intermediate parsing steps, artificially lowering recall.

## Evidence (verbatim from paper)

> In a quantitative evaluation we report precision, recall and $F_{2}$ scores on all four data sets. We conduct our experiments with an exact (a) and relaxed (b) containment match strategy.

## Citation

```bibtex
@misc{schneider2017analysing,
  title={Analysing Errors of Open Information Extraction Systems},
  author={Rudolf Schneider et al. (2017)},
  year={2017},
  note={arXiv:1707.07499}
}
```

- arXiv: 1707.07499