Wire57 Eval

---
name: wire57-eval
description: Evaluates Open Information Extraction systems on their ability to accurately extract relational tuples from text. It probes token-level precision and recall by matching predicted arguments and relations against a fine-grained, manually annotated gold standard. Use when the user wants to benchmark on WiRe57, or asks about evaluating this task. Reports token-weighted F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.08962
  bibtex_key: lechelle2018wire57
  confidence: high
---

# wire57-eval

> WiRe57 : A Fine-Grained Benchmark for Open Information Extraction — Léchelle et al. (2018) (arXiv:1809.08962, 2018)

## What this evaluates

Evaluates Open Information Extraction systems on their ability to accurately extract relational tuples from text. It probes token-level precision and recall by matching predicted arguments and relations against a fine-grained, manually annotated gold standard.

## Datasets

- **WiRe57** — total 347; splits: test (347); repo https://github.com/rali-udem/WiRe57

## Metrics

- `token-weighted F1` **(primary)** — range: [0, 1]
  - System-level precision and recall are computed as the sum of shared words across all matched tuple parts divided by the total length of predicted and gold tuples respectively. F1 is the harmonic mean of these system-level P and R. Matching is done greedily by max F1 score.

## Input / output format

**Input**: A sentence and its corresponding gold standard OIE tuples.

**Output**: A list of predicted OIE tuples in the format (arg1; rel; arg2; arg3; arg4; arg5).

## Scoring recipe

```python
def compute_wire57_f1(preds, gold):
    matches = []
    used_gold = set()
    for p in preds:
        best_f1, best_g = -1, None
        for j, g in enumerate(gold):
            if j in used_gold: continue
            if shares_core_words(p, g): # arg1, rel, arg2
                p_score = shared_words(p, g) / len(p)
                g_score = shared_words(p, g) / len(g)
                f1 = 2 * p_score * g_score / (p_score + g_score)
                if f1 > best_f1: best_f1, best_g = f1, j
        if best_g is not None:
            matches.append((p, gold[best_g]))
            used_gold.add(best_g)
    total_pred_len = sum(len(p) for p in preds)
    total_gold_len = sum(len(g) for g in gold)
    shared = sum(shared_words(p, g) for p, g in matches)
    prec = shared / total_pred_len if total_pred_len else 0
    rec = shared / total_gold_len if total_gold_len else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) else 0
```

## Common pitfalls

- Overlong extractions can artificially inflate recall if not penalized by token-level scoring.
- Coreference resolution and inferred words are explicitly excluded from the gold standard used for scoring, so systems attempting to resolve them will be unfairly penalized.
- Greedy matching order affects final scores; pairs with higher F1 are matched first, which may leave suboptimal matches for remaining tuples.

## Evidence (verbatim from paper)

> Our scorer computes precision and recall of a system’s predicted tuples at the token level. Precision is, briefly put, the proportion of extracted words that are found in the reference. Recall is the proportion of reference words found in the systems’ predictions. We match predicted tuples with reference ones by greedily removing from the potential match pool the pair with maximum F1 score, until no remaining tuples match. Hence, the overall performance metrics of an extractor are its token-weighted precision and recall over all tuples, i.e. precision_sys = ... recall_sys = ... F1_sys = ...

## Citation

```bibtex
@misc{lechelle2018wire57,
  title={WiRe57 : A Fine-Grained Benchmark for Open Information Extraction},
  author={Léchelle et al. (2018)},
  year={2018},
  note={arXiv:1809.08962}
}
```

- arXiv: 1809.08962