Compactie Eval

---
name: compactie-eval
description: Evaluates Open Information Extraction systems on their ability to extract compact, clause-level facts from text. It measures precision, recall, and F1 using token-level matching against gold triples, with a focus on avoiding over-specific extractions and handling overlapping constituents. Use when the user wants to benchmark on CaRB, Wire57, BenchIE, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.02880
  bibtex_key: [REDACTED]
  confidence: high
---

# compactie-eval

> CompactIE: Compact Facts in Open Information Extraction — Fatahi Bayat et al. (2022) (arXiv:2205.02880, 2022)

## What this evaluates

Evaluates Open Information Extraction systems on their ability to extract compact, clause-level facts from text. It measures precision, recall, and F1 using token-level matching against gold triples, with a focus on avoiding over-specific extractions and handling overlapping constituents.

## Datasets

- **CaRB** — total 641; splits: test (577); repo https://github.com/dair-iitd/CaRB
- **Wire57** — total 57; splits: test (56); repo https://github.com/rali-udem/WiRe57
- **BenchIE** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Standard F1 score (harmonic mean of precision and recall) computed using dataset-specific token-level matching scoring functions.

## Input / output format

**Input**: Input sentences (pre-processed to remove conjunctions).

**Output**: A set of Open Information Extraction triples (subject, relation, object).

## Scoring recipe

```python
def compute_f1(pred_triples, gold_triples):
    tp = sum(1 for p in pred_triples if any(token_match(p, g) for g in gold_triples))
    fp = len(pred_triples) - tp
    fn = len(gold_triples) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- CaRB and Wire57 are incomplete benchmarks; gold sets do not include all acceptable surface realizations, which can artificially suppress recall.
- Scoring functions explicitly penalize over-specific extractions, requiring models to balance compactness with recall.
- Pre-processing steps (e.g., removing conjunctions, excluding triples with clauses inside constituents) must be applied consistently for fair comparison.

## Evidence (verbatim from paper)

> We report precision (P), recall (R), and F1 computed by these scoring functions. Wire57 contains more fine-grained extractions than the CaRB dataset and its scoring function is more rigorous for compact facts since it penalizes over-specific extractions. However, both CaRB and Wire57 scoring functions are based on token-level matching of system extractions against ground truth facts.

## Citation

```bibtex
@misc{fatahibayat2022compactie,
  title={CompactIE: Compact Facts in Open Information Extraction},
  author={Fatahi Bayat et al. (2022)},
  year={2022},
  note={arXiv:2205.02880}
}
```

- arXiv: 2205.02880