Ielm Eval

---
name: ielm-eval
description: Evaluates the zero-shot open information extraction (OIE) capability of pre-trained language models by measuring their ability to extract subject-predicate-object triples from text without task-specific training or fine-tuning. It probes whether LMs inherently store rich, open-world relational knowledge that can be accessed via attention mechanisms. Use when the user wants to benchmark on CaRB, Re-OIE2016, TAC KBP-OIE, Wikidata-OIE, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.14128
  bibtex_key: wang2022ielm
  confidence: high
---

# ielm-eval

> IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models — Wang et al. (2022) (arXiv:2210.14128, 2022)

## What this evaluates

Evaluates the zero-shot open information extraction (OIE) capability of pre-trained language models by measuring their ability to extract subject-predicate-object triples from text without task-specific training or fine-tuning. It probes whether LMs inherently store rich, open-world relational knowledge that can be accessed via attention mechanisms.

## Datasets

- **CaRB** — total ?; splits: test (-1)
- **Re-OIE2016** — total ?; splits: test (-1)
- **TAC KBP-OIE** — total ?; splits: test (-1)
- **Wikidata-OIE** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R).
- `Precision` — range: [0, 1]
  - Ratio of correctly extracted triples to all predicted triples.
- `Recall` — range: [0, 1]
  - Ratio of correctly extracted triples to all gold triples.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring ranking quality across thresholds.

## Input / output format

**Input**: A sentence or document from which relational knowledge must be extracted.

**Output**: A set of triples (subject, predicate, object) representing the extracted relations.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_set = set(tuple(sorted(t)) for t in predictions)
    gold_set = set(tuple(sorted(t)) for t in gold)
    tp = len(pred_set & gold_set)
    p = tp / len(pred_set) if pred_set else 0
    r = tp / len(gold_set) if gold_set else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f1
```

## Common pitfalls

- spaCy noun chunker errors account for approximately 30% of argument extraction mistakes.
- Predicates located outside the argument pairs are not handled by the current extraction pipeline.
- Zero-shot evaluation uses no task-specific training or fine-tuning, measuring only inherent LM knowledge.
- Performance varies significantly between standard OIE datasets (CaRB, Re-OIE2016) and factual OIE datasets (TAC KBP-OIE, Wikidata-OIE).

## Evidence (verbatim from paper)

> The best zero-shot OIE system based on GPT-2XL obtains a +2.6% and a +3.1% absolute F1 improvement on TAC KBP-OIE and Wikidata-OIE respectively over the previous supervised state-of-the-art.

## Citation

```bibtex
@misc{wang2022ielm,
  title={IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2210.14128}
}
```

- arXiv: 2210.14128