Unsupervised Relation Extraction Eval

---
name: unsupervised-relation-extraction-eval
description: Evaluates the ability of language models to perform unsupervised relation extraction by predicting relation labels or tokens from contextual text. It probes factual grounding and context-constrained generation capabilities across varying relation types and corpus sources. Use when the user wants to benchmark on T-REx, Google-RE, ZSRE, TACRED, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.06804
  bibtex_key: goswami2020unsupervised
  confidence: high
---

# unsupervised-relation-extraction-eval

> Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion — Goswami et al. (2020) (arXiv:2010.06804, 2020)

## What this evaluates

Evaluates the ability of language models to perform unsupervised relation extraction by predicting relation labels or tokens from contextual text. It probes factual grounding and context-constrained generation capabilities across varying relation types and corpus sources.

## Datasets

- **T-REx** — total 34039; splits: test (34039)
- **Google-RE** — total 5528; splits: test (5528)
- **ZSRE** — total 42635; splits: test (42635)
- **TACRED** — total 6357; splits: test (6357)

## Metrics

- `EM` — range: [0, 1]
  - Exact Match (EM) is calculated as the fraction of instances where the predicted relation token(s) exactly match the gold label.
- `F1` **(primary)** — range: [0, 1]
  - F1 score is the harmonic mean of precision and recall computed over token/label overlaps between predictions and gold labels. F1 = 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Context sentence containing entity mentions and a target relation type to predict.

**Output**: Predicted relation token(s) or label, constrained to vocabulary present in the supporting context.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    em_scores = [1.0 if p == g else 0.0 for p, g in zip(preds, golds)]
    precisions, recalls, f1s = [], [], []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p.split()), set(g.split())
        if not g_set: continue
        tp = len(p_set & g_set)
        precisions.append(tp / len(p_set) if p_set else 0)
        recalls.append(tp / len(g_set))
    f1s = [2 * (p * r) / (p + r) if p + r > 0 else 0 for p, r in zip(precisions, recalls)]
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1s)/len(f1s)}
```

## Common pitfalls

- Assuming supervised fine-tuning is used; the protocol emphasizes unsupervised/constrained cloze completion.
- Overlooking the distinction between single-token and multi-token extraction types across benchmarks.
- Ignoring the information-theoretic context filtering step required before scoring.

## Evidence (verbatim from paper)

> Table 3: We consider four benchmarks that vary with respect to the type of target extractions, the quality of context to relation alignment, and the underlying corpus.  
Table 4: Effect of token expansion on ZSRE dataset.
Method | ZSRE (EM, F1) | TACRED (EM, F1)
No expansion | 42.4 | 46.1 | 49.6 | 50.3

## Citation

```bibtex
@misc{goswami2020unsupervised,
  title={Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion},
  author={Goswami et al. (2020)},
  year={2020},
  note={arXiv:2010.06804}
}
```

- arXiv: 2010.06804