Fredo Eval

---
name: fredo-eval
description: Evaluates few-shot document-level relation extraction by testing a model's ability to identify relations between entity pairs across documents using limited support examples. It specifically probes domain adaptation capabilities, handling of class imbalance, and robustness to NOTA (none-of-the-above) distributions in realistic document-level settings. Use when the user wants to benchmark on FREDo, or asks about evaluating this task. Reports macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.02048
  bibtex_key: popovic2022fewshotdocumentlevelrelationextraction
  confidence: high
---

# fredo-eval

> Few-Shot Document-Level Relation Extraction — Popovic et al. (2022) (arXiv:2205.02048, 2022)

## What this evaluates

Evaluates few-shot document-level relation extraction by testing a model's ability to identify relations between entity pairs across documents using limited support examples. It specifically probes domain adaptation capabilities, handling of class imbalance, and robustness to NOTA (none-of-the-above) distributions in realistic document-level settings.

## Datasets

- **FREDo** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/nicpopovic/FREDo

## Metrics

- `macro F1` **(primary)** — range: percent
  - Macro-averaged F1 score computed across all relation types. Precision, recall, and F1 are calculated per relation type and then averaged.

## Input / output format

**Input**: A query document containing multiple entity mentions and candidate pairs, alongside a support set of K labeled documents for each relation type (including NOTA).

**Output**: A predicted relation type label for each candidate entity pair in the query document.

## Scoring recipe

```python
f1_scores = []
for rel in all_relations:
    tp = sum(1 for p, g in zip(preds, golds) if p == rel and g == rel)
    fp = sum(1 for p, g in zip(preds, golds) if p == rel and g != rel)
    fn = sum(1 for p, g in zip(preds, golds) if p != rel and g == rel)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    f1_scores.append(f1)
macro_f1 = sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- NOTA (none-of-the-above) is treated as a relation type, requiring explicit handling of class imbalance and negative examples rather than standard binary classification.
- Cross-domain evaluation involves distribution shift between support and query documents, invalidating standard prototype averaging assumptions.
- Repeated entity mentions in a single document require explicit pooling (e.g., mean pooling) before relation embedding formation, which significantly impacts performance.

## Evidence (verbatim from paper)

> In order to get sufficient coverage to calculate representative macro $F_{1}$ scores on the development set, we sample 4k episodes. ... Results for FREDo in-domain task. Reported results are macro averages across relation types.

## Citation

```bibtex
@misc{popovic2022fewshotdocumentlevelrelationextraction,
  title={Few-Shot Document-Level Relation Extraction},
  author={Popovic et al. (2022)},
  year={2022},
  note={arXiv:2205.02048}
}
```

- arXiv: 2205.02048