Refind Relation Extraction Eval

---
name: refind-relation-extraction-eval
description: This benchmark evaluates a model's ability to perform relation extraction on complex, domain-specific financial documents (SEC 10-X filings). It specifically probes challenges such as numerical inference, semantic ambiguity between similar relation types, and directional dependency resolution in long-range financial text. Use when the user wants to benchmark on REFiND, or asks about evaluating this task. Reports micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.18322
  bibtex_key: kaur2023refind
  confidence: high
---

# refind-relation-extraction-eval

> REFiND: Relation Extraction Financial Dataset — Kaur et al. (2023) (arXiv:2305.18322, 2023)

## What this evaluates

This benchmark evaluates a model's ability to perform relation extraction on complex, domain-specific financial documents (SEC 10-X filings). It specifically probes challenges such as numerical inference, semantic ambiguity between similar relation types, and directional dependency resolution in long-range financial text.

## Datasets

- **REFiND** — total 29000; splits: train (-1), val (-1), test (-1)

## Metrics

- `micro-F1` **(primary)** — range: percent
  - Standard micro-averaged F1 score calculated globally across all entity pair groups and the entire dataset.
- `macro-F1` — range: percent
  - Standard macro-averaged F1 score calculated globally across all entity pair groups and the entire dataset.

## Input / output format

**Input**: Financial text sentences with entity mentions marked by special tokens: [E1], / / E1, [E2], and / / E2, along with an ordered pair of entity mentions (e1, e2).

**Output**: A single relation label from a predefined set of 22 relations across 8 entity types.

## Scoring recipe

```python
def compute_f1(preds, gold, average='micro'):
    # preds and gold are lists of relation labels
    # Calculate precision and recall per class or globally based on average
    # Return F1 score as percentage
    tp = sum(1 for p, g in zip(preds, gold) if p == g)
    fp = sum(1 for p, g in zip(preds, gold) if p != g and p in gold)
    fn = sum(1 for p, g in zip(preds, gold) if p != g and g in preds)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Models often fail numerical inference, treating percentages (e.g., 31% vs 100% equity) as regular tokens rather than performing logical checks for acquisition thresholds.
- Semantic ambiguity causes confusion between closely related relations like member_of, employee_of, and founder_of for the same entity pairs.
- Directional ambiguity leads to errors swapping acquired_by and subsidiary_of despite entity ordering cues.

## Evidence (verbatim from paper)

> To ensure a comprehensive evaluation of the benchmarks, we report both micro- and macro-F1 score metrics (evaluation details in Appendix A.3). We also evaluate the performance of each model on each entity pair group and the entire REFinD dataset to obtain a more detailed understanding of the models' strengths and weaknesses.

## Citation

```bibtex
@misc{kaur2023refind,
  title={REFiND: Relation Extraction Financial Dataset},
  author={Kaur et al. (2023)},
  year={2023},
  note={arXiv:2305.18322}
}
```

- arXiv: 2305.18322