Lrebenc Eval

---
name: lrebenc-eval
description: Evaluates relation extraction models under low-resource conditions (8-shot, 10%, 100% training data) across diverse domains and languages. It probes few-shot learning capabilities, robustness to long-tailed class distributions, and the effectiveness of data augmentation and self-training strategies. Use when the user wants to benchmark on SemEval 2010 Task 8, TACREV, DialogRE, DuIE2.0, Wiki80, ChemProt, SciERC, CMeIE, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.10678
  bibtex_key: xu2022lrebenc
  confidence: high
---

# lrebenc-eval

> Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study — Xin Xu et al. (2022) (arXiv:2210.10678, 2022)

## What this evaluates

Evaluates relation extraction models under low-resource conditions (8-shot, 10%, 100% training data) across diverse domains and languages. It probes few-shot learning capabilities, robustness to long-tailed class distributions, and the effectiveness of data augmentation and self-training strategies.

## Datasets

- **SemEval 2010 Task 8** — total ?; splits: train (-1), test (-1); repo https://github.com/zjunlp/KnowPrompt/tree/master/dataset/semeval
- **TACREV** — total ?; splits: train (-1), test (-1); repo https://github.com/DFKI-NLP/tacrev
- **DialogRE** — total ?; splits: train (-1), test (-1); repo https://dataset.org/dialogre/
- **DuIE2.0** — total ?; splits: train (-1), test (-1); repo https://www.luge.ai/#/luge/dataDetail?id=5
- **Wiki80** — total ?; splits: train (-1), test (-1); repo https://github.com/thunlp/OpenNRE/blob/master/benchmark/download_wiki80.sh
- **ChemProt** — total ?; splits: train (-1), test (-1); repo https://github.com/ncbi-nlp/BLUE_Benchmark
- **SciERC** — total ?; splits: train (-1), test (-1); repo http://nlp.cs.washington.edu/sciIE/
- **CMeIE** — total ?; splits: train (-1), test (-1); repo https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414

## Metrics

- `Macro F1` **(primary)** — range: percent
  - Computes the F1 score for each relation class independently and averages them equally across all classes. Emphasizes performance on minority/long-tail classes.
- `Micro F1` — range: percent
  - Aggregates true positives, false positives, and false negatives across all classes globally before computing precision, recall, and F1. Reflects overall instance-level accuracy.

## Input / output format

**Input**: A unified JSON object per instance containing a text string and a relational triple (head entity, tail entity, and target relation label). Multi-triple instances are split into single-triple instances.

**Output**: A single predicted relation label corresponding to the entity pair in the text.

## Scoring recipe

```python
def compute_f1(preds, golds, mode='macro'):
    classes = set(golds)
    if mode == 'macro':
        f1s = []
        for c in classes:
            tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
            fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
            fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
        return sum(f1s) / len(f1s)
    else:
        tp = sum(1 for p, g in zip(preds, golds) if p == g)
        fp = sum(1 for p, g in zip(preds, golds) if p != g)
        fn = len(golds) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Performance is highly sensitive to random data splits in few-shot settings; the paper averages results over 5 random samples to mitigate variance.
- Datasets with multiple triples per instance are split into single-triple instances for evaluation, which alters the original instance distribution and requires careful preprocessing.
- No validation set is used during training to enforce true few-shot evaluation, which can cause training instability and requires strict adherence to the fixed 10-epoch training protocol.

## Evidence (verbatim from paper)

> Since the performance of head and tail classes varies a lot, we use both Macro F1 and Micro F1 together as the evaluation metrics. We conduct experiments in three settings with different proportions of training data to simulate different resource levels: 8-shot, 10% and 100%.

## Citation

```bibtex
@misc{xu2022lrebenc,
  title={Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study},
  author={Xin Xu et al. (2022)},
  year={2022},
  note={arXiv:2210.10678}
}
```

- arXiv: 2210.10678