Core Fewshot Rc Eval

---
name: core-fewshot-rc-eval
description: Evaluates few-shot relation classification models on company and business entity relations, testing their ability to resolve entity ambiguity and adapt across domains using limited labeled examples. Use when the user wants to benchmark on CORE, or asks about evaluating this task. Reports Micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.12024
  bibtex_key: [REDACTED]
  confidence: high
---

# core-fewshot-rc-eval

> CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation — Borchert et al. (2023) (arXiv:2310.12024, 2023)

## What this evaluates

Evaluates few-shot relation classification models on company and business entity relations, testing their ability to resolve entity ambiguity and adapt across domains using limited labeled examples.

## Datasets

- **CORE** — total ?; splits: train (-1), test (-1); repo https://github.com/pnborchert/CORE

## Metrics

- `Micro F1` **(primary)** — range: [0, 1]
  - Standard micro-averaged F1 score: aggregates true positives, false positives, and false negatives across all classes before computing precision and recall.
- `Macro F1` — range: [0, 1]
  - Standard macro-averaged F1 score: computes the F1 score for each class independently and then takes the unweighted mean.

## Input / output format

**Input**: A text sentence containing two named entities, with a relation label from a set of N+1 relations (including NOTA).

**Output**: Predicted relation label for the entity pair.

## Scoring recipe

```python
def compute_f1(y_true, y_pred, average='micro'):
    from sklearn.metrics import f1_score
    return f1_score(y_true, y_pred, average=average, zero_division=0)
```

## Common pitfalls

- NOTA (None Of The Above) relation handling varies across datasets; this paper enforces a 50% NOTA rate for FewRel to match prior work.
- Episode sampling uses disjoint relation sets for train and test, meaning models never see test relations during training, testing true few-shot generalization.
- Micro vs Macro F1 can diverge significantly in imbalanced or NOTA-heavy settings; the paper reports both to capture class-level vs global performance.

## Evidence (verbatim from paper)

> We adopt the N-way K-shot evaluation setting, where the model is exposed to a limited number of instances $N	imes K$ during training. ... We evaluate RC models in the few-shot domain adaptation setting for CORE, FewRel and TA-CRED. ... Table 3: Micro F1 / Macro F1 for in-domain and out-of-domain models evaluated on CORE.

## Citation

```bibtex
@misc{borchert2023core,
  title={CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation},
  author={Borchert et al. (2023)},
  year={2023},
  note={arXiv:2310.12024}
}
```

- arXiv: 2310.12024