Nerel Eval

---
name: nerel-eval
description: Evaluates models on nested named entity recognition and relation extraction in Russian. It probes the ability to identify overlapping/contained entities and classify semantic relations between them, including cross-sentence and nested relations. Use when the user wants to benchmark on NEREL, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.13112
  bibtex_key: loukachevitch2021nerel
  confidence: high
---

# nerel-eval

> NEREL: A Russian Dataset with Nested Named Entities, Relations and Events — Loukachevitch et al. (2021) (arXiv:2108.13112, 2021)

## What this evaluates

Evaluates models on nested named entity recognition and relation extraction in Russian. It probes the ability to identify overlapping/contained entities and classify semantic relations between them, including cross-sentence and nested relations.

## Datasets

- **NEREL** — total 933; splits: train (746), dev (94), test (93); repo https://github.com/nerel-ds/NEREL

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R). Reported as a percentage.

## Input / output format

**Input**: Full document text or single sentence, tokenized into words.

**Output**: For NER: set of nested entity spans with types. For RE: set of relation triples (head entity, relation type, tail entity) including nested and cross-sentence relations.

## Scoring recipe

```python
def calculate_f1(predictions, gold):
    tp = len(set(predictions) & set(gold))
    fp = len(set(predictions) - set(gold))
    fn = len(set(gold) - set(predictions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100
```

## Common pitfalls

- Models must handle nested/overlapping entities, which breaks standard flat NER pipelines.
- Relation extraction includes cross-sentence/document-level relations, requiring document-level context rather than just single-sentence inputs.
- The MRC approach processes one sentence at a time, which may miss document-level relations.

## Evidence (verbatim from paper)

> Table 3 presents the results of nested NER on the NEREL dataset. The results show, that (i) contextualized BERT-based models outperform models based on static word representations; (ii) the Biaffine model is superior to the Pyramid model; (iii) the results of MRC approach surpass nested NER models’ results, most likely, due to the effective usage of additional external information. Table 4 presents the results of relation extraction on the NEREL dataset, grouped with respect to three relation types. | Method | P | R | F1 |

## Citation

```bibtex
@misc{loukachevitch2021nerel,
  title={NEREL: A Russian Dataset with Nested Named Entities, Relations and Events},
  author={Loukachevitch et al. (2021)},
  year={2021},
  note={arXiv:2108.13112}
}
```

- arXiv: 2108.13112