Histnero Eval

---
name: histnero-eval
description: Evaluates the ability of language models to recognize and classify named entities (PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE) in historical Romanian newspaper texts across four distinct geographical regions. Use when the user wants to benchmark on HistNERo, or asks about evaluating this task. Reports strict F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.00155
  bibtex_key: avram2024histnero
  confidence: high
---

# histnero-eval

> HistNERo: Historical Named Entity Recognition for the Romanian Language — Avram et al. (2024) (arXiv:2405.00155, 2024)

## What this evaluates

Evaluates the ability of language models to recognize and classify named entities (PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE) in historical Romanian newspaper texts across four distinct geographical regions.

## Datasets

- **HistNERo** — total ?; splits: test (-1)

## Metrics

- `strict F1-score` **(primary)** — range: percent
  - Computes the harmonic mean of precision and recall for exact matches of named entity boundaries and types. Strict F1 requires both the span boundaries and the entity type to match exactly between prediction and gold.
- `accuracy` — range: percent
  - Token-level accuracy measuring the proportion of correctly labeled tokens out of the total number of tokens.

## Input / output format

**Input**: Raw text from historical Romanian newspapers, tokenized for NER processing.

**Output**: Token-level sequence labels for five entity types: PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE.

## Scoring recipe

```python
def strict_f1(preds, golds):
    pred_ents = set(extract_entities(preds))
    gold_ents = set(extract_entities(golds))
    tp = len(pred_ents & gold_ents)
    fp = len(pred_ents - gold_ents)
    fn = len(gold_ents - pred_ents)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Strict F1 requires exact boundary and type matching; partial overlaps or type mismatches are counted as errors.
- Performance varies significantly across regions (e.g., Transylvania vs. Moldavia), so reporting only the aggregate score masks regional linguistic disparities.
- Domain adaptation techniques (gradient/loss reversal) are applied during training, not just evaluation, which affects the reported metrics.

## Evidence (verbatim from paper)

> We measure the accuracy and the strict F1-score when we evaluate based on the regions and only the strict F1-score when we evaluate based on the named entities.

## Citation

```bibtex
@misc{avram2024histnero,
  title={HistNERo: Historical Named Entity Recognition for the Romanian Language},
  author={Avram et al. (2024)},
  year={2024},
  note={arXiv:2405.00155}
}
```

- arXiv: 2405.00155