Norne Eval

---
name: norne-eval
description: Evaluates Named Entity Recognition (NER) performance on Norwegian text, testing the model's ability to identify and classify entity boundaries and types (PER, ORG, LOC, GPE, PROD, EVT, DRV) across Bokmål and Nynorsk variants. Use when the user wants to benchmark on NorNE, or asks about evaluating this task. Reports F1 (strict).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.12146
  bibtex_key: [REDACTED]
  confidence: high
---

# norne-eval

> NorNE: Annotating Named Entities for Norwegian — Jørgensen et al. (2019) (arXiv:1911.12146, 2019)

## What this evaluates

Evaluates Named Entity Recognition (NER) performance on Norwegian text, testing the model's ability to identify and classify entity boundaries and types (PER, ORG, LOC, GPE, PROD, EVT, DRV) across Bokmål and Nynorsk variants.

## Datasets

- **NorNE** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ltgoslo/norne

## Metrics

- `F1 (strict)` **(primary)** — range: [0, 1]
  - Entity-level F1 score requiring exact match on both predicted boundary and entity label. Follows SemEval 2013 task 9.1 strict evaluation scheme.

## Input / output format

**Input**: Tokenized sentence with word-level and character-level representations (CNN max-pooling + pre-trained word embeddings).

**Output**: Sequence of BIO-style tags (IOB2, IOB, IOBE, IOBS, or IOBES) for each token, indicating entity type and position.

## Scoring recipe

```python
def strict_entity_f1(predictions, gold):
    pred_entities = extract_entities(predictions) # (start, end, label)
    gold_entities = extract_entities(gold)
    tp = sum(1 for e in pred_entities if e in gold_entities)
    fp = len(pred_entities) - tp
    fn = len(gold_entities) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Evaluating at the token level instead of the required strict entity level.
- Interpreting higher F1 scores on coarser label sets (e.g., NorNE-6) as improved model capability, rather than a reduction in label ambiguity.
- Testing cross-standard (Bokmål to Nynorsk) performance without joint training, which artificially penalizes the model due to lexical distribution shifts.

## Evidence (verbatim from paper)

> For model evaluation we follow the scheme defined by the SemEval 2013 task 9.1 (Segura-Bedmar et al., 2013), using the re-implementation offered by David S. Batista. We report F1 for exact match on the entity level, i.e., both the predicted boundary and entity label must be correct. (This measure was dubbed strict in SemEval 2013 task 9.1.)

## Citation

```bibtex
@misc{jorgensen2019norne,
  title={NorNE: Annotating Named Entities for Norwegian},
  author={Jørgensen et al. (2019)},
  year={2019},
  note={arXiv:1911.12146}
}
```

- arXiv: 1911.12146