Norwegian Nlp Benchmark Eval

---
name: norwegian-nlp-benchmark-eval
description: Evaluates contextualized language models on core Norwegian NLP tasks, probing part-of-speech tagging, named entity recognition, sentiment analysis, and negation detection across Bokmål and Nynorsk dialects. Use when the user wants to benchmark on Norwegian Dependency Treebank (NDT), NorNE, NoReC_fine, NoReC_sentence, NoReC_neg, or asks about evaluating this task. Reports Accuracy, strict micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.06546
  bibtex_key: kutuzov2021largescale
  confidence: high
---

# norwegian-nlp-benchmark-eval

> Large-Scale Contextualised Language Modelling for Norwegian — Kutuzov et al. (2021) (arXiv:2104.06546, 2021)

## What this evaluates

Evaluates contextualized language models on core Norwegian NLP tasks, probing part-of-speech tagging, named entity recognition, sentiment analysis, and negation detection across Bokmål and Nynorsk dialects.

## Datasets

- **Norwegian Dependency Treebank (NDT)** — total ?; splits: train (-1), dev (-1), test (-1)
- **NorNE** — total ?; splits: train (-1), dev (-1), test (-1)
- **NoReC_fine** — total ?; splits: train (-1), dev (-1), test (-1)
- **NoReC_sentence** — total ?; splits: train (-1), dev (-1), test (-1)
- **NoReC_neg** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Calculated as the number of correct token predictions divided by the total number of tokens in the test set.
- `strict micro F1` **(primary)** — range: [0, 1]
  - Harmonic mean of micro-precision and micro-recall, strictly requiring exact match of both entity type and boundary surface string for a prediction to count as correct.
- `Targeted F1` — range: [0, 1]
  - Requires correctly extracting a target span (strict) and its polarity, averaged over the test set.
- `Macro F1` — range: [0, 1]
  - Harmonic mean of precision and recall calculated per class and then averaged across all classes.
- `Full negation F1` — range: [0, 1]
  - Evaluates the combined prediction of negation cues and their in-sentence scopes using standard F1 scoring.

## Input / output format

**Input**: Norwegian text sequences (sentences or documents), optionally augmented with POS tags, lemmas, or character embeddings for graph-based tasks.

**Output**: Task-specific predictions: token-level tags (POS), BIO sequences (NER), sentiment graph structures, or sentence-level polarity labels.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_strict_micro_f1(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- ELMo models are evaluated without fine-tuning (frozen embeddings + classifier), while BERT models are fully fine-tuned, creating an unfair comparison of adaptation capacity.
- Bokmål and Nynorsk are treated as separate evaluation splits; models trained predominantly on Bokmål often show significant performance drops on Nynorsk data.

## Evidence (verbatim from paper)

> The evaluation metrics here is 'strict' micro F1, requiring both the correct entity type and exact match of boundary surface string.

## Citation

```bibtex
@misc{kutuzov2021largescale,
  title={Large-Scale Contextualised Language Modelling for Norwegian},
  author={Kutuzov et al. (2021)},
  year={2021},
  note={arXiv:2104.06546}
}
```

- arXiv: 2104.06546