Norwegian Transformer Eval

---
name: norwegian-transformer-eval
description: Evaluates the cross-lingual transfer and domain adaptation capabilities of a Norwegian BERT model against multilingual and monolingual baselines on token-level (NER/POS) and sequence-level (sentiment/political affiliation) classification tasks. Use when the user wants to benchmark on NorNE, CoNLL-2003, NoReC, Norwegian Parliament Speeches, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.09617
  bibtex_key: kummervold2021operationalizing
  confidence: medium
---

# norwegian-transformer-eval

> Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model — Kummervold et al. (2021) (arXiv:2104.09617, 2021)

## What this evaluates

Evaluates the cross-lingual transfer and domain adaptation capabilities of a Norwegian BERT model against multilingual and monolingual baselines on token-level (NER/POS) and sequence-level (sentiment/political affiliation) classification tasks.

## Datasets

- **NorNE** — total ?; splits: test (-1)
- **CoNLL-2003** — total ?; splits: test (-1)
- **NoReC** — total ?; splits: test (-1)
- **Norwegian Parliament Speeches** — total 6000; splits: test (6000)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard token-level or entity-level F1 score computed between predicted and gold entity/span labels. Macro-averaged across classes.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified text sequences out of the total number of sequences in the evaluation set.

## Input / output format

**Input**: Tokenized text sequences for token classification; raw text sentences or paragraphs for sequence classification.

**Output**: Token-level entity labels (e.g., B-PER, I-ORG) for NER; discrete class labels (e.g., sentiment polarity or political party affiliation) for sequence classification.

## Scoring recipe

```python
def compute_f1(gold, pred):
    tp = sum(1 for g, p in zip(gold, pred) if g == p and g != 'O')
    fp = sum(1 for g, p in zip(gold, pred) if g != p and p != 'O')
    fn = sum(1 for g, p in zip(gold, pred) if g != p and g != 'O')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

def compute_accuracy(gold, pred):
    return sum(1 for g, p in zip(gold, pred) if g == p) / len(gold)
```

## Common pitfalls

- Varying tag sets and annotation guidelines across languages make direct cross-lingual performance comparisons unreliable.
- Fine-tuning hyperparameters (learning rate, epochs) are highly dataset-dependent and require careful validation-set tuning.
- OCR noise in historical corpora can significantly degrade model performance if confidence-based filtering is not applied during pre-training.

## Evidence (verbatim from paper)

> The optimal learning rate and number of epochs mainly depend on the size of and variance in the training corpus... we could still compare the performance of our model against that of English-only and multilingual BERT models.

## Citation

```bibtex
@misc{kummervold2021operationalizing,
  title={Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model},
  author={Kummervold et al. (2021)},
  year={2021},
  note={arXiv:2104.09617}
}
```

- arXiv: 2104.09617