Finnish Nlp Eval

---
name: finnish-nlp-eval
description: Evaluates Finnish NLP capabilities across four core tasks: part-of-speech tagging, named entity recognition, dependency parsing, and text classification on domain-specific and out-of-domain corpora. It probes a model's ability to handle morphologically rich language and varying text registers. Use when the user wants to benchmark on Finnish NLP Benchmarks, or asks about evaluating this task. Reports Average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.11639
  bibtex_key: chang2020bilingual
  confidence: high
---

# finnish-nlp-eval

> Towards Fully Bilingual Deep Language Modeling — Chang et al. (2020) (arXiv:2010.11639, 2020)

## What this evaluates

Evaluates Finnish NLP capabilities across four core tasks: part-of-speech tagging, named entity recognition, dependency parsing, and text classification on domain-specific and out-of-domain corpora. It probes a model's ability to handle morphologically rich language and varying text registers.

## Datasets

- **Finnish NLP Benchmarks** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `UPOS` — range: [0, 1]
  - Unambiguous Part-Of-Speech tag accuracy for POS tagging on Universal Dependencies treebanks.
- `F-score` — range: [0, 1]
  - Mention-level F-score for Named Entity Recognition using IOB annotations.
- `LAS` — range: [0, 1]
  - Labeled Attachment Score for dependency parsing (accuracy of predicted heads and relations).
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified documents for text classification.
- `Average` **(primary)** — range: [0, 1]
  - Algebraic mean of the four task scores (POS, NER, parsing, classification).

## Input / output format

**Input**: Tokenized Finnish text, typically truncated to at most 256 tokens.

**Output**: Task-specific predictions: POS tags, IOB entity tags, dependency heads/relations, or document class labels.

## Scoring recipe

```python
pos_score = upos_accuracy(y_true, y_pred)
ner_score = mention_level_f1(y_true, y_pred)
parse_score = labeled_attachment_score(y_true, y_pred)
cls_score = document_accuracy(y_true, y_pred)
average = (pos_score + ner_score + parse_score + cls_score) / 4
```

## Common pitfalls

- Uses gold segmentation for dependency parsing instead of predicted segmentation.
- Truncates all documents to 256 tokens, which may disadvantage models with longer context windows.
- Averages different metrics (UPOS, F1, LAS, Accuracy) without normalization, which can skew the final score.

## Evidence (verbatim from paper)

> The metric for POS tagging is UPOS, NER F-score, dependency parsing LAS, and text classification accuracy.

## Citation

```bibtex
@misc{chang2020bilingual,
  title={Towards Fully Bilingual Deep Language Modeling},
  author={Chang et al. (2020)},
  year={2020},
  note={arXiv:2010.11639}
}
```

- arXiv: 2010.11639