Fin Bench V2 Eval

---
name: fin-bench-v2-eval
description: Evaluates large language models on Finnish language capabilities across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, truthfulness, and alignment. It probes both multiple-choice and generative capabilities under varying prompt formulations (cloze vs. multiple-choice) and shot configurations. Use when the user wants to benchmark on FIN-bench-v2, or asks about evaluating this task. Reports normalized accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.13330
  bibtex_key: [REDACTED]
  confidence: high
---

# fin-bench-v2-eval

> FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models — Kytöniemi et al. (2025) (arXiv:2512.13330, 2025)

## What this evaluates

Evaluates large language models on Finnish language capabilities across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, truthfulness, and alignment. It probes both multiple-choice and generative capabilities under varying prompt formulations (cloze vs. multiple-choice) and shot configurations.

## Datasets

- **FIN-bench-v2** — total ?; splits: test (-1); repo https://github.com/TurkuNLP/FIN-bench-v2

## Metrics

- `normalized accuracy` **(primary)** — range: [0, 1]
  - Calculated as the number of correct answers divided by the total number of answer options, or via conditional likelihood comparison over predefined options for multiple-choice tasks.
- `MC2 accuracy` — range: [0, 1]
  - A separate metric used specifically for the TruthfulQA MC2 task variant, returning a distinct accuracy score.
- `Exact Match / F1` — range: [0, 1]
  - Exact Match measures perfect string overlap with reference answers; F1 computes token-level precision and recall for generative QA tasks like SQuAD.
- `BLEU / ROUGE` — range: [0, 1]
  - Automated similarity metrics used to evaluate free-form text generation against reference answers, particularly for TruthfulQA.

## Input / output format

**Input**: Finnish language prompts formatted as either Cloze Formulation (CF) or Multiple-Choice Formulation (MCF), with five distinct prompt variants per task. Context is provided in 0-shot, 1-shot, or 5-shot configurations.

**Output**: For multiple-choice tasks: the model computes conditional likelihoods over predefined options to identify the correct continuation. For generative tasks: free-form text output compared against reference answers.

## Scoring recipe

```python
def compute_normalized_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_gen_metrics(predictions, references):
    em = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip()) / len(references)
    f1 = token_f1(predictions, references)
    return em, f1
```

## Common pitfalls

- Prompt sensitivity: minor wording changes across the five prompt variants can cause substantial score variance (e.g., Belebele task).
- Formulation preference: some instruction-tuned models (e.g., Poro family) degrade in MCF compared to CF, treating option lists as noise rather than helpful constraints.
- Translation artifacts: models trained on synthetic machine-translated data may artificially outperform those trained on human-authored Finnish data due to shared stylistic features.

## Evidence (verbatim from paper)

> For multiple-choice tasks, we report the normalized accuracy score. The only exception is the TruthfulQA MC2 task, which returns a separate metric commonly referred to as the MC2 accuracy.

## Citation

```bibtex
@misc{kytoniemi2025finbenchv2,
  title={FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models},
  author={Kytöniemi et al. (2025)},
  year={2025},
  note={arXiv:2512.13330}
}
```

- arXiv: 2512.13330