Fin Bench Eval

---
name: fin-bench-eval
description: Evaluates Finnish large language models on a curated suite of 11 tasks spanning arithmetic, reasoning, general knowledge, emotion classification, and linguistic understanding. It probes few-shot generalization and task-specific capabilities in a low-resource language setting. Use when the user wants to benchmark on FIN-bench, or asks about evaluating this task. Reports mean accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.05640
  bibtex_key: [REDACTED]
  confidence: high
---

# fin-bench-eval

> FinGPT: Large Generative Models for a Small Language — Luukkonen et al. (2023) (arXiv:2311.05640, 2023)

## What this evaluates

Evaluates Finnish large language models on a curated suite of 11 tasks spanning arithmetic, reasoning, general knowledge, emotion classification, and linguistic understanding. It probes few-shot generalization and task-specific capabilities in a low-resource language setting.

## Datasets

- **FIN-bench** — total 3919; splits: test (3919)

## Metrics

- `mean accuracy` **(primary)** — range: [0, 1]
  - Accuracy is computed per task. For tasks with subtasks (Cause and effect, Arithmetic), accuracy is first averaged over subtasks, then averaged across all tasks to produce a single mean accuracy score.

## Input / output format

**Input**: Natural language questions or prompts in Finnish, presented in zero- to three-shot in-context learning format.

**Output**: Model-generated text responses, typically a direct answer or selected option corresponding to the prompt.

## Scoring recipe

```python
task_accuracies = []
for task in tasks:
    if task.has_subtasks:
        subtask_accs = [accuracy(task.subtask_i) for i in subtasks]
        task_acc = mean(subtask_accs)
    else:
        task_acc = accuracy(task)
    task_accuracies.append(task_acc)
return mean(task_accuracies)
```

## Common pitfalls

- Small models may show better zero-shot than few-shot performance due to pattern copying from preceding context rather than genuine reasoning.
- Performance can drop unexpectedly between 8B and 13B models due to overfitting on limited non-repeated text, not just scaling laws.
- Subtask accuracies must be averaged before computing the overall mean to prevent tasks with more examples from dominating the final score.

## Evidence (verbatim from paper)

> We evaluate models on FIN-bench in zero- to three-shot settings and summarize results using mean accuracy across all tasks. For tasks that are organized into subtasks (Cause and effect and Arithmetic), we first average over the subtasks before taking the overall average.

## Citation

```bibtex
@misc{luukkonen2023fingpt,
  title={FinGPT: Large Generative Models for a Small Language},
  author={Luukkonen et al. (2023)},
  year={2023},
  note={arXiv:2311.05640}
}
```

- arXiv: 2311.05640