Dutch Llm Bench Eval

---
name: dutch-llm-bench-eval
description: Evaluates Dutch LLMs on reasoning, sentiment analysis, linguistic acceptability, world knowledge, and word sense disambiguation using zero-shot multiple-choice and binary classification tasks. Use when the user wants to benchmark on ARC (Dutch), DBRD, Dutch CoLA, Global MMLU (Dutch), XLWIC-NL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.15450
  bibtex_key: [REDACTED]
  confidence: high
---

# dutch-llm-bench-eval

> Fietje: An open, efficient LLM for Dutch — Vanroy et al. (2024) (arXiv:2412.15450, 2024)

## What this evaluates

Evaluates Dutch LLMs on reasoning, sentiment analysis, linguistic acceptability, world knowledge, and word sense disambiguation using zero-shot multiple-choice and binary classification tasks.

## Datasets

- **ARC (Dutch)** — total 7787; splits: test (-1)
- **DBRD** — total 2224; splits: test (2224)
- **Dutch CoLA** — total 2400; splits: test (2400); HF `GroNLP/dutch-cola`
- **Global MMLU (Dutch)** — total ?; splits: test (-1)
- **XLWIC-NL** — total 1004; splits: test (1004)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct predictions out of total test instances. Computed as (number of exact matches between model output and gold label) / (total instances).
- `confidence intervals` — range: other
  - Statistical interval computed over 5 independent evaluation runs with sampling (temperature=1, no top_p/top_k). Reflects model confidence and prediction variance across stochastic passes.

## Input / output format

**Input**: Zero-shot prompts in Dutch. Multiple-choice tasks present the question and predefined options. Binary tasks (sentiment, acceptability, WSD) present the query with explicit instruction to choose between two allowed labels.

**Output**: Constrained generation restricted to a predefined set of valid labels/options via the Outlines library. Models output exactly one allowed string (e.g., 'positive', 'negative', or the exact option text) without hallucination or need for post-processing.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    # predictions: list of 5 runs, each run is a list of model outputs
    accuracies = []
    for run_preds in predictions:
        correct = sum(1 for p, g in zip(run_preds, gold_labels) if p == g)
        accuracies.append(correct / len(gold_labels))
    mean_acc = sum(accuracies) / len(accuracies)
    ci_low = min(accuracies)
    ci_high = max(accuracies)
    return mean_acc, (ci_low, ci_high)
```

## Common pitfalls

- Using unconstrained generation causes label hallucination; the protocol strictly requires constrained decoding (Outlines) to ensure valid outputs.
- Confidence intervals are derived from 5 stochastic runs (temperature=1), not from a single deterministic pass or bootstrap over data.
- Benchmarks like ARC and Global MMLU are machine-translated and culturally adapted, which may introduce translation artifacts not present in native Dutch evaluation.

## Evidence (verbatim from paper)

> Each benchmark was run five times to compute the confidence intervals. In the benchmarks, sampling was enabled (no top p or top k selection, and temperature 1). ... Outlines was used as a backbone of the benchmarking suite ... This library enables you to constrain a model’s output by regular expressions, JSON schema, or simply a list of options. ... leading to clean benchmarking results that do not require any post-processing.

## Citation

```bibtex
@misc{vanroy2024fietje,
  title={Fietje: An open, efficient LLM for Dutch},
  author={Vanroy et al. (2024)},
  year={2024},
  note={arXiv:2412.15450}
}
```

- arXiv: 2412.15450