Dutch Financial Benchmark Eval

---
name: dutch-financial-benchmark-eval
description: Evaluates LLMs on domain-specific financial tasks in Dutch, including sentiment analysis, named entity recognition, relation extraction, query answering, and headline classification. It also tests cross-lingual adaptability by benchmarking the Dutch model on English financial data. Use when the user wants to benchmark on Dutch Financial Benchmark, English Financial Benchmark, or asks about evaluating this task. Reports zero-shot performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12835
  bibtex_key: noels2024dutchfinancialllm
  confidence: medium
---

# dutch-financial-benchmark-eval

> A Dutch Financial Large Language Model — Noels et al. (2024) (arXiv:2410.12835, 2024)

## What this evaluates

Evaluates LLMs on domain-specific financial tasks in Dutch, including sentiment analysis, named entity recognition, relation extraction, query answering, and headline classification. It also tests cross-lingual adaptability by benchmarking the Dutch model on English financial data.

## Datasets

- **Dutch Financial Benchmark** — total 500; splits: test (500); repo https://github.com/snoels/fingeit
- **English Financial Benchmark** — total ?; splits: test (-1); repo https://github.com/snoels/fingeit

## Metrics

- `zero-shot performance` **(primary)** — range: [0, 1]
  - Standard classification metrics (accuracy or F1) computed per task (SA, NER, RE, QA, headline classification). The exact metric variant is not specified in the text, but performance is reported comparatively across models.

## Input / output format

**Input**: Instruction prompt specifying a financial task (e.g., sentiment classification, NER, relation extraction, QA, or headline classification) paired with a Dutch or English financial text snippet.

**Output**: Task-specific answer (e.g., class label, extracted entities, relation triple, or free-text answer). Raw outputs are post-processed via an independent LLM to extract and format the answer according to the instruction.

## Scoring recipe

```python
predictions = post_process_llm_output(raw_output, task_type)
if task_type in ['SA', 'headline']:
    score = accuracy(predictions, gold_labels)
elif task_type in ['NER', 'RE']:
    score = f1_score(predictions, gold_labels, mode='strict')
elif task_type == 'QA':
    score = exact_match_or_f1(predictions, gold_answers)
return score
```

## Common pitfalls

- LLMs without task-specific fine-tuning frequently fail to follow instructions or generate outputs in the required format, necessitating an LLM-based post-processing step for reliable answer extraction.
- Cross-lingual evaluation requires careful handling of language mismatches (e.g., a Dutch model receiving English instructions), which the authors address via translation during the post-processing stage.

## Evidence (verbatim from paper)

> We assess the zero-shot performance of all the models against 500 randomly chosen test samples to decrease the computational cost. ... FinGEITje outperforms the other models across all tasks, indicating the effectiveness of language-specific instruction tuning for domain-specific tasks.

## Citation

```bibtex
@misc{noels2024dutchfinancialllm,
  title={A Dutch Financial Large Language Model},
  author={Noels et al. (2024)},
  year={2024},
  note={arXiv:2410.12835}
}
```

- arXiv: 2410.12835