Dutch Book Review Sentiment Eval

---
name: dutch-book-review-sentiment-eval
description: Evaluates the effectiveness of Universal Language Model Fine-tuning (ULMFiT) versus traditional SVM classifiers on low-resource, domain-specific sentiment classification tasks using small training sets. Use when the user wants to benchmark on Dutch book reviews, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.00896
  bibtex_key: [REDACTED]
  confidence: medium
---

# dutch-book-review-sentiment-eval

> The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews — van der Burgh et al. (2019) (arXiv:1910.00896, 2019)

## What this evaluates

Evaluates the effectiveness of Universal Language Model Fine-tuning (ULMFiT) versus traditional SVM classifiers on low-resource, domain-specific sentiment classification tasks using small training sets.

## Datasets

- **Dutch book reviews** — total 15000; splits: train (15000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly classified book reviews out of the total number of reviews in the evaluation set, expressed as a percentage.

## Input / output format

**Input**: Raw Dutch book review text, tokenized by whitespace and padded to uniform length within batches.

**Output**: Binary sentiment label (positive or negative).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Hyperparameters were optimized separately for each training set size and fold, which may not generalize to other dataset sizes.
- TF-IDF features for the SVM baseline were trained on the same 105k documents used for ULMFiT pre-training, potentially inflating baseline performance compared to a strictly in-domain TF-IDF setup.
- Small dataset sizes (100–1600 samples) make results highly sensitive to random seed and fold partitioning.

## Evidence (verbatim from paper)

> For comparison we also trained two models, one SVM and one ULMFiT model, with manually tuned hyperparameters on all available book reviews in the training set (15k). These models achieved 93.84% (ULMFiT) and 89.16% (SVM).

## Citation

```bibtex
@misc{vanderburgh2019ulmfit,
  title={The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews},
  author={van der Burgh et al. (2019)},
  year={2019},
  note={arXiv:1910.00896}
}
```

- arXiv: 1910.00896