Scandinavian Sentiment Eval

---
name: scandinavian-sentiment-eval
description: Evaluates whether translating low-resource language data into English and applying large-scale English/multilingual models outperforms training native monolingual models for sentiment classification. It probes the efficiency and effectiveness of cross-lingual data reuse versus isolated language-specific pre-training. Use when the user wants to benchmark on Sentiment datasets (Swedish, Danish, Norwegian, Finnish, English), or asks about evaluating this task. Reports binary accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.10441
  bibtex_key: [REDACTED]
  confidence: medium
---

# scandinavian-sentiment-eval

> Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead? — Isbister et al. (2021) (arXiv:2104.10441, 2021)

## What this evaluates

Evaluates whether translating low-resource language data into English and applying large-scale English/multilingual models outperforms training native monolingual models for sentiment classification. It probes the efficiency and effectiveness of cross-lingual data reuse versus isolated language-specific pre-training.

## Datasets

- **Sentiment datasets (Swedish, Danish, Norwegian, Finnish, English)** — total ?; splits: test (-1)

## Metrics

- `binary accuracy` **(primary)** — range: [0, 1]
  - Standard binary classification accuracy: the proportion of correctly predicted labels out of the total number of instances.

## Input / output format

**Input**: Text instances in Swedish, Danish, Norwegian, Finnish, or English, either in their original language or machine-translated to English.

**Output**: Binary classification label (positive or negative sentiment).

## Scoring recipe

```python
def calculate_binary_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Finnish results degrade significantly due to poor machine translation quality, skewing cross-lingual comparisons.
- Performance gains may be driven by pre-training data scale rather than model architecture, requiring careful control of training corpus sizes.
- Cross-lingual evaluation shows anomalies (e.g., Norwegian model underperforming on English data) that require separate analysis from native-language results.

## Evidence (verbatim from paper)

> From this we report the binary accuracy, with the results for the BERT models available in Table 3, and the XLM-R results in Table 4.

## Citation

```bibtex
@misc{isbister2021stop,
  title={Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?},
  author={Isbister et al. (2021)},
  year={2021},
  note={arXiv:2104.10441}
}
```

- arXiv: 2104.10441