South Slavic Encoder Eval

---
name: south-slavic-encoder-eval
description: Evaluates multilingual encoder models on South Slavic languages (Croatian and Serbian) across three diverse NLP tasks: named entity recognition, parliamentary sentiment regression, and causal commonsense reasoning. Tests whether cost-efficient additional pretraining can match dedicated monolingual encoders without full from-scratch training. Use when the user wants to benchmark on hr500k, ReLDI-NormTagNER-hr, SETimes.SR, ReLDI-NormTagNER-sr, ParlaSent (HBS), COPA (Croatian & Serbian), or asks about evaluating this task. Reports macro F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.05428
  bibtex_key: ljubesic2024diet
  confidence: high
---

# south-slavic-encoder-eval

> Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining — Ljubešić et al. (2024) (arXiv:2404.05428, 2024)

## What this evaluates

Evaluates multilingual encoder models on South Slavic languages (Croatian and Serbian) across three diverse NLP tasks: named entity recognition, parliamentary sentiment regression, and causal commonsense reasoning. Tests whether cost-efficient additional pretraining can match dedicated monolingual encoders without full from-scratch training.

## Datasets

- **hr500k** — total 499635; splits: train (-1), dev (-1), test (-1)
- **ReLDI-NormTagNER-hr** — total 89855; splits: train (-1), dev (-1), test (-1)
- **SETimes.SR** — total 92271; splits: train (-1), dev (-1), test (-1)
- **ReLDI-NormTagNER-sr** — total 97673; splits: train (-1), dev (-1), test (-1)
- **ParlaSent (HBS)** — total 5200; splits: train (2600), test (2600)
- **COPA (Croatian & Serbian)** — total 1000; splits: train (400), dev (100), test (500)

## Metrics

- `macro F1 score` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all entity types and datasets, aggregated over 3 random runs.
- `R^2 score` — range: other
  - Coefficient of determination for ordinal regression (0–5 scale), averaged over 5 random runs.
- `accuracy` — range: [0, 1]
  - Percentage of correctly predicted plausible alternatives, averaged over 10 random runs.

## Input / output format

**Input**: Tokenized text sequences for NER; single sentences for sentiment; premise sentence paired with two candidate continuations for COPA.

**Output**: Token-level entity labels for NER; ordinal regression score (0–5) for sentiment; binary class index (0 or 1) selecting the more plausible alternative for COPA.

## Scoring recipe

```python
def score_ner(preds_list, gold_list):
    f1s = [macro_f1(p, g) for p, g in zip(preds_list, gold_list)]
    return mean(f1s)

def score_sentiment(preds, gold):
    return r_squared(preds, gold)

def score_copa(preds, gold):
    return accuracy(preds, gold)

# Aggregation over runs
ner_final = mean([score_ner(p, g) for p, g in run_3_times()])
sent_final = mean([score_sentiment(p, g) for p, g in run_5_times()])
copa_final = mean([score_copa(p, g) for p, g in run_10_times()])
```

## Common pitfalls

- Hyperparameter tuning is task- and dataset-specific: separate searches are performed for standard vs. non-standard NER datasets due to size/diversity differences, and sentiment uses a marked validation subset rather than a held-out dev set.
- Sentiment is treated as a regression task with an ordinal 0–5 scale, not a classification task, so accuracy is not reported for it.
- Results are not single-run values; they must be averaged over multiple random seeds (3 for NER, 5 for sentiment, 10 for COPA) to match the paper's reporting protocol.

## Evidence (verbatim from paper)

> For named entity recognition, we train and test each model three times and report aggregated results in the macro F1 score. For sentiment, we perform five runs, and report average $R^{2}$ scores. For COPA, we average over 10 runs and report the accuracy score.

## Citation

```bibtex
@misc{ljubesic2024diet,
  title={Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining},
  author={Ljubešić et al. (2024)},
  year={2024},
  note={arXiv:2404.05428}
}
```

- arXiv: 2404.05428