Smol Chrf Eval

---
name: smol-chrf-eval
description: Evaluates machine translation quality for 115 under-represented languages using professionally translated parallel data. It measures the improvement in character-level n-gram F-score (ChrF) after fine-tuning a baseline model on the Smol dataset compared to the unfine-tuned baseline. Use when the user wants to benchmark on SmolSent, SmolDoc, or asks about evaluating this task. Reports ChrF.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.12301
  bibtex_key: caswell2025smol
  confidence: high
---

# smol-chrf-eval

> SMOL: Professionally translated parallel data for 115 under-represented languages — Caswell et al. (2025) (arXiv:2502.12301, 2025)

## What this evaluates

Evaluates machine translation quality for 115 under-represented languages using professionally translated parallel data. It measures the improvement in character-level n-gram F-score (ChrF) after fine-tuning a baseline model on the Smol dataset compared to the unfine-tuned baseline.

## Datasets

- **SmolSent** — total ?; splits: test (-1)
- **SmolDoc** — total ?; splits: test (-1)

## Metrics

- `ChrF` **(primary)** — range: [0, 100]
  - Character n-gram F-score, computed as the harmonic mean of character-level precision and recall across n-grams (typically up to 6-grams).

## Input / output format

**Input**: Source sentence or document in English.

**Output**: Translated sentence or document in the target language.

## Scoring recipe

```python
def compute_chrf(hypotheses, references, n_max=6):
    hyp_ngrams = [get_char_ngrams(h, n_max) for h in hypotheses]
    ref_ngrams = [get_char_ngrams(r, n_max) for r in references]
    matches = sum(min(h.count(g), r.count(g)) for h, r in zip(hyp_ngrams, ref_ngrams))
    prec = matches / sum(len(h) for h in hyp_ngrams)
    rec = matches / sum(len(r) for r in ref_ngrams)
    return 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- The paper reports $\Delta_{FT}$ (baseline ChrF minus fine-tuned ChrF) as the primary sorting metric, not absolute ChrF.
- ChrF operates at the character level, making it sensitive to punctuation, casing, and tokenization differences.
- Results are presented per language pair; macro-averaging across languages may obscure performance on specific language families or scripts.

## Evidence (verbatim from paper)

> Results are sorted by the $\Delta_{FT}$, which is the ChrF of the Both model minus the ChrF of the finetuned Both model—in other words, how much the finetuning on Smol improved the baseline model.

## Citation

```bibtex
@misc{caswell2025smol,
  title={SMOL: Professionally translated parallel data for 115 under-represented languages},
  author={Caswell et al. (2025)},
  year={2025},
  note={arXiv:2502.12301}
}
```

- arXiv: 2502.12301