Ukrainian Text Classification Eval

---
name: ukrainian-text-classification-eval
description: Evaluates cross-lingual transfer methods for Ukrainian text classification across toxicity, formality, and natural language inference tasks. It compares translation-based baselines, LLM prompting, and adapter/fine-tuning approaches on both machine-translated and semi-natural Ukrainian test sets. Use when the user wants to benchmark on Ukrainian Toxicity (Translated & Semi-natural), Ukrainian Formality (Translated & Semi-natural), Ukrainian NLI (Translated & Semi-natural), or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.02043
  bibtex_key: dementieva2024crosslingual
  confidence: high
---

# ukrainian-text-classification-eval

> Cross-lingual Text Classification Transfer: The Case of Ukrainian — Dementieva et al. (2024) (arXiv:2404.02043, 2024)

## What this evaluates

Evaluates cross-lingual transfer methods for Ukrainian text classification across toxicity, formality, and natural language inference tasks. It compares translation-based baselines, LLM prompting, and adapter/fine-tuning approaches on both machine-translated and semi-natural Ukrainian test sets.

## Datasets

- **Ukrainian Toxicity (Translated & Semi-natural)** — total ?; splits: test (-1)
- **Ukrainian Formality (Translated & Semi-natural)** — total ?; splits: test (-1)
- **Ukrainian NLI (Translated & Semi-natural)** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels out of all predictions.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions.
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positive instances.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Raw Ukrainian text for classification tasks. For backtranslation baselines, English text is provided to the translation system first.

**Output**: Predicted class label: 'toxic' or 'non-toxic' for toxicity; 'formal' or 'informal' for formality; 'entailment', 'neutral', or 'contradiction' for NLI.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    correct = sum(p == g for p, g in zip(preds, golds))
    acc = correct / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'pos')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'pos' and g != 'pos')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'pos' and g == 'pos')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1
```

## Common pitfalls

- Backtranslation baseline is only evaluated on the semi-natural test set, as the translated test set column shows '—' for this method.
- Toxicity labels from Jigsaw are collapsed into a binary 'toxic'/'non-toxic' scheme, which may obscure nuances present in the original multi-class dataset.
- Semi-natural test sets are constructed from heterogeneous sources (tweets, legal acts, fiction), potentially introducing domain bias not present in the translated benchmarks.

## Evidence (verbatim from paper)

> |  | Acc | Pr | Re | F1 | Acc | Pr | Re | F1 |
Табл. 3: Ukrainian Texts Classification results. We divide methods into two groups – not requiring and requiring fine-tuning. Then, bold numbers denote the best results within the methods group and a test set, underline – overall best scores for the task.

## Citation

```bibtex
@misc{dementieva2024crosslingual,
  title={Cross-lingual Text Classification Transfer: The Case of Ukrainian},
  author={Dementieva et al. (2024)},
  year={2024},
  note={arXiv:2404.02043}
}
```

- arXiv: 2404.02043