Semeval2023 Task12 Eval

---
name: semeval2023-task12-eval
description: Multilingual sentiment classification across low-resource African languages, including zero-shot generalization to unseen languages. Use when the user wants to benchmark on SemEval-2023 Task 12, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.02607
  bibtex_key: homskiy2023dnnatsemeval
  confidence: high
---

# semeval2023-task12-eval

> DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning — Homskiy et al. (2023) (arXiv:2305.02607, 2023)

## What this evaluates

Multilingual sentiment classification across low-resource African languages, including zero-shot generalization to unseen languages.

## Datasets

- **SemEval-2023 Task 12** — total ?; splits: train (-1), validation (-1), test (-1); repo https://github.com/Daniil153/SemEval2023_Task12

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score (harmonic mean of precision and recall across all sentiment classes).

## Input / output format

**Input**: Raw text string in one of the target African languages.

**Output**: Predicted sentiment label (e.g., positive, negative, neutral).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    precisions, recalls = [], []
    for label in unique_labels:
        tp = sum(p == label and g == label for p, g in zip(predictions, gold))
        fp = sum(p == label and g != label for p, g in zip(predictions, gold))
        fn = sum(p != label and g == label for p, g in zip(predictions, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        precisions.append(prec)
        recalls.append(rec)
    avg_prec = sum(precisions) / len(precisions)
    avg_rec = sum(recalls) / len(recalls)
    return 2 * (avg_prec * avg_rec) / (avg_prec + avg_rec) if (avg_prec + avg_rec) > 0 else 0.0
```

## Common pitfalls

- The paper uses StratifiedKFold 5-fold cross-validation on the training set and ensembles predictions, rather than a single train/test split.
- Zero-shot tracks (Tigrinya, Oromo) are evaluated separately without fine-tuning on those languages, requiring model selection based on the target metric from other languages.
- Text preprocessing was tested but ultimately discarded as it did not improve F1 scores.

## Evidence (verbatim from paper)

> For every track except the last two (zero-shot), we employed StratifiedKFold (Pedregosa et al., 2011) with 5 folds to partition the training sample into training and validation sets. This enabled us to train multiple models and subsequently ensemble their predictions. ... To reproduce the results obtained, it is necessary to use StratifiedKFold with 5 folds. Train the model on each training fold. ... Table 1: Results of the DN team in all tracks of the competition ... Our F1

## Citation

```bibtex
@misc{homskiy2023dnnatsemeval,
  title={DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning},
  author={Homskiy et al. (2023)},
  year={2023},
  note={arXiv:2305.02607}
}
```

- arXiv: 2305.02607