Sold Sentence Eval

---
name: sold-sentence-eval
description: This benchmark evaluates the capability of machine learning models to detect offensive language in Sinhala text. It probes binary text classification performance on a highly imbalanced dataset of Sinhala tweets, measuring how well models distinguish between offensive and non-offensive content. Use when the user wants to benchmark on SOLD, or asks about evaluating this task. Reports macro-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.00851
  bibtex_key: ranasinghe2022sold
  confidence: high
---

# sold-sentence-eval

> SOLD: Sinhala Offensive Language Dataset — Tharindu Ranasinghe et al. (arXiv:2212.00851, 2022)

## What this evaluates

This benchmark evaluates the capability of machine learning models to detect offensive language in Sinhala text. It probes binary text classification performance on a highly imbalanced dataset of Sinhala tweets, measuring how well models distinguish between offensive and non-offensive content.

## Datasets

- **SOLD** — total ?; splits: train (-1), test (-1); repo https://github.com/Sinhala-NLP/SOLD

## Metrics

- `macro-averaged F1-score` **(primary)** — range: [0, 1]
  - The arithmetic mean of the F1-scores computed for each class (OFF and NOT) independently. It treats all classes equally regardless of their support.

## Input / output format

**Input**: Raw Sinhala sentences (tweets) to be classified.

**Output**: Binary label indicating whether the input sentence is offensive ('OFF') or not offensive ('NOT').

## Scoring recipe

```python
def macro_f1(predictions, gold):
    classes = ['OFF', 'NOT']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- The dataset has a highly imbalanced label distribution, making accuracy misleading; macro-averaged F1 is required.
- Performance metrics are averaged across five different random seeds to ensure statistical reliability.

## Evidence (verbatim from paper)

> As the label distribution is highly imbalanced, we evaluate and compare the performance of the different models using macro-averaged F1-score. We further report per-class Precision (P), Recall (R), F1-score (F1), and weighted average.

## Citation

```bibtex
@misc{ranasinghe2022sold,
  title={SOLD: Sinhala Offensive Language Dataset},
  author={Tharindu Ranasinghe et al.},
  year={2022},
  note={arXiv:2212.00851}
}
```

- arXiv: 2212.00851