Kyrgyz Sst2 Eval

---
name: kyrgyz-sst2-eval
description: Evaluates sentiment classification capability on Kyrgyz language text. It measures how effectively a model can distinguish between positive and negative sentiments using a manually annotated benchmark dataset. Use when the user wants to benchmark on kyrgyz-sst2, or asks about evaluating this task. Reports F1-score (Weighted).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.20182
  bibtex_key: metinov2025kyrgyzbert
  confidence: high
---

# kyrgyz-sst2-eval

> KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP — Metinov et al. (2025) (arXiv:2511.20182, 2025)

## What this evaluates

Evaluates sentiment classification capability on Kyrgyz language text. It measures how effectively a model can distinguish between positive and negative sentiments using a manually annotated benchmark dataset.

## Datasets

- **kyrgyz-sst2** — total ?; splits: test (-1)

## Metrics

- `F1-score (Weighted)` **(primary)** — range: [0, 1]
  - Weighted F1-score, which computes the F1-score for each sentiment class and averages them weighted by the number of true instances for each class.

## Input / output format

**Input**: Kyrgyz language text sentences.

**Output**: Predicted sentiment label (positive or negative) per sentence.

## Scoring recipe

```python
def weighted_f1(preds, gold):
    classes = sorted(set(preds) | set(gold))
    f1s, weights = [], []
    for c in classes:
        tp = sum(p == c and g == c for p, g in zip(preds, gold))
        fp = sum(p == c and g != c for p, g in zip(preds, gold))
        fn = sum(p != c and g == c for p, g in zip(preds, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        weights.append(sum(1 for g in gold if g == c))
    return sum(f * w for f, w in zip(f1s, weights)) / sum(weights)
```

## Common pitfalls

- Zero-shot base models perform near random chance (~0.32 F1), so readers must not assume pre-trained multilingual models work out-of-the-box for Kyrgyz.
- The benchmark reports weighted F1, not accuracy; comparing against accuracy-based baselines or misinterpreting the metric will lead to incorrect conclusions about model performance.

## Evidence (verbatim from paper)

> The results clearly indicate that task-specific finetuning is essential, as the base models perform poorly, with F1-scores near the level of a random classifier (Table [II]). Our primary finding is that finetuned ‘KyrgyzBert‘ achieves a strong F1-score of 0.8280, confirming that a compact, monolingual model is highly effective for downstream tasks in Kyrgyz.

## Citation

```bibtex
@misc{metinov2025kyrgyzbert,
  title={KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP},
  author={Metinov et al. (2025)},
  year={2025},
  note={arXiv:2511.20182}
}
```

- arXiv: 2511.20182