Kazsandra Eval

---
name: kazsandra-eval
description: Evaluates multilingual sentiment classification models on Kazakh customer reviews, probing their ability to handle code-switching, mixed scripts, and imbalanced class distributions across polarity and numerical score prediction tasks. Use when the user wants to benchmark on KazSAnDRA, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.19335
  bibtex_key: [REDACTED]
  confidence: high
---

# kazsandra-eval

> KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes — Yeshpanov et al. (2024) (arXiv:2403.19335, 2024)

## What this evaluates

Evaluates multilingual sentiment classification models on Kazakh customer reviews, probing their ability to handle code-switching, mixed scripts, and imbalanced class distributions across polarity and numerical score prediction tasks.

## Datasets

- **KazSAnDRA** — total 180064; splits: train (-1), val (-1), test (-1); repo https://github.com/IS2AI/KazSAnDRA

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1-score, calculated as the unweighted arithmetic mean of the F1-scores for each class. F1 for a class is 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Raw text customer reviews in Kazakh, often containing code-switching and mixed scripts (Cyrillic/Latin/Russian).

**Output**: For Polarity Classification (PC): binary sentiment label (positive/negative). For Score Classification (SC): integer rating from 1 to 5.

## Scoring recipe

```python
def macro_f1(predictions, gold):
    classes = sorted(set(predictions) | set(gold))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Using micro-averaging instead of macro-averaging, which would mask poor performance on minority classes in the imbalanced dataset.
- Assuming score classification (SC) will perform similarly to polarity classification (PC); SC is significantly harder (F1 ~0.39 vs 0.81) due to fine-grained rating prediction.
- Ignoring code-switching and mixed-script variations in Kazakh reviews, which can degrade model performance if not handled during preprocessing or fine-tuning.

## Evidence (verbatim from paper)

> Several conventional metrics were used to evaluate the performance of the models, including accuracy (A), precision (P), recall (R), and F${}_{1}$-score (F${}_{1}$). Given the imbalanced nature of the dataset, where all classes carry equal importance, we opted for macro-averaging, calculated from the arithmetic (i.e., unweighted) mean of all F${}_{1}$-scores per class, and thus ensuring equal treatment of all classes during the evaluation, resulting in a stronger penalty if the model performs worse on minority classes

## Citation

```bibtex
@misc{yeshpanov2024kazsandra,
  title={KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes},
  author={Yeshpanov et al. (2024)},
  year={2024},
  note={arXiv:2403.19335}
}
```

- arXiv: 2403.19335