Banglabook Sentiment Eval

---
name: banglabook-sentiment-eval
description: Evaluates the ability of models to classify Bangla book reviews into three sentiment categories (Positive, Neutral, Negative). It probes product-specific sentiment analysis in a low-resource language, testing both contextual understanding and robustness to class imbalance and lexical overlap. Use when the user wants to benchmark on BANGLABOOK, or asks about evaluating this task. Reports weighted average F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.06595
  bibtex_key: kabir2023banglabook
  confidence: high
---

# banglabook-sentiment-eval

> BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews — Kabir et al. (2023) (arXiv:2305.06595, 2023)

## What this evaluates

Evaluates the ability of models to classify Bangla book reviews into three sentiment categories (Positive, Neutral, Negative). It probes product-specific sentiment analysis in a low-resource language, testing both contextual understanding and robustness to class imbalance and lexical overlap.

## Datasets

- **BANGLABOOK** — total 158065; splits: train (-1), val (-1), test (-1); repo https://github.com/mohsinulkabir14/BanglaBook

## Metrics

- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated per class to balance performance across imbalanced positive/negative samples.
- `weighted average F1-score` **(primary)** — range: [0, 1]
  - Weighted mean of per-class F1-scores, where weights correspond to the number of true instances for each class, providing a single aggregate score for the entire dataset.

## Input / output format

**Input**: Raw Bangla text of a book review.

**Output**: Categorical sentiment label: 'Positive', 'Neutral', or 'Negative'.

## Scoring recipe

```python
from sklearn.metrics import f1_score
# predictions and gold are lists of class labels
f1_macro = f1_score(gold, predictions, average='macro')
f1_weighted = f1_score(gold, predictions, average='weighted')
```

## Common pitfalls

- Class imbalance causes models to heavily favor the 'Positive' class, leading to poor 'Neutral' and 'Negative' recall.
- High lexical overlap between classes (e.g., positive words appearing in negative/neutral reviews) confuses bag-of-words and n-gram features.
- Rating-based labeling may introduce incongruent samples where the text sentiment contradicts the numerical rating, affecting ground truth reliability.

## Evidence (verbatim from paper)

> We select F1-score and weighted average F1-score to evaluate the models because the dataset has an uneven class distribution. F1-score is the harmonic mean of precision and recall and it helps balance the metric across the imbalanced positive/negative samples (Sokolova et al., 2006). All our experiments are done using scikit-learn, pytorch, and transformers (Vaswani et al., 2017) and run on Google Colaboratory. The training, testing, and validation split of the entire dataset was 70-20-10 with previously unseen samples in the test and validation set.

## Citation

```bibtex
@misc{kabir2023banglabook,
  title={BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews},
  author={Kabir et al. (2023)},
  year={2023},
  note={arXiv:2305.06595}
}
```

- arXiv: 2305.06595