Dialectal Sentiment Classification Eval

---
name: dialectal-sentiment-classification-eval
description: Evaluates sentiment classification performance across different English dialects (en-US, en-AU, en-UK, en-IN) and tests how label proximity, review length, and sentiment density affect model generalization. Use when the user wants to benchmark on Google Place Reviews (Dialectal Sentiment), or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.11216
  bibtex_key: srirag2024experiences
  confidence: high
---

# dialectal-sentiment-classification-eval

> Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English — Srirag et al. (2024) (arXiv:2410.11216, 2024)

## What this evaluates

Evaluates sentiment classification performance across different English dialects (en-US, en-AU, en-UK, en-IN) and tests how label proximity, review length, and sentiment density affect model generalization.

## Datasets

- **Google Place Reviews (Dialectal Sentiment)** — total ?; splits: en-US (-1), en-AU (-1), en-UK (-1), en-IN (-1)

## Metrics

- `F1-Score` **(primary)** — range: [0, 1]
  - Macro-averaged F1-score computed across all sentiment classes (1★–5★). Calculated as the harmonic mean of precision and recall per class, then averaged.

## Input / output format

**Input**: Plain text of user-generated Google Place reviews.

**Output**: Predicted sentiment label (1★–5★ star rating or mapped simple/hard condition).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    classes = [1, 2, 3, 4, 5]
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- 'Simple' vs 'hard' conditions refer to label proximity (e.g., star rating similarity), not linguistic difficulty.
- Sampling strategies (length/density) create non-i.i.d. subsets that may not reflect real-world dialectal distribution.
- Averaging F1 across locales can mask severe performance drops on outer-circle dialects like en-IN.

## Evidence (verbatim from paper)

> The highest performance on the task is reported by models fine-tuned on en-AU (simple) with the F1-Score of 97.2. The worst performance is reported by the baseline models, with the average F1-Score of 93.5. Models report a degraded performance across reviews from all locales when the labels are changed from simple to hard, with an average decrease in the F1-score of 13.7.

## Citation

```bibtex
@misc{srirag2024experiences,
  title={Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English},
  author={Srirag et al. (2024)},
  year={2024},
  note={arXiv:2410.11216}
}
```

- arXiv: 2410.11216