Besstie Eval

---
name: besstie-eval
description: Evaluates language models' ability to classify sentiment and detect sarcasm across three distinct varieties of English (Australian, Indian, and British). It probes cross-variety generalization and the impact of domain (Google reviews vs. Reddit comments) on model performance. Use when the user wants to benchmark on BESSTIE, or asks about evaluating this task. Reports F-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.04726
  bibtex_key: srirag2024besstie
  confidence: high
---

# besstie-eval

> BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English — Srirag et al. (2024) (arXiv:2412.04726, 2024)

## What this evaluates

Evaluates language models' ability to classify sentiment and detect sarcasm across three distinct varieties of English (Australian, Indian, and British). It probes cross-variety generalization and the impact of domain (Google reviews vs. Reddit comments) on model performance.

## Datasets

- **BESSTIE** — total ?; splits: train (-1), test (-1)

## Metrics

- `F-Score` **(primary)** — range: [0, 1]
  - Macro-averaged F1-score, computed as the harmonic mean of precision and recall across all classes. Values range from 0 to 1, with higher indicating better performance.

## Input / output format

**Input**: Textual user-generated content (Google Places reviews or Reddit comments) labeled with a specific English variety (en-AU, en-IN, or en-UK).

**Output**: Classification labels for sentiment (e.g., positive/negative) and sarcasm (e.g., sarcastic/not sarcastic).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    # Calculate precision and recall per class
    # Macro-average across classes
    # Return F1 score
    pass
```

## Common pitfalls

- Sarcasm labels are only available for the Reddit subset; Google Places reviews lack sarcasm annotations.
- Fine-tuning on a single variety significantly degrades cross-variety generalization for sarcasm classification.
- Encoder models consistently outperform decoder models on these classification tasks due to architectural suitability.

## Evidence (verbatim from paper)

> We first present our results on the two tasks where models are trained and evaluated on the same variety. Figures[3] and[4] describe the model performances, reported using F-Score, on the sentiment and sarcasm classification tasks respectively. While sentiment classification is performed using both google and reddit subsets, due to the absence of sarcasm labels in google subset (as shown in Table[5]), sarcasm classification is conducted only on reddit subset.

## Citation

```bibtex
@misc{srirag2024besstie,
  title={BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English},
  author={Srirag et al. (2024)},
  year={2024},
  note={arXiv:2412.04726}
}
```

- arXiv: 2412.04726