Stance Detection Sentiment Eval

---
name: stance-detection-sentiment-eval
description: Evaluates a model's ability to classify stance (in favor, against, neutral) toward climate change and related targets, while jointly predicting sentiment (positive, negative, neutral). It probes the synergy between stance classification and sentiment analysis using text and topic features. Use when the user wants to benchmark on Climate Change Tweets Dataset, SemEval-2016 Task 6.A, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.03533
  bibtex_key: upadhyaya2022multitask
  confidence: high
---

# stance-detection-sentiment-eval

> A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets — Upadhyaya et al. (2022) (arXiv:2211.03533, 2022)

## What this evaluates

Evaluates a model's ability to classify stance (in favor, against, neutral) toward climate change and related targets, while jointly predicting sentiment (positive, negative, neutral). It probes the synergy between stance classification and sentiment analysis using text and topic features.

## Datasets

- **Climate Change Tweets Dataset** — total ?; splits: 5-fold CV (-1); repo https://github.com/apoorva-upadhyaya/Climate-Change-Tweets
- **SemEval-2016 Task 6.A** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall, averaged across stance classes (in favor, against, neutral). Reported as mean ± standard deviation over 5 folds.
- `Accuracy` — range: percent
  - Ratio of correctly predicted instances to total instances. Reported as mean ± standard deviation over 5 folds.

## Input / output format

**Input**: Tweet text concatenated with topic features (5 topics × 10 words each). For SemEval, tweet text with target context.

**Output**: Two labels per instance: stance label (in favor, against, neutral) and sentiment label (positive, negative, neutral).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = ['in favor', 'against', 'neutral']
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    f1 = sum(f1s) / len(f1s)
    return acc * 100, f1 * 100
```

## Common pitfalls

- Oversampling the minority class (deniers) is applied only to the k-1 training folds in each cross-validation split, not the validation fold.
- Topic features are fixed to m=5 topics and p=10 words per topic; varying these changes performance significantly.
- Metrics are reported as mean ± standard deviation over 5 stratified folds, not a single train/test split.

## Evidence (verbatim from paper)

> For the experiments, we perform stratified k-fold cross-validation on our dataset, oversample the minority class (deniers) in the k-1 training data using the sklearn resampling technique, and report the averaged scores and standard deviation (over 5 folds) for the accuracy and F1 scores.

## Citation

```bibtex
@misc{upadhyaya2022multitask,
  title={A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets},
  author={Upadhyaya et al. (2022)},
  year={2022},
  note={arXiv:2211.03533}
}
```

- arXiv: 2211.03533