Stance Detection Eval

---
name: stance-detection-eval
description: This benchmark evaluates a model's ability to classify the stance (InFavor, Against, or None) of text towards a specific target or query. It specifically probes cross-target generalization by training on multiple targets and testing on a held-out target, while also assessing robustness to sarcastic or figurative language through intermediate sarcasm pre-training. Use when the user wants to benchmark on SemEval 2016 Task 6A Dataset, Multi-Perspective Consumer Health Query Data (MPCHI), or asks about evaluating this task. Reports average macro F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.03787
  bibtex_key: [REDACTED]
  confidence: high
---

# stance-detection-eval

> Sarcasm Detection as a Catalyst: Improving Stance Detection with Cross-Target Capabilities — Nkhata et al. (2025) (arXiv:2503.03787, 2025)

## What this evaluates

This benchmark evaluates a model's ability to classify the stance (InFavor, Against, or None) of text towards a specific target or query. It specifically probes cross-target generalization by training on multiple targets and testing on a held-out target, while also assessing robustness to sarcastic or figurative language through intermediate sarcasm pre-training.

## Datasets

- **SemEval 2016 Task 6A Dataset** — total ?; splits: train (-1), test (-1)
- **Multi-Perspective Consumer Health Query Data (MPCHI)** — total ?; splits: train (-1), test (-1)

## Metrics

- `average macro F1-score` **(primary)** — range: [0, 1]
  - Macro F1-score computed exclusively over the 'InFavor' and 'Against' classes, then averaged. The 'None' class is explicitly excluded from the calculation.

## Input / output format

**Input**: Text instance (tweet or sentence) paired with a target entity or query. Text is preprocessed via case folding, stemming, stop-word removal, and hashtag processing, then tokenized using the pre-trained language model's default tokenizer.

**Output**: A single class label from {InFavor, Against, None} produced via a softmax layer over three output nodes.

## Scoring recipe

```python
def macro_f1_infavor_against(preds, gold):
    mask = [g != 'None' for g in gold]
    p_f = [x for x, m in zip(preds, mask) if m]
    g_f = [x for x, m in zip(gold, mask) if m]
    f1s = []
    for cls in ['InFavor', 'Against']:
        tp = sum(1 for a, b in zip(p_f, g_f) if a == cls and b == cls)
        fp = sum(1 for a, b in zip(p_f, g_f) if a == cls and b != cls)
        fn = sum(1 for a, b in zip(p_f, g_f) if a != cls and b == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- The evaluation explicitly excludes the 'None' class from the F1 calculation; including it will produce incorrect scores.
- Cross-Target Stance Detection (CTSD) uses a leave-one-out split (train on 4 targets, test on 1), not a standard random train/test split.
- Sarcasm detection is used only as an intermediate pre-training task to improve representation learning, not as a direct evaluation metric or output class.

## Evidence (verbatim from paper)

> In alignment with previous studies[[5]][[7]][[60]], the evaluation of our model is based on the average macro F1-score for the InFavor and Against classes.

## Citation

```bibtex
@misc{nkhata2025sarcasm,
  title={Sarcasm Detection as a Catalyst: Improving Stance Detection with Cross-Target Capabilities},
  author={Nkhata et al. (2025)},
  year={2025},
  note={arXiv:2503.03787}
}
```

- arXiv: 2503.03787