Contextual Sarcasm Detection Eval

---
name: contextual-sarcasm-detection-eval
description: Evaluates a model's ability to detect sarcasm in contextual settings across Reddit comments, tweets, and multi-turn dialogues. It probes the capacity to capture sentiment incongruity and contextual cues rather than relying on surface-level lexical features. Use when the user wants to benchmark on SARC 2.0, Twitter, Sarcasm Corpus V2 Dialogues, or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.11838
  bibtex_key: wen2025transformer
  confidence: high
---

# contextual-sarcasm-detection-eval

> A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection — Wen et al. (2025) (arXiv:2503.11838, 2025)

## What this evaluates

Evaluates a model's ability to detect sarcasm in contextual settings across Reddit comments, tweets, and multi-turn dialogues. It probes the capacity to capture sentiment incongruity and contextual cues rather than relying on surface-level lexical features.

## Datasets

- **SARC 2.0** — total 1300000; splits: train (118940), test (56118); repo https://nlp.cs.princeton.edu/old/SARC/2.0/
- **Twitter** — total 1956; splits: train (1368), test (588)
- **Sarcasm Corpus V2 Dialogues** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` — range: [0, 1]
  - Fraction of correctly classified instances out of the total number of instances.
- `recall` — range: [0, 1]
  - True positive rate: proportion of actual sarcastic instances correctly identified by the model.
- `F1-Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. The final reported value is the average across 5-fold cross-validation runs.

## Input / output format

**Input**: Raw text of a social media comment, tweet, or dialogue turn.

**Output**: Binary classification label indicating whether the text is sarcastic or non-sarcastic.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    acc = (tp + sum(p == g for p, g in zip(preds, gold))) / len(gold)
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, rec, f1
# Final reported metric = mean(metric_values across 5 CV folds)
```

## Common pitfalls

- Must use the 'primary main balanced variation' of SARC 2.0, not the raw imbalanced corpus, as class distribution significantly impacts metrics.
- Results must be averaged across 5-fold cross-validation; single-run or single-split scores are not comparable to the reported protocol.
- Prototype distance metric (Euclidean for RoBERTa vs Cosine for SBERT) affects interpretability but does not change the primary classification metric calculation.

## Evidence (verbatim from paper)

> We used accuracy, recall, and F1-Score as metrics to evaluate models’ performance. Since we use 5-fold cross-validation, we calculated the average of 5 experiment results on the test dataset for each metric as the final result.

## Citation

```bibtex
@misc{wen2025transformer,
  title={A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2503.11838}
}
```

- arXiv: 2503.11838