Phrase Level Polarity Eval

---
name: phrase-level-polarity-eval
description: Classifies the sentiment polarity of specific phrases within tweets, requiring models to handle contextual disambiguation, sarcasm, and informal language. Use when the user wants to benchmark on Twitter2015-test, or asks about evaluating this task. Reports macro-averaged F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.02387
  bibtex_key: rosenthal2015semeval
  confidence: high
---

# phrase-level-polarity-eval

> SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)

## What this evaluates

Classifies the sentiment polarity of specific phrases within tweets, requiring models to handle contextual disambiguation, sarcasm, and informal language.

## Datasets

- **Twitter2015-test** — total ?; splits: test (-1)

## Metrics

- `macro-averaged F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall averaged across all classes (positive, negative, neutral). F1 = 2 * (macro_prec * macro_rec) / (macro_prec + macro_rec).

## Input / output format

**Input**: A phrase extracted from a tweet.

**Output**: Sentiment polarity label (positive, negative, or neutral).

## Scoring recipe

```python
prec = rec = 0
for class in ['pos', 'neg', 'neu']:
    prec += precision(y_true, y_pred, class)
    rec += recall(y_true, y_pred, class)
macro_prec = prec / 3
macro_rec = rec / 3
f1 = 2 * (macro_prec * macro_rec) / (macro_prec + macro_rec)
```

## Common pitfalls

- Class imbalance is significant; the majority class baseline achieves ~38% F1, so models must significantly outperform simple majority voting.
- Sarcasm and slang can flip polarity, requiring contextual models rather than simple bag-of-words approaches.

## Evidence (verbatim from paper)

> The results (macro-averaged F1 score) for subtask A are shown in Table 10. The participating systems are ranked by their score on the Twitter2015-test dataset, which is the official ranking for subtask A

## Citation

```bibtex
@misc{rosenthal2015semeval,
  title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2015)},
  year={2015},
  note={SemEval-2015 / arXiv:1912.02387}
}
```

- arXiv: 1912.02387