Prior Polarity Degree Eval

---
name: prior-polarity-degree-eval
description: Evaluates the ability of sentiment lexicons or models to assign accurate real-valued polarity scores to individual terms, measuring rank correlation with gold standards. Use when the user wants to benchmark on Term test set, or asks about evaluating this task. Reports Kendall's τ coefficient.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.02387
  bibtex_key: [REDACTED]
  confidence: high
---

# prior-polarity-degree-eval

> SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)

## What this evaluates

Evaluates the ability of sentiment lexicons or models to assign accurate real-valued polarity scores to individual terms, measuring rank correlation with gold standards.

## Datasets

- **Term test set** — total ?; splits: test (-1)

## Metrics

- `Kendall's τ coefficient` **(primary)** — range: [-1, 1]
  - Rank correlation coefficient measuring the correspondence between predicted and gold term polarity rankings. Ranges from -1 to 1.
- `Spearman's ρ coefficient` — range: [-1, 1]
  - Rank correlation coefficient measuring the similarity between predicted and gold term polarity rankings. Ranges from -1 to 1.

## Input / output format

**Input**: A single term (word or phrase).

**Output**: Real-valued sentiment score.

## Scoring recipe

```python
tau = kendalltau(y_true, y_pred)
rho = spearmanrho(y_true, y_pred)
# Higher values indicate better rank correlation with gold scores
```

## Common pitfalls

- Many teams used unsupervised lexicon-based approaches; supervised word-embedding models performed best.
- Baseline uses a simple fallback between two public lexicons (Hashtag Sentiment Lexicon and Sentiment140), scoring ~0.277 avgDiff.

## Evidence (verbatim from paper)

> The results are presented in Table 14. ... Kendall's τ coefficient, Spearman's ρ coefficient

## Citation

```bibtex
@misc{rosenthal2015semeval,
  title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2015)},
  year={2015},
  note={SemEval-2015 / arXiv:1912.02387}
}
```

- arXiv: 1912.02387