Sst Sentiment Analysis Eval

---
name: sst-sentiment-analysis-eval
description: Evaluates a model's ability to classify the sentiment of movie review phrases into five fine-grained categories, measuring classification accuracy and error rates. The benchmark probes hierarchical sentiment understanding at the phrase level rather than the full sentence level. Use when the user wants to benchmark on Stanford Sentiment Treebank (SST), or asks about evaluating this task. Reports Error Rate (Fine-Grained).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.13149
  bibtex_key: patro2019revisiting
  confidence: high
---

# sst-sentiment-analysis-eval

> Revisiting Paraphrase Question Generator using Pairwise Discriminator — Patro et al. (2019) (arXiv:1912.13149, 2019)

## What this evaluates

Evaluates a model's ability to classify the sentiment of movie review phrases into five fine-grained categories, measuring classification accuracy and error rates. The benchmark probes hierarchical sentiment understanding at the phrase level rather than the full sentence level.

## Datasets

- **Stanford Sentiment Treebank (SST)** — total 222000; splits: train (126000), val (30000), test (66000)

## Metrics

- `Error Rate (Fine-Grained)` **(primary)** — range: [0, 1]
  - 1 - Accuracy; calculated as the number of misclassified phrases divided by the total number of phrases in the fine-grained 5-class setting ({Very Negative, Negative, Neutral, Positive, Very Positive}). Lower values indicate better performance.

## Input / output format

**Input**: Movie review phrase or sentence text (string)

**Output**: Sentiment label from {Very Negative, Negative, Neutral, Positive, Very Positive}

## Scoring recipe

```python
def compute_error_rate(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    error_rate = 1 - (correct / len(gold_labels))
    return error_rate
```

## Common pitfalls

- The evaluation is performed at the phrase level, not the full sentence level, which differs from standard SST sentence-level benchmarks and requires phrase-level annotations.
- Error rate is reported instead of accuracy, so readers must remember that lower values indicate better performance, contrary to most classification benchmarks.

## Evidence (verbatim from paper)

> | Model | Error Rate (Fine-Grained) |

## Citation

```bibtex
@misc{patro2019revisiting,
  title={Revisiting Paraphrase Question Generator using Pairwise Discriminator},
  author={Patro et al. (2019)},
  year={2019},
  note={arXiv:1912.13149}
}
```

- arXiv: 1912.13149