Sst Sentiment Eval

---
name: sst-sentiment-eval
description: Evaluates a model's ability to perform sentiment classification on constituent trees, testing both fine-grained (5-class) and binary sentiment prediction at the sentence root and phrase levels. Use when the user wants to benchmark on Stanford Sentiment Treebank, TREC, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1611.06788
  bibtex_key: teng2016bidirectional
  confidence: high
---

# sst-sentiment-eval

> Bidirectional Tree-Structured LSTM with Head Lexicalization — Teng et al. (2016) (arXiv:1611.06788, 2016)

## What this evaluates

Evaluates a model's ability to perform sentiment classification on constituent trees, testing both fine-grained (5-class) and binary sentiment prediction at the sentence root and phrase levels.

## Datasets

- **Stanford Sentiment Treebank** — total ?; splits: train (-1), test (-1)
- **TREC** — total ?; splits: train (5452), test (500), dev (500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified instances. Evaluated separately for sentence root labels and phrase labels on SST, and for question type on TREC.

## Input / output format

**Input**: Constituent parse tree with word tokens at leaves (SST), or raw question sentence (TREC).

**Output**: Sentiment label (0-4 for fine-grained, binary) for each node, or question type class (ENTY, HUM, LOC, DESC, NUM, ABBR).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100
```

## Common pitfalls

- Evaluating only root-level accuracy ignores the phrase-level supervision used in training and testing for SST.
- TREC lacks a development set; the paper randomly splits 500 training examples for dev, which may introduce variance compared to standard fixed splits.

## Evidence (verbatim from paper)

> accuracy increases from 51.2 to 52.8, and the binary sentiment classification accuracy increases from 88.5 to 89.2, which demonstrates the effectiveness of the head lexicalization mechanism.

## Citation

```bibtex
@misc{teng2016bidirectional,
  title={Bidirectional Tree-Structured LSTM with Head Lexicalization},
  author={Teng et al. (2016)},
  year={2016},
  note={arXiv:1611.06788}
}
```

- arXiv: 1611.06788