Sentiment Analysis Eval

---
name: sentiment-analysis-eval
description: Evaluates a model's ability to classify text sentiment into binary or fine-grained polarity categories. Specifically probes how well the model handles negation scope and polarity disentanglement through multi-task learning. Use when the user wants to benchmark on SST-binary, SST-fine, SemEval-binary, SemEval-fine, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.07610
  bibtex_key: barnes2019improving
  confidence: high
---

# sentiment-analysis-eval

> Improving Sentiment Analysis with Multi-task Learning of Negation — Barnes et al. (2019) (arXiv:1906.07610, 2019)

## What this evaluates

Evaluates a model's ability to classify text sentiment into binary or fine-grained polarity categories. Specifically probes how well the model handles negation scope and polarity disentanglement through multi-task learning.

## Datasets

- **SST-binary** — total ?; splits: test (-1)
- **SST-fine** — total ?; splits: test (-1)
- **SemEval-binary** — total ?; splits: test (-1)
- **SemEval-fine** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly classified instances out of the total number of instances. Reported as mean accuracy and standard deviation over five independent runs.

## Input / output format

**Input**: Text snippets (sentences or phrases) from the respective sentiment datasets.

**Output**: Sentiment polarity label (binary: positive/negative; fine-grained: very negative, negative, neutral, positive, very positive).

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- Comparing models with different parameter counts without controlling for capacity, which confounds architectural improvements with model size.
- Using different pooling strategies (e.g., mean vs. max pooling) that significantly impact performance independent of the core architecture.
- Ignoring statistical significance testing across multiple runs/tasks without applying a Bonferroni correction.

## Evidence (verbatim from paper)

> Table 2 shows the mean accuracy and standard deviation of single-task sentiment models (STL), multi-task models with SFU auxiliary negation data (MTL-SFU) and multi-task models with ConanDoyle-neg auxiliary negation data (MTL-CD) over five runs.

## Citation

```bibtex
@misc{barnes2019improving,
  title={Improving Sentiment Analysis with Multi-task Learning of Negation},
  author={Barnes et al. (2019)},
  year={2019},
  note={arXiv:1906.07610}
}
```

- arXiv: 1906.07610