Imdb Sentiment Eval

---
name: imdb-sentiment-eval
description: Tests the model's capability to generate fixed-length representations for variable-length documents containing multiple sentences. It probes whether the method can scale to longer texts and outperform traditional bag-of-words baselines on a large-scale sentiment classification benchmark. Use when the user wants to benchmark on IMDB dataset, or asks about evaluating this task. Reports error rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1405.4053
  bibtex_key: le2014distributed
  confidence: high
---

# imdb-sentiment-eval

> Distributed Representations of Sentences and Documents — Quoc V. Le et al. (2014) (arXiv:1405.4053, 2014)

## What this evaluates

Tests the model's capability to generate fixed-length representations for variable-length documents containing multiple sentences. It probes whether the method can scale to longer texts and outperform traditional bag-of-words baselines on a large-scale sentiment classification benchmark.

## Datasets

- **IMDB dataset** — total 100000; splits: train (25000), test (25000), unlabeled_train (50000); repo http://ai.Stanford.edu/amaas/data/sentiment/index.html

## Metrics

- `error rate` **(primary)** — range: percent
  - 1 - (number of correctly classified test instances / total test instances). Binary classification (Positive/Negative).

## Input / output format

**Input**: Movie reviews consisting of multiple sentences.

**Output**: Predicted sentiment label (Positive or Negative).

## Scoring recipe

```python
predictions = classifier.predict(paragraph_vectors)
errors = sum(1 for p, g in zip(predictions, gold_labels) if p != g)
error_rate = errors / len(gold_labels)
```

## Common pitfalls

- Paragraph vectors for test reviews are learned at test time via gradient descent after freezing the rest of the network.
- The model is trained on 75,000 documents (25k labeled + 50k unlabeled) to learn word and paragraph vectors before classification.
- A neural network with one hidden layer (50 units) is used instead of a linear logistic classifier.

## Evidence (verbatim from paper)

> The method described in this paper is the only approach that goes significantly beyond the barrier of 10% error rate. It achieves 7.42% which is another 1.3% absolute improvement (or 15% relative improvement) over the best previous result.

## Citation

```bibtex
@misc{le2014distributed,
  title={Distributed Representations of Sentences and Documents},
  author={Quoc V. Le et al. (2014)},
  year={2014},
  note={arXiv:1405.4053}
}
```

- arXiv: 1405.4053