Mvn Text Classification Eval

---
name: mvn-text-classification-eval
description: Evaluates text classification models on sentiment analysis and news categorization tasks. It probes the model's ability to aggregate diverse feature views (word-level and n-gram) to predict fine-grained sentiment categories and news topics. Use when the user wants to benchmark on Stanford Sentiment Treebank, AG News, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1704.05907
  bibtex_key: guo2017endtoend
  confidence: high
---

# mvn-text-classification-eval

> End-to-End Multi-View Networks for Text Classification — Guo et al. (2017) (arXiv:1704.05907, 2017)

## What this evaluates

Evaluates text classification models on sentiment analysis and news categorization tasks. It probes the model's ability to aggregate diverse feature views (word-level and n-gram) to predict fine-grained sentiment categories and news topics.

## Datasets

- **Stanford Sentiment Treebank** — total 11855; splits: train (-1), dev (-1), test (-1)
- **AG News** — total 127600; splits: train (120000), test (7600)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of the total number of test instances.
- `error rate` — range: [0, 1]
  - 1 - accuracy, reported for the AG News benchmark in Table 3.

## Input / output format

**Input**: Raw text sentences (SST) or news articles (AG News), converted to 300-dimensional GloVe word embeddings and optionally augmented with CNN-derived n-gram features.

**Output**: Predicted class label (5 fine-grained sentiment classes for SST, 4 news categories for AG News).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# For AG News error rate:
error_rate = 1 - compute_accuracy(predictions, gold_labels)
```

## Common pitfalls

- Evaluating on phrases instead of sentences for the Stanford Sentiment Treebank (paper explicitly restricts test evaluation to sentences only).
- Using custom or different train/dev/test splits instead of the exact splits from Socher et al. (2013).
- Confusing error rate with accuracy when reading AG News results, as Table 3 reports error rates while the text discusses accuracy improvements.

## Evidence (verbatim from paper)

> We use the same splits for training, dev, and test data as in (Socher et al., 2013) to predict the fine-grained 5-class sentiment categories of the sentences. ... The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table 1.

## Citation

```bibtex
@misc{guo2017endtoend,
  title={End-to-End Multi-View Networks for Text Classification},
  author={Guo et al. (2017)},
  year={2017},
  note={arXiv:1704.05907}
}
```

- arXiv: 1704.05907