Ptb Wikitext2 Lm Eval

---
name: ptb-wikitext2-lm-eval
description: Evaluates next-token prediction accuracy and long-range dependency modeling in language models, with a specific focus on handling rare and out-of-vocabulary words without expanding vocabulary size. Use when the user wants to benchmark on Penn Treebank, WikiText-2, or asks about evaluating this task. Reports perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1609.07843
  bibtex_key: merity2016pointer
  confidence: high
---

# ptb-wikitext2-lm-eval

> Pointer Sentinel Mixture Models — Merity et al. (2016) (arXiv:1609.07843, 2016)

## What this evaluates

Evaluates next-token prediction accuracy and long-range dependency modeling in language models, with a specific focus on handling rare and out-of-vocabulary words without expanding vocabulary size.

## Datasets

- **Penn Treebank** — total ?; splits: train (-1), val (-1), test (-1)
- **WikiText-2** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `perplexity` **(primary)** — range: [0, inf)
  - Exponential of the negative average log-likelihood of the ground truth tokens: exp(-1/N * sum(log p(x_i))). Lower values indicate better language modeling performance.

## Input / output format

**Input**: A sequence of tokens (words/subwords) from the dataset.

**Output**: A probability distribution over the fixed vocabulary and pointer targets for the next token.

## Scoring recipe

```python
def compute_perplexity(log_probs):
    # log_probs: list of log probabilities for each ground truth token
    avg_log_prob = sum(log_probs) / len(log_probs)
    return math.exp(-avg_log_prob)
```

## Common pitfalls

- Truncated BPTT must use k1=1, k2=L to ensure the pointer component receives full backpropagation windows; standard k1=k2 splits cause uneven gradient flow.
- Test-time Monte Carlo dropout averaging (1000 forward passes) is required for fair comparison with variational LSTM baselines but drastically increases inference cost.
- Vocabulary cutoffs differ across datasets; WikiText-2 contains many OOV words relative to PTB's cutoff, which heavily impacts perplexity scores for models without pointer mechanisms.

## Evidence (verbatim from paper)

> Table 2 compares the pointer sentinel-LSTM to a variety of other models on the Penn Treebank dataset. The pointer sentinel-LSTM achieves the lowest perplexity, followed by the recent Recurrent Highway Networks (Zilly et al., 2016). The medium pointer sentinel-LSTM model also achieves lower perplexity than the large LSTM models. We halve the learning rate when validation perplexity is worse than the previous iteration, stopping training when validation perplexity fails to improve for three epochs or when 64 epochs are reached.

## Citation

```bibtex
@misc{merity2016pointer,
  title={Pointer Sentinel Mixture Models},
  author={Merity et al. (2016)},
  year={2016},
  note={arXiv:1609.07843}
}
```

- arXiv: 1609.07843