Sequence Modeling Eval

---
name: sequence-modeling-eval
description: Evaluates sequence modeling capabilities, specifically long-term memory retention and contextual understanding across synthetic stress tests and real-world benchmarks. It compares generic temporal convolutional networks against canonical recurrent architectures (LSTM, GRU, RNN) on tasks requiring prediction of sequential data. Use when the user wants to benchmark on Adding problem, Sequential MNIST, P-MNIST, Copy memory, Nottingham, JSB Chorales, PTB, Wikitext-103, LAMBADA, text8, or asks about evaluating this task. Reports Perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.01271
  bibtex_key: bai2018empirical
  confidence: high
---

# sequence-modeling-eval

> An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling — Bai et al. (2018) (arXiv:1803.01271, 2018)

## What this evaluates

Evaluates sequence modeling capabilities, specifically long-term memory retention and contextual understanding across synthetic stress tests and real-world benchmarks. It compares generic temporal convolutional networks against canonical recurrent architectures (LSTM, GRU, RNN) on tasks requiring prediction of sequential data.

## Datasets

- **Adding problem** — total ?; splits: train (-1), test (-1)
- **Sequential MNIST** — total ?; splits: train (-1), test (-1)
- **P-MNIST** — total ?; splits: train (-1), test (-1)
- **Copy memory** — total ?; splits: train (-1), test (-1)
- **Nottingham** — total ?; splits: train (-1), test (-1)
- **JSB Chorales** — total ?; splits: train (-1), test (-1)
- **PTB** — total ?; splits: train (-1), val (-1), test (-1)
- **Wikitext-103** — total ?; splits: train (-1), val (-1), test (-1)
- **LAMBADA** — total ?; splits: train (-1), val (-1), test (-1)
- **text8** — total ?; splits: train (-1), test (-1)

## Metrics

- `MSE` — range: [0, inf)
  - Mean Squared Error between predicted and target values, used for the adding problem.
- `Accuracy` — range: [0, 1]
  - Percentage of correctly predicted elements in the sequence, used for synthetic stress tests and classification tasks.
- `Perplexity` **(primary)** — range: [1, inf)
  - Exponential of the average negative log-likelihood per token: exp(-1/N * sum(log p(x_i))). Lower values indicate better language modeling performance.
- `Bits per character` — range: [0, inf)
  - Average negative log-likelihood per character in base-2 logarithm: -1/N * sum(log2 p(x_i)). Used for character-level language modeling.

## Input / output format

**Input**: Sequences of discrete tokens (characters, words, musical notes) or continuous values (MNIST pixels, synthetic stress test inputs) presented in temporal order.

**Output**: Next element in the sequence (token, note, or value) predicted autoregressively or via teacher forcing.

## Scoring recipe

```python
def compute_perplexity(predictions, targets):
    log_probs = log(predictions[range(len(targets)), targets])
    return exp(-mean(log_probs))

def compute_accuracy(predictions, targets):
    correct = sum(pred == target for pred, target in zip(predictions, targets))
    return correct / len(targets)
```

## Common pitfalls

- Comparing generic TCNs/RNNs against highly specialized, domain-tuned architectures without noting the architectural differences.
- Failing to match parameter counts or receptive fields when comparing TCNs to RNNs, leading to unfair capacity advantages.
- Ignoring the impact of hyperparameter tuning differences, as RNNs required grid search while TCNs used minimal tuning.

## Evidence (verbatim from paper)

> On the smaller PTB corpus, an optimized LSTM architecture (with recurrent and embedding dropout, etc.) outperforms the TCN, while the TCN outperforms both GRU and vanilla RNN. However, on the much larger Wikitext-103 corpus and the LAMBADA dataset, without any hyperparameter search, the TCN outperforms the LSTM results of Grave et al. (2017), achieving much lower perplexities.

## Citation

```bibtex
@misc{bai2018empirical,
  title={An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling},
  author={Bai et al. (2018)},
  year={2018},
  note={arXiv:1803.01271}
}
```

- arXiv: 1803.01271