Compressive Transformer Eval

---
name: compressive-transformer-eval
description: Evaluates the ability of transformer-based architectures to model long-range dependencies efficiently by compressing past hidden states into a fixed-size memory. It probes sequence modeling capabilities across text, audio, and visual domains, measuring how well compressed representations preserve salient information for next-token prediction and task completion. Use when the user wants to benchmark on Enwiki8, WikiText-103, PG-19, DMLab-30 (rooms_select_nonmatching_object), or asks about evaluating this task. Reports perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.05507
  bibtex_key: rae2019compressive
  confidence: high
---

# compressive-transformer-eval

> Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae et al. (2019) (arXiv:1911.05507, 2019)

## What this evaluates

Evaluates the ability of transformer-based architectures to model long-range dependencies efficiently by compressing past hidden states into a fixed-size memory. It probes sequence modeling capabilities across text, audio, and visual domains, measuring how well compressed representations preserve salient information for next-token prediction and task completion.

## Datasets

- **Enwiki8** — total 100000000; splits: train (90000000), val (5000000), test (5000000)
- **WikiText-103** — total ?; splits: test (-1)
- **PG-19** — total ?; splits: test (-1)
- **DMLab-30 (rooms_select_nonmatching_object)** — total ?; splits: test (-1)

## Metrics

- `perplexity` **(primary)** — range: other
  - exp(mean(negative_log_likelihood(gold_tokens, predictions))). Measures the effective branching factor of the model's probability distribution over the next token.
- `bits-per-character` — range: other
  - mean(negative_log_likelihood(gold_chars, predictions)) / log(2). Standard character-level language modeling metric where lower is better.

## Input / output format

**Input**: Sequences of characters, subword tokens, audio waveforms, or visual observation frames depending on the domain.

**Output**: Probability distribution over the next token/character, or sampled token.

## Scoring recipe

```python
def compute_perplexity(log_probs):
    avg_nll = np.mean(log_probs)
    return np.exp(avg_nll)

def compute_bpc(log_probs):
    avg_nll = np.mean(log_probs)
    return avg_nll / np.log(2)
```

## Common pitfalls

- Confusing character-level perplexity (Enwiki8) with word-level perplexity (WikiText-103/PG-19), which are not directly comparable due to different tokenization vocabularies.
- Including dynamic evaluation (test-time training) in main comparisons, which the authors explicitly exclude to ensure fair model-to-model baselines.
- Assuming higher compression rates always improve performance; the paper notes performance degrades if compression is too aggressive (e.g., rate 1 fails on RL tasks).

## Evidence (verbatim from paper)

> The proposed model achieves the new state-of-the-art on this dataset with 0.97 bits-per-character.

## Citation

```bibtex
@misc{rae2019compressive,
  title={Compressive Transformers for Long-Range Sequence Modelling},
  author={Jack W. Rae et al. (2019)},
  year={2019},
  note={arXiv:1911.05507}
}
```

- arXiv: 1911.05507