Long Range Arena Eval

---
name: long-range-arena-eval
description: Evaluates efficient Transformer architectures on long-context sequence modeling tasks spanning text, images, and structured data. Probes capabilities in hierarchical reasoning, spatial navigation, and retrieval over sequences up to 16K tokens. Use when the user wants to benchmark on ListOps, Text Classification, Retrieval, Image Classification, Pathfinder / Path-X, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.04006
  bibtex_key: tay2020longrangearena
  confidence: high
---

# long-range-arena-eval

> Long Range Arena: A Benchmark for Efficient Transformers — Tay et al. (2020) (arXiv:2011.04006, 2020)

## What this evaluates

Evaluates efficient Transformer architectures on long-context sequence modeling tasks spanning text, images, and structured data. Probes capabilities in hierarchical reasoning, spatial navigation, and retrieval over sequences up to 16K tokens.

## Datasets

- **ListOps** — total ?; splits: train (-1), test (-1)
- **Text Classification** — total ?; splits: train (-1), test (-1)
- **Retrieval** — total ?; splits: train (-1), test (-1)
- **Image Classification** — total ?; splits: train (-1), test (-1)
- **Pathfinder / Path-X** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances.
- `steps per second` — range: other
  - Throughput metric measuring the number of training steps completed per second on the specified hardware.
- `peak memory usage` — range: other
  - Maximum GPU/TPU memory consumed per device during evaluation, measured in gigabytes.

## Input / output format

**Input**: Task-specific sequences: mathematical expression lists, byte-level character sequences, image patches, or path graphs. Sequence lengths typically range from 1K to 16K tokens.

**Output**: Discrete class labels corresponding to the task (e.g., 10-way classification for ListOps, binary/multi-class for others).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Results are based on fixed hyperparameters across all models to ensure fairness, so they do not represent optimally tuned performance for any single architecture.
- Efficiency metrics (speed/memory) are highly dependent on the specific hardware (4x4 TPU V3) and implementation details used in the benchmark.
- Models frequently overfit training data but show a large generalization gap on test sets, particularly for image classification tasks.

## Evidence (verbatim from paper)

> Byte-level classification is shown to be difficult and challenging especially when no pretraining or contextual embeddings are used. The best model only obtains 65.90 accuracy.

## Citation

```bibtex
@misc{tay2020longrangearena,
  title={Long Range Arena: A Benchmark for Efficient Transformers},
  author={Tay et al. (2020)},
  year={2020},
  note={arXiv:2011.04006}
}
```

- arXiv: 2011.04006