Long Sequence Modeling Eval

---
name: long-sequence-modeling-eval
description: Evaluates the capability of sequence modeling architectures to capture long-range dependencies across text, audio, and image modalities, as well as their computational efficiency and compatibility with standard Transformer and CNN backbones. Use when the user wants to benchmark on Long Range Arena (LRA), Speech Commands (SC), WikiText-103, GLUE, ImageNet-1k, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.09298
  bibtex_key: li2022whatmakesconvolutional
  confidence: high
---

# long-sequence-modeling-eval

> What Makes Convolutional Models Great on Long Sequence Modeling? — Yuhong Li et al. (arXiv:2210.09298, 2022)

## What this evaluates

Evaluates the capability of sequence modeling architectures to capture long-range dependencies across text, audio, and image modalities, as well as their computational efficiency and compatibility with standard Transformer and CNN backbones.

## Datasets

- **Long Range Arena (LRA)** — total ?; splits: test (-1)
- **Speech Commands (SC)** — total ?; splits: test (-1)
- **WikiText-103** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: test (-1)
- **ImageNet-1k** — total ?; splits: train (-1), val (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels over the total number of instances in the evaluation set.
- `perplexity` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the ground-truth tokens in the sequence.
- `top-1 accuracy` — range: [0, 1]
  - Fraction of instances where the highest-probability predicted class matches the ground truth label.

## Input / output format

**Input**: Variable-length sequences (text tokens, raw audio waveforms, or image patches) depending on the specific dataset and task.

**Output**: Class labels for classification tasks; next-token probability distributions for language modeling.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name == 'accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_name == 'perplexity':
        return math.exp(-sum(log_probs) / len(log_probs))
    elif metric_name == 'top-1 accuracy':
        return sum(p.argmax(axis=1) == g for p, g in zip(predictions, gold)) / len(gold)
```

## Common pitfalls

- LRA tasks require models to process sequences up to 16K tokens without truncation, testing true long-range dependency capture rather than local context.
- Speech Commands raw waveform classification lacks strong non-SSM baselines, making comparisons against MFCC-based methods potentially uneven.
- GLUE evaluation in this work excludes tasks with fewer than 5K training samples to avoid fine-tuning instability, which may skew average scores.

## Evidence (verbatim from paper)

> SGConv achieves a 1% improvement in average accuracy upon well-tuned S4 variants introduced in Gu et al. (2022b).

## Citation

```bibtex
@misc{li2022whatmakesconvolutional,
  title={What Makes Convolutional Models Great on Long Sequence Modeling?},
  author={Yuhong Li et al.},
  year={2022},
  note={arXiv:2210.09298}
}
```

- arXiv: 2210.09298