Randumb Ocl Eval

---
name: randumb-ocl-eval
description: Evaluates continual learning methods in online, exemplar-free, and low-exemplar regimes by measuring how well a model retains knowledge of previously seen classes after processing a single pass of sequential data. It specifically tests whether fixed random representations can match or exceed learned representations in these constrained settings. Use when the user wants to benchmark on MNIST, CIFAR10, CIFAR100, TinyImageNet200, miniImageNet100, or asks about evaluating this task. Reports average_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.08823
  bibtex_key: [REDACTED]
  confidence: high
---

# randumb-ocl-eval

> Random Representations Outperform Online Continually Learned Representations — Prabhu et al. (2024) (arXiv:2402.08823, 2024)

## What this evaluates

Evaluates continual learning methods in online, exemplar-free, and low-exemplar regimes by measuring how well a model retains knowledge of previously seen classes after processing a single pass of sequential data. It specifically tests whether fixed random representations can match or exceed learned representations in these constrained settings.

## Datasets

- **MNIST** — total ?; splits: test (-1)
- **CIFAR10** — total ?; splits: test (-1)
- **CIFAR100** — total ?; splits: test (-1)
- **TinyImageNet200** — total ?; splits: test (-1)
- **miniImageNet100** — total ?; splits: test (-1)

## Metrics

- `average_accuracy` **(primary)** — range: percent
  - Average classification accuracy computed on the test set of all previously encountered classes after completing one full pass over the data stream.

## Input / output format

**Input**: Normalized images flattened into vectors (784-dim for MNIST, 3072-dim for others) or 768-dim features from pretrained ViT-B/16 models. One sample (or two with augmentation) is presented per timestep.

**Output**: Class label predictions from a linear classifier trained over fixed random Fourier features or pretrained embeddings.

## Scoring recipe

```python
predictions = model.predict(test_set)
accuracies = []
for task in past_tasks:
    mask = test_set.labels == task
    acc = accuracy_score(test_set.labels[mask], predictions[mask])
    accuracies.append(acc)
return np.mean(accuracies)
```

## Common pitfalls

- The evaluation uses a strict one-pass online setting, unlike standard offline continual learning benchmarks that allow multiple passes or full-batch training.
- RanDumb is evaluated with zero memory (exemplar-free), yet results are compared against methods using replay buffers up to 5k samples, which can skew direct performance comparisons.
- Task ordering does not affect the average accuracy metric in Benchmarks A-E due to equal sample counts per dataset, a detail often overlooked when interpreting continual learning results.

## Evidence (verbatim from paper)

> We measure accuracy on the test set of all past seen classes after completing the full one-pass over the dataset. We take the average accuracy after the last task on all past tasks.

## Citation

```bibtex
@misc{prabhu2024randumb,
  title={Random Representations Outperform Online Continually Learned Representations},
  author={Prabhu et al. (2024)},
  year={2024},
  note={arXiv:2402.08823}
}
```

- arXiv: 2402.08823