Continual Learning Accuracy Eval

---
name: continual-learning-accuracy-eval
description: Evaluates a model's ability to learn sequentially across multiple tasks without catastrophic forgetting. It measures how well the model retains accuracy on previously learned tasks while adapting to new ones. Use when the user wants to benchmark on Split MNIST, Permuted MNIST, Split CIFAR-10/100, or asks about evaluating this task. Reports average classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1703.04200
  bibtex_key: zenke2017synaptic
  confidence: high
---

# continual-learning-accuracy-eval

> Continual Learning Through Synaptic Intelligence — Zenke et al. (2017) (arXiv:1703.04200, 2017)

## What this evaluates

Evaluates a model's ability to learn sequentially across multiple tasks without catastrophic forgetting. It measures how well the model retains accuracy on previously learned tasks while adapting to new ones.

## Datasets

- **Split MNIST** — total ?; splits: train (-1), test (-1)
- **Permuted MNIST** — total ?; splits: train (-1), test (-1)
- **Split CIFAR-10/100** — total ?; splits: train (-1), val (-1)

## Metrics

- `average classification accuracy` **(primary)** — range: [0, 1]
  - Computed as the mean of classification accuracies across all tasks learned up to the current step. Accuracy per task is the fraction of correctly predicted labels on the held-out test or validation set.

## Input / output format

**Input**: Grayscale (MNIST) or RGB (CIFAR) image inputs fed into an MLP or CNN architecture.

**Output**: Categorical class predictions (digit 0-9 for MNIST, class 0-99 for CIFAR-100) via softmax output.

## Scoring recipe

```python
accuracies = []
for task in tasks_learned:
    preds = model.predict(task.test_data)
    acc = (preds == task.test_labels).mean()
    accuracies.append(acc)
metric_value = sum(accuracies) / len(accuracies)
```

## Common pitfalls

- Optimizer state reset policy differs between benchmarks (reset for Split MNIST/CIFAR, maintained for Permuted MNIST), which significantly impacts reported accuracy.
- Multi-head loss must be used during training to prevent label distribution crosstalk; evaluating on a single shared head without masking can yield misleading results.
- Confusion between training accuracy and validation/test accuracy when reporting performance trends.

## Evidence (verbatim from paper)

> To evaluate the performance, we computed the average classification accuracy on all previous tasks as a function of number of tasks trained.

## Citation

```bibtex
@misc{zenke2017synaptic,
  title={Continual Learning Through Synaptic Intelligence},
  author={Zenke et al. (2017)},
  year={2017},
  note={arXiv:1703.04200}
}
```

- arXiv: 1703.04200