Mnist Cl Scenarios Eval

---
name: mnist-cl-scenarios-eval
description: Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting across three distinct continual learning splits of MNIST: incremental task, domain, and class learning. It measures how well baseline and continual learning methods retain knowledge from previous tasks while adapting to new ones. Use when the user wants to benchmark on MNIST, or asks about evaluating this task. Reports average accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.12488
  bibtex_key: hsu2018revaluating
  confidence: high
---

# mnist-cl-scenarios-eval

> Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines — Hsu et al. (2018) (arXiv:1810.12488, 2018)

## What this evaluates

Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting across three distinct continual learning splits of MNIST: incremental task, domain, and class learning. It measures how well baseline and continual learning methods retain knowledge from previous tasks while adapting to new ones.

## Datasets

- **MNIST** — total 70000; splits: train (60000), test (10000); HF `mnist`; repo https://github.com/GT-RIPL/Continual-Learning-Benchmark

## Metrics

- `average accuracy (%)` **(primary)** — range: percent
  - Mean classification accuracy across all tasks seen in the sequence, calculated as (correct predictions / total samples) * 100. Results are averaged over 10 independent runs to report mean ± std.

## Input / output format

**Input**: 32x32 grayscale images (zero-padded from 28x28) with standard normalization to zero mean and unit variance, paired with class labels.

**Output**: Softmax probability distribution over 10 classes, with the predicted class taken as the argmax.

## Scoring recipe

```python
def compute_avg_accuracy(task_preds, task_labels):
    accs = []
    for preds, labels in zip(task_preds, task_labels):
        accs.append((preds == labels).mean() * 100)
    return sum(accs) / len(accs)
```

## Common pitfalls

- Using Adam as the default optimizer, which the authors demonstrate performs poorly compared to SGD, Adagrad, or L2 regularization in continual settings.
- Failing to equalize memory overhead across methods, which unfairly advantages rehearsal-based approaches over parameter-regularization methods.
- Not reporting standard deviations over 10 independent runs, as the protocol explicitly requires averaging across multiple seeds.

## Evidence (verbatim from paper)

> Table 2: The average accuracy (%, higher is better) of all seen tasks after learning the task sequence generated by Split MNIST. The standard train/test split was used, with 60k training images (~6k per digit) and 10k test images (~1k per digit). Each value is the average of 10 runs.

## Citation

```bibtex
@misc{hsu2018revaluating,
  title={Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines},
  author={Hsu et al. (2018)},
  year={2018},
  note={arXiv:1810.12488}
}
```

- arXiv: 1810.12488