Dualnet Cl Eval

---
name: dualnet-cl-eval
description: Evaluates a model's ability to learn sequentially from a stream of tasks without catastrophic forgetting, while adapting quickly to new tasks. It probes both task-aware (with task IDs) and task-free (without task IDs) continual learning settings, measuring final accuracy, forgetting, and knowledge transfer. Use when the user wants to benchmark on Split miniImageNet, CORE50, or asks about evaluating this task. Reports ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.00175
  bibtex_key: pham2021dualnet
  confidence: high
---

# dualnet-cl-eval

> DualNet: Continual Learning, Fast and Slow — Pham et al. (2021) (arXiv:2110.00175, 2021)

## What this evaluates

Evaluates a model's ability to learn sequentially from a stream of tasks without catastrophic forgetting, while adapting quickly to new tasks. It probes both task-aware (with task IDs) and task-free (without task IDs) continual learning settings, measuring final accuracy, forgetting, and knowledge transfer.

## Datasets

- **Split miniImageNet** — total ?; splits: train (-1), val (-1)
- **CORE50** — total ?; splits: train (-1), val (-1)

## Metrics

- `ACC` **(primary)** — range: percent
  - Averaged classification accuracy across all tasks/classes evaluated at the end of the training sequence.
- `FM` — range: percent
  - Forgetting Measure: the maximum drop in accuracy on previously learned tasks from their peak performance to the end of training.
- `LA` — range: percent
  - Learning Accuracy: the accuracy on the first task(s) at the end of training, indicating knowledge transfer or retention of initial knowledge.

## Input / output format

**Input**: Image patches with class labels. In the task-aware (TA) setting, a task identifier is also provided. In the task-free (TF) setting, no task identifiers are given.

**Output**: Predicted class label for each image.

## Scoring recipe

```python
acc = mean([accuracy(task_preds, task_labels) for task in all_tasks])
fm = max([peak_acc[task] - final_acc[task] for task in all_tasks])
la = accuracy(first_task_preds, first_task_labels)
```

## Common pitfalls

- Task-aware vs task-free protocols require different memory buffer sizes (50 samples/task vs 100 samples/class) and evaluation procedures.
- Data augmentation is applied during training but explicitly disabled during inference, which can cause performance drops if not handled correctly.
- Metrics are only reported at the end of the full training sequence, not per-task or incremental steps.

## Evidence (verbatim from paper)

> For a comprehensive evaluation, we run the experiments five times and report the averaged accuracy of all tasks/classes at the end of training (ACC), the forgetting measure (Chaudhry et al., 2018) (FM), and the learning accuracy (LA) (Riemer et al., 2019).

## Citation

```bibtex
@misc{pham2021dualnet,
  title={DualNet: Continual Learning, Fast and Slow},
  author={Pham et al. (2021)},
  year={2021},
  note={arXiv:2110.00175}
}
```

- arXiv: 2110.00175