Ocl Accuracy Retention Eval

---
name: ocl-accuracy-retention-eval
description: Evaluates an online continual learning model's ability to rapidly adapt to incoming data streams while retaining knowledge of past classes without catastrophic forgetting, under strict computational budgets and fixed feature extractors. Use when the user wants to benchmark on CGLM, CLOC, or asks about evaluating this task. Reports a_t.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.09253
  bibtex_key: prabhu2023online
  confidence: high
---

# ocl-accuracy-retention-eval

> Online Continual Learning Without the Storage Constraint — Prabhu et al. (2023) (arXiv:2305.09253, 2023)

## What this evaluates

Evaluates an online continual learning model's ability to rapidly adapt to incoming data streams while retaining knowledge of past classes without catastrophic forgetting, under strict computational budgets and fixed feature extractors.

## Datasets

- **CGLM** — total ?; splits: pretrain (-1), train (-1), test (-1)
- **CLOC** — total 39000000; splits: pretrain (-1), train (-1), test (-1)

## Metrics

- `a_t` **(primary)** — range: [0, 1]
  - Average online accuracy until timestep t, computed as the cumulative fraction of correct predictions over all timesteps seen so far: a_t = (1/t) * sum_{i=1 to t} 1(y_i == y_hat_i).
- `IR_h` — range: [0, 1]
  - Information retention over the last h timesteps, measuring catastrophic forgetting mitigation: IR_h = (1/h) * sum_{t=T-h to T} 1(y_t == y_hat_t).

## Input / output format

**Input**: Sequential incoming images processed through a fixed pretrained ResNet50 backbone and a 2-layer MLP projector to 256-dimensional normalized features.

**Output**: Predicted class label y_hat_i for each incoming image.

## Scoring recipe

```python
def compute_a_t(predictions, labels, t):
    correct = sum(1 for y, y_hat in zip(labels[:t], predictions[:t]) if y == y_hat)
    return correct / t

def compute_IR_h(predictions, labels, T, h):
    correct = sum(1 for y, y_hat in zip(labels[T-h:T], predictions[T-h:T]) if y == y_hat)
    return correct / h
```

## Common pitfalls

- Confusing online accuracy (a_t) with final test accuracy; a_t is computed cumulatively over all timesteps seen so far, not just the current batch.
- Assuming methods update the backbone; all baselines and ACM use a fixed pretrained ResNet50 feature extractor, with only the classifier/adaptor updated.
- Overlooking the computational budget constraint; fairness is enforced by limiting all methods to one gradient update per batch, regardless of storage availability.

## Evidence (verbatim from paper)

> We follow Cai et al. (2021), measuring average online accuracy until the current timestep $t$ ($a_{t}$) as a metric for measuring rapid adaptation, given by $a_{t}\=\nicefrac{{1}}{{t}}\sum_{i\=1}^{t}\mathds{1}_{y_{i}\=\hat{y}_{i}}$ where $\mathds{1}_{(\cdot)}$ is the indicator function. We additionally measure information retention, i.e. mitigating catastrophic forgetting, after online training on unseen samples from a test set. Formally, information retention for $h$ timesteps (${IR}_{h}$) at time $T$, is defined as ${IR}_{h}\=\nicefrac{{1}}{{h}}\sum_{t\=T-h}^{T}\mathds{1}_{y_{t}\=\hat{y}_{t}}$.

## Citation

```bibtex
@misc{prabhu2023online,
  title={Online Continual Learning Without the Storage Constraint},
  author={Prabhu et al. (2023)},
  year={2023},
  note={arXiv:2305.09253}
}
```

- arXiv: 2305.09253