Open Domain Continual Learning Eval

---
name: open-domain-continual-learning-eval
description: Evaluates open-domain continual learning (ODCL) in vision-language models by measuring how well a model adapts to a stream of new image classification domains while preserving previously learned knowledge and zero-shot capabilities on unseen domains. Use when the user wants to benchmark on Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, SUN397, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.10245
  bibtex_key: li2024coleclip
  confidence: high
---

# open-domain-continual-learning-eval

> CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning — Li et al. (2024) (arXiv:2403.10245, 2024)

## What this evaluates

Evaluates open-domain continual learning (ODCL) in vision-language models by measuring how well a model adapts to a stream of new image classification domains while preserving previously learned knowledge and zero-shot capabilities on unseen domains.

## Datasets

- **Aircraft** — total ?; splits: test (-1)
- **Caltech101** — total ?; splits: test (-1)
- **CIFAR100** — total ?; splits: test (-1)
- **DTD** — total ?; splits: test (-1)
- **EuroSAT** — total ?; splits: test (-1)
- **Flowers** — total ?; splits: test (-1)
- **Food** — total ?; splits: test (-1)
- **MNIST** — total ?; splits: test (-1)
- **OxfordPet** — total ?; splits: test (-1)
- **StanfordCars** — total ?; splits: test (-1)
- **SUN397** — total ?; splits: test (-1)

## Metrics

- `Avg` **(primary)** — range: percent
  - Average accuracy over all datasets across all time steps. Avg = (1/T) * sum_{t=1}^{T} A_t, where A_t = (1/T) * sum_{i=1}^{T} A_t^i and A_t^i is accuracy on dataset t after training on task i.
- `Last` — range: percent
  - Average accuracy across all domain datasets at the final time step (after training on all T tasks).
- `Transfer` — range: percent
  - Average zero-shot accuracy on unseen domains. Transfer = (1/T) * sum_{t=1}^{T} T_t, where T_t = (1/(t-1)) * sum_{i=1}^{t-1} A_t^i, measuring accuracy on dataset t before it was seen.
- `Forgetting` — range: percent
  - Average accuracy over all datasets from the learned time step to the last time step. Forgetting = (1/T) * sum_{t=1}^{T} F_t, where F_t = (1/(T-t+1)) * sum_{i=t}^{T} A_t^i.

## Input / output format

**Input**: Image input processed by a frozen CLIP image encoder with task-specific prompts; text class vocabulary processed by a frozen CLIP text encoder with LoRA-based PEFT modules.

**Output**: Predicted class labels for each image, evaluated as classification accuracy (%) on seen and unseen domains.

## Scoring recipe

```python
T = total tasks
A = dict mapping (dataset, time_step) -> accuracy

# Last: accuracy at final step
last_acc = [A[(d, T)] for d in datasets]
Last = mean(last_acc)

# Avg: average accuracy across all steps and datasets
avg_acc = [mean([A[(d, i)] for i in range(1, T+1)]) for d in datasets]
Avg = mean(avg_acc)

# Transfer: zero-shot accuracy on unseen domains
transfer_acc = []
for t in range(1, T+1):
    unseen_accs = [A[(d, i)] for i in range(1, t)]
    transfer_acc.append(mean(unseen_accs) if unseen_accs else 0)
Transfer = mean(transfer_acc)

# Forgetting: accuracy retention from learning step to end
forget_acc = []
for t in range(1, T+1):
    retention_accs = [A[(d, i)] for i in range(t, T+1)]
    forget_acc.append(mean(retention_accs))
Forgetting = mean(forget_acc)
```

## Common pitfalls

- ODCL-CIL (no task ID at inference) is significantly more challenging than ODCL-TIL (task ID known); results drop noticeably in CIL settings.
- Avg and Transfer metrics are only applicable to methods with zero-shot capabilities (e.g., CLIP, ZSCL, CoLeCLIP); closed-domain baselines like CODA-Prompt and LAE lack these metrics and should not be compared on them.
- The paper primarily reports results under Order-I (alphabetical dataset arrival); Order-II (random arrival) results are in the appendix and may yield different performance rankings.

## Evidence (verbatim from paper)

> To effectively evaluate the performance of open-domain CL, following [43], three performance metrics are used, namely Avg, Last, and Transfer, to evaluate the model's ability to preserve knowledge from the pre-training and downstream tasks. Additionally, we introduce a measure called Forgetting, focusing exclusively on the prevention of catastrophic forgetting of knowledge learned from the seen domains.

## Citation

```bibtex
@misc{li2024coleclip,
  title={CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2403.10245}
}
```

- arXiv: 2403.10245