Continual Instruction Tuning Eval

---
name: continual-instruction-tuning-eval
description: Evaluates the ability of large multimodal models to sequentially learn new instruction-following tasks without catastrophically forgetting previously acquired capabilities. It measures both retained performance on old tasks and the degree of forgetting across sequential training stages. Use when the user wants to benchmark on Flickr30k, TextCaps, VQA v2, OCR-VQA, GQA, VizWiz, TextVQA, or asks about evaluating this task. Reports Average performance ($A_t$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.16206
  bibtex_key: he2023continual
  confidence: high
---

# continual-instruction-tuning-eval

> Continual Instruction Tuning for Large Multimodal Models — He et al. (2023) (arXiv:2311.16206, 2023)

## What this evaluates

Evaluates the ability of large multimodal models to sequentially learn new instruction-following tasks without catastrophically forgetting previously acquired capabilities. It measures both retained performance on old tasks and the degree of forgetting across sequential training stages.

## Datasets

- **Flickr30k** — total ?; splits: test (-1)
- **TextCaps** — total ?; splits: test (-1)
- **VQA v2** — total ?; splits: test (-1)
- **OCR-VQA** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)

## Metrics

- `Average performance ($A_t$)` **(primary)** — range: percent
  - The mean evaluation score across all tasks seen up to the current training stage $t$. Computed as $A_t = \frac{1}{t} \sum_i A_{t,i}$, where $A_{t,i}$ is the score on task $i$ after training on task $t$.
- `Average forgetting ($F_t$)` — range: percent
  - The mean drop in performance on previously learned tasks relative to their peak historical scores. Computed as $F_t = \frac{1}{t-1} \sum_i \max_{j < t}(A_{j,i}) - A_{t,i}$.

## Input / output format

**Input**: An image paired with a natural language instruction or question specific to the current task (e.g., image captioning, visual question answering).

**Output**: A natural language text response (caption or answer) generated by the model.

## Scoring recipe

```python
def compute_cl_metrics(history):
    # history[t][i] = score on task i after stage t
    t = len(history) - 1
    A_t = sum(history[t][i] for i in range(t + 1)) / (t + 1)
    F_t = sum(max(history[j][i] for j in range(t)) - history[t][i] for i in range(t)) / t
    return A_t, F_t
```

## Common pitfalls

- The forgetting metric $F_t$ compares current performance to the maximum historical score across ALL previous stages, not just the immediately preceding stage.
- Sequential fine-tuning (Seq FT) baselines often perform worse than the initial zero-shot model on old tasks, indicating severe catastrophic forgetting rather than simple performance plateaus.
- Task-similarity-informed regularization weights are adaptive based on image-instruction-output similarity, not constant values, which drastically changes anti-forgetting behavior.

## Evidence (verbatim from paper)

> For each task and dataset, we report the widely adopted metrics as shown in Appendix A following [5]. Let $A_{t,i}$ be the evaluation score on task $\mathcal{T}_i$ after training on task $\mathcal{T}_t$. We compute the average performance on all seen tasks after training on each task $\mathcal{T}_i$: $$ A _ {t} = \frac {1}{t} \sum _ {i} A _ {t, i} $$ To measure the degree of forgetting, we also report the average forgetting on all old tasks after each stage $t$: $$ F _ {t} = \frac {1}{t - 1} \sum _ {i} \max _ {j < t} \left(A _ {j, i}\right) - A _ {t, i} $$

## Citation

```bibtex
@misc{he2023continual,
  title={Continual Instruction Tuning for Large Multimodal Models},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2311.16206}
}
```

- arXiv: 2311.16206