Oasis Cit Eval

---
name: oasis-cit-eval
description: Evaluates online sample selection methods for continual visual instruction tuning by measuring how effectively models learn from sequential data subsets. It probes the model's capacity for knowledge retention across tasks and its ability to adapt to new domains without catastrophic forgetting. Use when the user wants to benchmark on MICVIT, COAST, Adapt, Long Sequence, TRACE, or asks about evaluating this task. Reports $A_{last}$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02011
  bibtex_key: lee2025oasis
  confidence: high
---

# oasis-cit-eval

> OASIS: Online Sample Selection for Continual Visual Instruction Tuning — Minjae Lee et al. (2025) (arXiv:2506.02011, 2025)

## What this evaluates

Evaluates online sample selection methods for continual visual instruction tuning by measuring how effectively models learn from sequential data subsets. It probes the model's capacity for knowledge retention across tasks and its ability to adapt to new domains without catastrophic forgetting.

## Datasets

- **MICVIT** — total ?; splits: continual (-1)
- **COAST** — total ?; splits: continual (-1)
- **Adapt** — total ?; splits: continual (-1)
- **Long Sequence** — total ?; splits: continual (-1)
- **TRACE** — total ?; splits: continual (-1)

## Metrics

- `$A_{last}$` **(primary)** — range: percent
  - Accuracy measured at the end of the continual training process across all tasks.
- `$A_{avg}$` — range: percent
  - Average accuracy computed at each task boundary during the continual learning sequence.
- `Kernel Density` — range: other
  - Mean pairwise similarity of selected samples under a Gaussian kernel; lower values indicate higher sample diversity.

## Input / output format

**Input**: Image-text pairs formatted as visual instruction tuning prompts, processed sequentially across multiple tasks in a continual learning setting.

**Output**: Textual responses generated by the MLLM/LLM to the given instructions.

## Scoring recipe

```python
def compute_accuracy(preds, golds):
    return sum(p == g for p, g in zip(preds, golds)) / len(golds) * 100

def compute_A_last(task_accuracies):
    return task_accuracies[-1]

def compute_A_avg(task_accuracies):
    return sum(task_accuracies) / len(task_accuracies)
```

## Common pitfalls

- COAST benchmark assumes balanced task sizes (20k samples per task), which contradicts real-world data imbalance and may skew selection baselines.
- Adapt benchmark contains COCO images that overlap with LLaVA's instruction-tuning data, potentially inflating reported accuracy due to data leakage.
- Accuracy metrics are reported as raw numbers in tables but represent percentages; statistical significance is determined via Welch's t-test at 0.05 level.

## Evidence (verbatim from paper)

> Metrics. We report $A_{last}$, the accuracy measured at the end of training, and $A_{avg}$, the average accuracy measured at each task boundary. Benchmarks. We evaluate on a range of CIT benchmarks, including text-only benchmarks (Long Sequence, TRACE) and multi-modal benchmarks (COAST, Adapt). However, COAST assumes balanced task sizes (i.e., 20k samples per task), which contradicts real-world data imbalance, while Adapt includes datasets containing COCO images that overlap with LLaVA’s instruction-tuning data. To address these limitations, we introduce Multi-image Imbalanced Continual Visual Instruction Tuning (MICVIT)...

## Citation

```bibtex
@misc{lee2025oasis,
  title={OASIS: Online Sample Selection for Continual Visual Instruction Tuning},
  author={Minjae Lee et al. (2025)},
  year={2025},
  note={arXiv:2506.02011}
}
```

- arXiv: 2506.02011