Coin Eval

---
name: coin-eval
description: Evaluates continual instruction tuning in multimodal large language models by measuring how well they retain task-specific instruction alignment and underlying reasoning knowledge when trained sequentially on diverse datasets. Use when the user wants to benchmark on ScienceQA, TextVQA, ImageNet, GQA, VizWiz, Grounding, VQAv2, OCR-VQA, or asks about evaluating this task. Reports Truth Alignment.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.08350
  bibtex_key: chen2024coin
  confidence: high
---

# coin-eval

> CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model — Cheng Chen et al. (2024) (arXiv:2403.08350, 2024)

## What this evaluates

Evaluates continual instruction tuning in multimodal large language models by measuring how well they retain task-specific instruction alignment and underlying reasoning knowledge when trained sequentially on diverse datasets.

## Datasets

- **ScienceQA** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **ImageNet** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **Grounding** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test (-1)
- **OCR-VQA** — total ?; splits: test (-1)

## Metrics

- `Truth Alignment` **(primary)** — range: percent
  - Task-specific accuracy measuring whether the model's output exactly matches the ground truth or follows the required instruction format.
- `Reasoning Capability` — range: [0, 10]
  - LLM-judged score (0-10) assessing whether the model retains underlying reasoning knowledge despite failing to align with the task instruction.
- `MAA` — range: percent
  - Mean Accuracy Across tasks, calculated as the arithmetic average of per-task Truth Alignment accuracies.
- `BWT` — range: percent
  - Backward Transfer, measuring catastrophic forgetting as the average performance drop on previously learned tasks after sequential fine-tuning.

## Input / output format

**Input**: Multimodal prompt consisting of an image and a text instruction/question (e.g., 'What is written on the front of that building? Answer the question using a single word or phrase.')

**Output**: Text response matching the task's expected format (e.g., single word/phrase, option letter, or OCR token).

## Scoring recipe

```python
def compute_metrics(predictions, golds, previous_accuracies, current_accuracies):
    truth_alignment = [1 if p == g else 0 for p, g in zip(predictions, golds)]
    truth_alignment_acc = sum(truth_alignment) / len(truth_alignment)
    reasoning_capability = llm_judge_score(predictions, golds) # 0-10 scale
    maa = sum(truth_alignment_acc) / num_tasks
    bwt = sum(current_acc - prev_acc for prev_acc, current_acc in zip(previous_accuracies, current_accuracies)) / num_tasks
    return truth_alignment_acc, reasoning_capability, maa, bwt
```

## Common pitfalls

- Confusing Truth Alignment (instruction following) with Reasoning Capability (knowledge retention); the paper explicitly shows they degrade independently.
- Assuming multi-task training is always the upper bound; task gaps can make it perform worse than sequential fine-tuning on the first task.
- Ignoring task order diversity; random vs. alphabetical ordering significantly impacts BWT and overall MAA due to knowledge interference.

## Evidence (verbatim from paper)

> Quantitative results about the ability of Truth Alignment and Reasoning Capability are shown in Tab. 2 and Tab. 3, respectively. For the results of truth alignment of Tab. 2, we have the following observations: ... However, due to the absence of techniques to regulate learning, these models suffer from forgetting, resulting in -32.62 of LLaVA and -16.94 of Qwen-VL in terms of BWT.

## Citation

```bibtex
@misc{chen2024coin,
  title={CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model},
  author={Cheng Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.08350}
}
```

- arXiv: 2403.08350