Diffusion Instruction Tuning Eval

---
name: diffusion-instruction-tuning-eval
description: This protocol evaluates the vision-language alignment and zero-shot generalization capabilities of fine-tuned VLMs across diverse multimodal tasks. It measures how effectively aligning VLM cross-attention with diffusion model attention maps improves performance on document understanding, reasoning, real-world visual comprehension, and hallucination detection benchmarks. Use when the user wants to benchmark on AI2D, ChartQA, OCRBench, DocVQA, InfoVQA, MME, MMBench, ScienceQA, MMStar, MMMU, RealworldQA, SEED, HallucinationBench, POPE, VQAv2, OK-VQA, TextVQA, VizWiz, HatefulMemes, COCO, Flickr30K, WorldMedQA-V, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.06814
  bibtex_key: jin2025diffusioninstructiontuning
  confidence: high
---

# diffusion-instruction-tuning-eval

> Diffusion Instruction Tuning — Chen Jin et al. (2025) (arXiv:2502.06814, 2025)

## What this evaluates

This protocol evaluates the vision-language alignment and zero-shot generalization capabilities of fine-tuned VLMs across diverse multimodal tasks. It measures how effectively aligning VLM cross-attention with diffusion model attention maps improves performance on document understanding, reasoning, real-world visual comprehension, and hallucination detection benchmarks.

## Datasets

- **AI2D** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **ScienceQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: val (-1)
- **RealworldQA** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **HallucinationBench** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test (-1)
- **OK-VQA** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **HatefulMemes** — total ?; splits: test (-1)
- **COCO** — total ?; splits: val (-1)
- **Flickr30K** — total ?; splits: test (-1)
- **WorldMedQA-V** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact-match or fuzzy-match accuracy against gold labels, computed per benchmark's default protocol. Reported as percentage or normalized score depending on the specific benchmark.

## Input / output format

**Input**: Multimodal instances consisting of an image and a text prompt/question, formatted according to each VLM's specific tokenizer and instruction template.

**Output**: Text generation (answers, captions, or multiple-choice selections) produced autoregressively by the VLM.

## Scoring recipe

```python
def evaluate(prediction, gold, benchmark_name):
    if benchmark_name in ['VQAv2', 'OK-VQA', 'TextVQA', 'MMMU', 'ScienceQA']:
        return exact_match_or_fuzzy_match(prediction, gold)
    elif benchmark_name in ['COCO', 'Flickr30K']:
        return compute_cider_or_rouge(prediction, gold)
    elif benchmark_name in ['POPE', 'HallucinationBench']:
        return compute_precision_recall_f1(prediction, gold)
    else:
        return benchmark_default_metric(prediction, gold)
```

## Common pitfalls

- Data overlap between fine-tuning datasets and benchmark test sets can artificially inflate scores; the paper uses a qualitative overlap scoring system to flag this risk.
- Benchmark metrics vary significantly across subsets (e.g., MMBench has four language subsets), requiring careful aggregation and separate reporting rather than a single averaged score.
- Cross-attention and self-attention VLM architectures respond differently to the alignment method, necessitating architecture-specific baselines to avoid misleading performance claims.

## Evidence (verbatim from paper)

> We evaluate these models on 20 standard VLM benchmarks, comparing them against 23 baseline models ([Section 7.2]). Results in [Figure 10] show up to 72% improvement in zero-shot accuracy, demonstrating Lavender’s effectiveness in vision-text alignment. All benchmarks are evaluated using their default metrics.

## Citation

```bibtex
@misc{jin2025diffusioninstructiontuning,
  title={Diffusion Instruction Tuning},
  author={Chen Jin et al. (2025)},
  year={2025},
  note={arXiv:2502.06814}
}
```

- arXiv: 2502.06814