Qwen Vl Series Finetune

---
name: "qwen-vl-series-finetune"
description: "Route Qwen-VL-series multimodal fine-tuning, preference training,
  classification, adapter merge, and Gradio serving tasks."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Qwen-VL Series Finetune

Use this repo skill for the public Qwen-VL-series training repo that fine-tunes Qwen2-VL, Qwen2.5-VL, Qwen3-VL, and Qwen3.5 multimodal models with Hugging Face, DeepSpeed, PEFT, TRL, and Liger.

This skill is a router, not a monolithic manual. Start here, then follow the sub-skill that matches the user request.

## What this skill covers

- Multimodal dataset preparation and validation for SFT, DPO, GRPO, and classification.
- Supervised fine-tuning, including full finetuning, LoRA, vision LoRA, video training, and evaluation during training.
- Preference training with DPO and GRPO, including reasoning fields and Liger GRPO options.
- Sequence classification training, custom heads, class-imbalance losses, and early stopping.
- Adapter merging and Gradio inference for merged or adapter-backed models.

## Route map

- `sub-skills/data-and-multimodal/` — build or validate JSON datasets, image/video path handling, token formatting, and reasoning fields.
- `sub-skills/sft-training/` — full SFT, LoRA/QLoRA, vision LoRA, video finetuning, and DeepSpeed launch planning.
- `sub-skills/preference-training/` — DPO and GRPO workflows, reward functions, and reasoning-aware preference data.
- `sub-skills/classification-training/` — sequence classification labels, losses, metrics, and early stopping.
- `sub-skills/serving-and-adapters/` — LoRA merge, model loading, and Gradio inference.

## Read first

- `references/install.md` for install and inspection-environment guidance.
- `references/bundled-runtime.md` for the bundled `src/` tree, DeepSpeed configs, and executable training/merge/serving helpers.
- `references/workflow-map.md` for the fastest path from user request to sub-skill.
- `references/data-formats.md` for shared JSON schemas and media/routing rules.
- `references/cli-reference.md` for the main entry points and key flags.
- `references/model-compatibility.md` for model-family and backend caveats.
- `references/troubleshooting.md` for the most common setup and runtime failures.

## Minimal environment check

Before any CUDA-heavy task, run the bundled diagnostic:

- `scripts/check_environment.py`

Use it to confirm package imports, CUDA visibility, and optional serving dependencies without downloading weights or datasets.

## Operating rules

- Keep runtime paths inside this skill tree; do not point future agents back to the source checkout.
- Use the bundled `src/` entrypoints and DeepSpeed configs here; command helpers run from the skill root and set `PYTHONPATH=src`.
- Use the bundled scripts and references here instead of the original repo launch scripts.
- Treat CPU help/import checks as valid only for parser, schema, and pure-Python utilities.
- Treat actual training, multimodal generation, Gradio serving, and adapter merge as CUDA workflows.
- For Qwen3.5, prefer `--disable_flash_attn2 True` unless the user explicitly needs another backend.
- For video workflows, prefer the PyAV/FFmpeg path documented in the references and treat `fps`/`nframes` as mutually exclusive.

## Good first questions

If the user asks for something broad, ask which workflow they want:

- data preparation or validation
- supervised finetuning
- DPO or GRPO
- classification training
- merge or serve

If the user already named one of those workflows, jump directly to that sub-skill.

## Provenance and router metadata

- `references/repo-provenance.md` records the source checkout state.
- `references/repo-routing-metadata.json` records the managed router placement used by import tooling.

## Safe default

If the user asks for a command, prefer a dry-run or command-builder script first, then the execution command after the data, model, and backend assumptions are explicit.