Multimodal Llm

---
name: multimodal-llm
description: "Plan Tevatron multimodal and LLM retriever/ranker workflows,
  including Qwen, ColPali, DSE, RepLLaMA, RankLLaMA, vLLM, and asset-aware
  datasets."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Multimodal and LLM Workflows

Use this sub-skill when a Tevatron task names multimodal retrievers, large LLM retrievers, Qwen-family embedding or vision/audio models, ColPali, DSE, RepLLaMA, RankLLaMA, vLLM encoding, LoRA adapters, or dataset assets for image/audio/video records.

Do not use this sub-skill for ordinary text JSONL schema work, generic dense training, FAISS search mechanics, or generic cross-encoder reranking. Route those to `../data-preparation/`, `../training/`, `../encoding-retrieval/`, or `../reranking/` and use this sub-skill only for multimodal/LLM-specific choices.

## Start Here

1. Read `references/multimodal-workflows.md` to choose Qwen2.5-VL, Qwen-Omni, DSE, ColPali, `train_mm`, `encode_mm`, `vllm_encode_mm`, asset fields, and modality toggles.
2. Read `references/llm-retrievers.md` for RepLLaMA, Qwen3-Embedding, RankLLaMA routing, LoRA, pooling, prefixes, padding, EOS, and text `vllm_encode` trade-offs.
3. Run `python scripts/check_multimodal_dependencies.py --workflow <profile> --json` before recommending a multimodal or LLM command in a minimal environment.
4. Read `references/troubleshooting.md` before changing asset paths, processor/tokenizer names, vLLM flags, FlashAttention settings, shard counts, or batch sizes.

## Route by Workflow

- **Qwen2.5-VL and Qwen-Omni retrievers**: use packaged `tevatron.retriever.driver.train_mm` and `tevatron.retriever.driver.encode_mm`; keep `query_image`, `query_audio`, `query_video`, corpus `image`, `audio`, and `video` fields asset-aware.
- **Document screenshot retrieval**: use Qwen/DSE or ColPali planning from `references/multimodal-workflows.md`; DSE and ColPali example scripts are reference patterns unless their code is separately bundled into a project workflow.
- **RepLLaMA and Qwen3-Embedding**: use text `tevatron.retriever.driver.encode`, `tevatron.retriever.driver.train`, or `tevatron.retriever.driver.vllm_encode` with matching prefixes, `--pooling last` or family-specific pooling, normalization, EOS, and padding choices from `references/llm-retrievers.md`.
- **RankLLaMA**: route cross-encoder scoring and training to `../reranking/`, but use this sub-skill to preserve LLM LoRA, tokenizer, prefix, and memory assumptions when the request compares RankLLaMA with RepLLaMA.
- **vLLM encoding**: prefer it only when `vllm` is installed and the model/adapter supports vLLM embedding; note that packaged `vllm_encode_mm` currently passes prompt plus image, not audio/video.

## Required Checks

Before handing off a plan, confirm:

- Optional packages are explicit: `torch`, `PIL`/`Pillow`, `qwen_omni_utils`, `qwen_vl_utils`, `vllm`, `peft`, suitable `transformers`, FlashAttention, DeepSpeed, and model-specific remote code are not guaranteed by core Tevatron.
- The driver matches the model family: `train_mm`/`encode_mm` for current Qwen-Omni/VL multimodal support, text `train`/`encode`/`vllm_encode` for LLM embedding models, and `../reranking/` for RankLLaMA cross-encoders.
- Dataset asset fields and `--assets_path` or `train_yaml` corpus `assets_path` are consistent before any GPU/model launch.
- Query and corpus encoding use compatible model, adapter, tokenizer/processor, pooling, normalization, prefix, EOS, padding, and shard settings.
- Downstream retrieval matches embedding type: dense pickle outputs go to `../encoding-retrieval/`, while ColPali-style late-interaction outputs need family-specific scoring.