Claude Skills by feiyang-k

Use this skill when you want a simple generative image-to-text model trained on large-scale image-text data with minimal architectural complexity. Avoid it when you need a more complex architecture with cross-attention or dual encoders.

Use this skill when you want to unify object detection and phrase grounding training data for a model that detects objects from language descriptions. Avoid it when you do not need language-guided detection or grounding.

Use this skill when you want to understand safety evaluation and mitigation strategies for deploying multimodal LLMs. Avoid it when safety evaluation is not your focus.

Use this skill when you need an open-set object detector trained on grounded text-image data to detect any object described in natural language. Avoid it when you only need closed-set detection on fixed categories.

Use this skill when you want an efficient 8B VLM training recipe with curated data combining web interleaved, paired, and instruction data. Avoid it when you need a larger model or have a different data strategy.

Use this skill when you want to use ImageNet-21K for vision encoder pretraining with proper preprocessing and training recipes. Avoid it when ImageNet-1K or CLIP pretraining is sufficient.

Use this skill when you want to curate a high-quality academic-task-oriented instruction dataset for VLM fine-tuning. Avoid it when you need only synthetic conversation data or cannot access the academic VQA datasets.

Use this skill when you want to optimize multi-task VLM training by dynamically modulating gradients from different loss functions. Avoid it when single-loss training is sufficient.

Use this skill when you want to curate multi-task instruction data and make the vision encoder instruction-aware for better VLM fine-tuning. Avoid it when you have a single task and do not need instruction-conditioned visual feature extraction.

Use this skill when you want training data for a VLM that can both understand and compose interleaved text-image content like articles. Avoid it when you only need image understanding without composition.

Use this skill when you need a progressive data pipeline that aligns a large vision encoder with an LLM through contrastive, generative, and instruction-tuning stages. Avoid it when you are building a small model and cannot afford multi-stage pretraining.

Use this skill when you want to progressively scale VLM data from alignment to multi-task to high-quality instruction data with dynamic resolution. Avoid it when you do not have multi-stage data or cannot afford progressive training.

Use this skill when you want to understand how label noise in training data affects optimization and generalization, particularly for noisy web data. Avoid it when you do not deal with noisy labels.

Use this skill when you want to align an LLM with just 1,000 carefully curated instruction examples, demonstrating that quality trumps quantity. Avoid it when you have abundant alignment data or need maximum diversity.

Use this skill when you want to understand Meta's comprehensive approach to pretraining data curation covering text, code, multimodal, and multilingual data at massive scale. Avoid it when you do not need insights into large-scale pretraining data recipes.

Use this skill when you want to efficiently add visual instruction-following to LLaMA using lightweight adapters with zero-initialized attention. Avoid it when full fine-tuning is feasible or you prefer other PEFT methods.

Use this skill when you want to train a safety classifier for filtering harmful content in LLM/VLM inputs and outputs. Avoid it when you do not need safety filtering or have an existing safety classifier.

Use this skill when you want to adapt the LLaVA data recipe for compact language models like Gemma-2B for efficient VLM deployment. Avoid it when you are using a larger LLM and do not need compact VLMs.

Use this skill when you want to scale up instruction tuning data with higher quality and diversity for improved VLM reasoning. Avoid it when you have limited compute and cannot handle dynamic high-resolution image processing.

Use this skill when you want to transfer strong image understanding from LLaVA-NeXT to video through efficient frame sampling and video-specific data. Avoid it when you are building a video-from-scratch VLM.

Use this skill when you want a unified training recipe for single-image, multi-image, and video understanding with curated data at each stage. Avoid it when you only need single-image understanding.

Use this skill when you want to align a VLM using RLHF with factually augmented reward signals that penalize hallucination. Avoid it when you do not need RLHF-based alignment for your VLM.

Use this skill when you want to extend VLM context length for processing many images or long videos by transferring long context from language training. Avoid it when you do not need long-context visual processing.

Use this skill when you want to efficiently fine-tune LLMs/VLMs on domain-specific data using low-rank adapter matrices instead of full fine-tuning. Avoid it when you can afford full fine-tuning or need maximum model capacity.

Use this skill when you want to pre-train a ViT by masking 75% of image patches and reconstructing them, learning rich visual features. Avoid it when contrastive pretraining (CLIP, DINO) is sufficient.

Use this skill when you want to train embeddings that work at multiple dimensionalities, enabling flexible compute-quality tradeoffs at retrieval time. Avoid it when you only need fixed-dimension embeddings.

Use this skill when you want a data recipe for training a phone-deployable VLM that approaches GPT-4V quality on key benchmarks. Avoid it when you are not targeting mobile deployment.

Use this skill when you want to understand optimal data mixing recipes for multimodal LLM pretraining across interleaved, image-text, and text-only data. Avoid it when you only have one data type or cannot run ablation studies to tune the mix.

Use this skill when you want to create instruction data for high-resolution VLMs that can handle documents, dense text, and detailed images. Avoid it when you only work with low-resolution images or do not need document understanding.

Use this skill when you want a modular VLM training approach using visual abstractor and two-stage training on alignment then instruction data. Avoid it when you prefer simpler projection approaches.

Use this skill when you want to improve VLM training through modality-adaptive modules and curated multi-task data that promotes vision-language collaboration. Avoid it when you are using a simple vision-language projection without modality adaptation.

Use this skill when you need a comprehensive survey of multimodal learning with transformers covering fusion strategies, pretraining, and applications. Avoid it when you need a specific method rather than a broad overview.

Use this skill when you want to understand how CLIP learns multimodal neurons that respond to both visual and textual concepts for debugging data quality. Avoid it when you do not need interpretability or representation analysis.

Use this skill when you want to train an object detector using image captions as weak supervision instead of bounding box annotations. Avoid it when you have abundant bounding box annotations.

Use this skill when you want to train CLIP models on open datasets (LAION) with reproducible training recipes and systematic data ablations. Avoid it when you are using pre-trained CLIP models and do not need to train your own.

Use this skill when you want insights on training a smaller but stronger VLM through improved data quality, SigLIP encoder, and efficient training recipe. Avoid it when you are not optimizing VLM training efficiency.

Use this skill when you want to train a natively multimodal model that processes variable numbers of images at any resolution without fixed token budgets. Avoid it when fixed-resolution VLM approaches work for your needs.

Use this skill when you want to evaluate and reduce object hallucination in VLMs using a polling-based evaluation framework. Avoid it when you are not concerned about object hallucination or have other evaluation methods.

Use this skill when you want to understand multi-resolution pretraining strategies where images are processed at progressively higher resolutions during training. Avoid it when single-resolution training is sufficient.

Use this skill when you need a multi-stage data pipeline covering pretraining, multi-task training, and instruction tuning with grounding and OCR data. Avoid it when you need only basic image understanding without grounding or OCR capabilities.

Use this skill when you want to train a VLM with native dynamic resolution support and multi-stage data covering images, videos, and documents. Avoid it when you use fixed-resolution input and do not need dynamic resolution.

Use this skill when you want to recaption a billion-scale dataset like DataComp-1B using an open-source VLM to improve CLIP training. Avoid it when your dataset captions are already high-quality or you lack the compute for billion-scale recaptioning.

Use this skill when you want to build a high-quality web text corpus through aggressive deduplication and filtering of Common Crawl. Avoid it when you have a curated text corpus or do not need web-scale text data.

Use this skill when you want to collect fine-grained human feedback on VLM outputs to reduce hallucination through RLHF. Avoid it when you do not need RLHF or hallucination is not a concern.

Use this skill when you need a systematic approach to curating paired image-text and interleaved data at web scale for autoregressive VLM pretraining. Avoid it when you only need a small fine-tuning dataset.

Use this skill when you want to understand how to scale instruction fine-tuning across number of tasks, model size, and chain-of-thought data for maximum benefit. Avoid it when you are doing single-task fine-tuning.

Use this skill when you want to improve image-text datasets by replacing web alt-text with richer synthetic captions from a captioning model. Avoid it when you already have high-quality detailed captions or the captioner lacks domain knowledge.

Use this skill when you want to scale vision transformers to 22B parameters using JFT data with efficient training techniques. Avoid it when you do not need a very large vision encoder.

Use this skill when you want to scale vision models efficiently using sparse Mixture-of-Experts for handling more training data per compute. Avoid it when dense models are sufficient for your scale.

Use this skill when you want to remove semantically redundant examples from large-scale training data using embedding-based deduplication. Avoid it when you only need exact or near-exact duplicate removal or your dataset is small enough that redundancy is not a concern.