Claude Skills by feiyang-k

Use this skill when you want to create structured QA data for autonomous driving VLMs using graph-based scene understanding. Avoid it when you do not work with autonomous driving or need general-purpose VQA.

Use this skill when you want to create training data for a VLM that can both refer to and ground objects at point, box, and free-form region granularities. Avoid it when you only need image-level understanding without spatial referring.

Use this skill when you need to build a large-scale multi-task dataset with diverse vision annotations (caption, detection, segmentation, grounding) using automated annotation engines. Avoid it when you only need a single task type or already have comprehensive multi-task annotations.

Use this skill when you want to create instruction data for remote sensing VLMs covering satellite image understanding, change detection, and visual grounding. Avoid it when you do not work with remote sensing or satellite imagery.

Use this skill when you want to create training data for a VLM that generates text grounded at the pixel level with segmentation masks. Avoid it when you do not need pixel-level grounding in VLM outputs.

Use this skill when you want to generate balanced, compositional visual QA data from scene graphs using programmatic question generation. Avoid it when you need free-form questions that cannot be generated from scene graph templates.

Use this skill when you want to automatically generate grounded segmentation annotations by combining Grounding DINO for text-based detection with SAM for segmentation. Avoid it when you have manual grounding annotations or do not need automated annotation.

Use this skill when you want to generate harder negative examples for contrastive learning by mixing embeddings of existing negatives. Avoid it when random negatives provide sufficient contrastive signal.

Use this skill when you want to leverage a frozen large language model as a text encoder for high-fidelity text-to-image generation. Avoid it when CLIP text encoders are sufficient for your generation needs.

Use this skill when you want to generate high-quality instruction data using GPT-4 for fine-tuning smaller models. Avoid it when you cannot afford GPT-4 API calls or have sufficient human-written data.

Use this skill when you want to align a VLM with just 200 carefully curated high-quality instruction examples, pushing the LIMA principle to multimodal. Avoid it when you have more than 200 good instruction examples available.

Use this skill when you want to generate synthetic image editing instruction data by combining GPT-3 text editing with Prompt-to-Prompt image editing. Avoid it when you have real image editing data or do not need editing capability.

Use this skill when you need to construct grounded image-text data with bounding box annotations linked to text spans for training grounding-capable VLMs. Avoid it when you do not need spatial grounding or bounding box outputs from your VLM.

Use this skill when you want to augment CLIP training text by rewriting captions with an LLM to improve text diversity. Avoid it when your captions are already diverse and high-quality or you cannot afford LLM rewriting.

Use this skill when you want to pre-train a document understanding model using unified text and image masking on document images with OCR. Avoid it when you do not need document layout understanding or OCR-dependent processing.

Use this skill when you want to create training data for a VLM that outputs segmentation masks through language-based reasoning. Avoid it when you do not need segmentation output or can use standard segmentation datasets.

Use this skill when you want to train a VLM to evaluate its own outputs for self-improvement without external AI judges. Avoid it when you have access to external AI judges or do not need self-evaluation.

Use this skill when you want to generate biomedical visual instruction data by aligning PMC image-caption pairs with LLM-generated QA. Avoid it when you need general-purpose instruction data or lack biomedical image-caption sources.

Use this skill when you want to scale video instruction data for training a unified video understanding VLM with diverse temporal reasoning tasks. Avoid it when you only need image understanding or have limited video data.

Use this skill when you want to generate instruction data by prompting an aligned LLM with just the system prompt to elicit user-like instructions from the model itself. Avoid it when you have instruction data or prefer structured generation approaches.

Use this skill when you want to create instruction data for multi-image understanding tasks like comparison, reasoning across images, and temporal understanding. Avoid it when you only need single-image instruction data.

Use this skill when you want to align a VLM using a small set of self-curated high-quality image-description pairs. Avoid it when you have abundant alignment data or do not need to fix generation quality issues.

Use this skill when you want to augment training data by linearly interpolating between pairs of training examples and their labels. Avoid it when you need discrete, unmodified training examples.

Use this skill when you want to create training images by combining four cropped images into a mosaic grid for object detection training. Avoid it when standard single-image augmentation is sufficient.

Use this skill when you want to enable multimodal few-shot learning by training a visual encoder to produce prefix tokens for a frozen language model. Avoid it when you have abundant training data and do not need few-shot adaptation.

Use this skill when you want to build a comprehensive synthetic data generation pipeline using strong models to create instruction data at scale. Avoid it when you have sufficient real instruction data or cannot run large model inference.

Use this skill when you want to convert academic PDF documents into structured markup using a vision transformer without traditional OCR. Avoid it when you need OCR for general documents or handwriting rather than academic papers.

Use this skill when you want to distill reasoning capability by generating step-by-step explanation traces from a strong teacher model. Avoid it when you do not need reasoning capability or have sufficient reasoning data.

Use this skill when you want to create pixel-level instruction tuning data where the VLM can understand and describe individual pixel regions in images. Avoid it when bounding box-level understanding is sufficient.

Use this skill when you want to create instruction data that includes in-context examples for multi-modal in-context learning. Avoid it when you do not need in-context learning capability in your VLM.

Use this skill when you want to generate captions for 70M video clips using multiple cross-modality teacher models. Avoid it when you only need image captions or cannot process video at this scale.

Use this skill when you want to generate synthetic textbook-quality code data for training a small but highly capable code model. Avoid it when you have sufficient high-quality code data or are not training a code model.

Use this skill when you want to generate synthetic textbook-quality training data to teach reasoning to small language models. Avoid it when you are training a large model where data quality filtering alone suffices.

Use this skill when you want to train high-capability small models using heavily filtered web data augmented with synthetic textbook-quality data. Avoid it when you are training a large model where data quality filtering is less impactful.

Use this skill when you want to pre-train a document understanding model by parsing web page screenshots into their HTML structure. Avoid it when you do not need document understanding or your documents are not web-like.

Use this skill when you want to train a high-quality text-to-image model efficiently using carefully curated high-quality captions from an LLM. Avoid it when you already have a well-trained text-to-image model.

Use this skill when you want to create instruction data for LLMs to understand 3D point clouds using GPT-4 to generate point cloud descriptions. Avoid it when you do not work with point cloud data.

Use this skill when you want simple automated data augmentation with only two hyperparameters (number and magnitude of transforms). Avoid it when you have a custom augmentation pipeline or do not need automated augmentation.

Use this skill when you want to create region-level instruction data for VLMs that understand and describe image regions with fine-grained detail. Avoid it when image-level understanding is sufficient.

Use this skill when you want a unified segmentation model trained on diverse segmentation data types (semantic, instance, panoptic, interactive) for VLM grounding. Avoid it when you only need one segmentation type.

Use this skill when you want to bootstrap instruction data from a seed set by having the model generate its own instructions iteratively. Avoid it when you have abundant human-written instructions or need domain expertise the model lacks.

Use this skill when you want to generate high-quality detailed captions using GPT-4V and then train a captioner to scale caption generation. Avoid it when you have sufficient high-quality captions or cannot afford GPT-4V API calls for the seed set.

Use this skill when you want to generate high-quality video captions using GPT-4V and train an open-source video captioner for scaling. Avoid it when you do not work with video or have sufficient video captions.

Use this skill when you want to create referential dialogue data where the VLM and user exchange spatial references (bounding boxes) naturally within conversation. Avoid it when you do not need spatial references in conversation.

Use this skill when you want to generate preference data for VLM alignment using AI feedback instead of human feedback. Avoid it when you can collect human preferences or do not need preference-based alignment.

Use this skill when you want to use text-to-image diffusion models to generate synthetic training images for augmenting VLM training data. Avoid it when you have sufficient real training images or synthetic images would not improve your task.

Use this skill when you want to train CLIP-style models entirely on synthetic images generated by Stable Diffusion instead of real web images. Avoid it when you have sufficient real image data or synthetic quality is insufficient.

Use this skill when you want to scale up visual instruction tuning data to 4.2M samples using GPT-4 with detailed image annotations. Avoid it when you already have sufficient instruction data or cannot afford GPT-4 generation at scale.

Use this skill when you want to generate synthetic images with realistic text overlaid on natural scenes for text detection/recognition training. Avoid it when you have sufficient real text detection data or need non-text image generation.

Use this skill when you want to refine text prompts for image generation to produce higher quality, more faithful synthetic training images. Avoid it when your prompts already produce satisfactory synthetic images.