Claude Skills by feiyang-k

Use this skill when you want an improved vision-language encoder with better localization, dense features, and multilingual support. Avoid it when SigLIP v1 or CLIP is sufficient for your needs.

Use this skill when you want a contrastive loss that handles noisy image-text pairs better by using sigmoid instead of softmax. Avoid it when you are already using softmax contrastive loss successfully on clean data.

Use this skill when you want to build a cleaned and deduplicated open text corpus from RedPajama for LLM pretraining. Avoid it when you already have a sufficient clean text corpus.

Use this skill when you want to understand the C4 text cleaning methodology and its impact on LLM pretraining quality. Avoid it when you are not cleaning web text or have a different cleaning pipeline.

Use this skill when you want a unified image tokenizer that supports both understanding and generation tasks in VLMs. Avoid it when separate tokenizers for understanding and generation work for you.

Use this skill when you want to unify training data across vision, language, audio, and action modalities for a single autoregressive model. Avoid it when you only need vision-language capability without audio or action.

Use this skill when you want to expand a VLM's vision vocabulary by training additional vision tokens for document, chart, and dense text understanding. Avoid it when CLIP's visual vocabulary is sufficient for your needs.

Use this skill when you want to create instruction data for a VLM that understands both visual and audio content in videos. Avoid it when you only need visual understanding without audio.

Use this skill when you want to train a unified image-video VLM by aligning visual representations before projecting to the language model. Avoid it when you only need image understanding or cannot curate both image and video data.

Use this skill when you want systematic ablation insights on VLM pretraining choices: frozen vs unfrozen LLM, interleaved vs paired data, text mixing. Avoid it when you have a well-established pretraining recipe.

Use this skill when you want to train a VLM that processes raw image patches without heavy vision preprocessing, using a single transformer for both modalities. Avoid it when you need strong visual features from a dedicated vision encoder.

Use this skill when you want to adapt a pre-trained vision model to a new task by prepending learnable visual prompt tokens, training only the prompts. Avoid it when full fine-tuning is affordable or LoRA is preferred.

Use this skill when you want to train a unified VLM using mixture-of-modality-experts that can serve as both dual-encoder and fusion-encoder. Avoid it when you prefer simpler single-mode architectures.

Use this skill when you want to understand which visual tokenizer properties matter most for VLM performance. Avoid it when you have already chosen your visual encoder.

Use this skill when you want to correct VLM hallucinations post-hoc by extracting claims, verifying them against the image, and rewriting responses. Avoid it when you want to prevent hallucination during training rather than correct it post-hoc.

Use this skill when you want to select the most informative samples for annotation using coreset-based active learning in embedding space. Avoid it when you have unlimited annotation budget or random selection is sufficient.

Use this skill when you want to understand scaling behavior when using multiple LLM calls in compound systems for tasks like data generation or evaluation. Avoid it when you use single LLM calls and do not build compound systems.

Use this skill when you want to select diverse, uncertain samples for annotation using gradient embeddings that capture both uncertainty and diversity. Avoid it when simple uncertainty sampling is sufficient or you cannot compute gradients.

Use this skill when you want to beat standard scaling laws by pruning low-quality or redundant data points using perplexity or EL2N scores. Avoid it when you do not have compute for data quality scoring or your data is already curated.

Use this skill when you want to order training examples from easy to hard during training to improve convergence and final performance. Avoid it when random data ordering works well for your task.

Use this skill when you want to predict LLM performance from data mixing ratios and optimize the mix without expensive full-scale training. Avoid it when you have a single data source or cannot run ablation experiments.

Use this skill when you want to assign fair value to individual training examples based on their marginal contribution using Shapley value theory. Avoid it when you cannot afford the computational cost of data Shapley estimation.

Use this skill when you want to map training examples as easy, ambiguous, or hard based on training dynamics (confidence, variability) for data selection. Avoid it when you cannot afford to train a model first to compute training dynamics.

Use this skill when you want to select the best alignment data from a large pool using complexity and quality scoring. Avoid it when you do not have a large pool of alignment data to select from.

Use this skill when you want to prune training data using early-epoch error norms (EL2N scores) to select the most informative examples. Avoid it when you cannot compute early training predictions or all data is equally important.

Use this skill when you want to prune training data by identifying examples that are frequently forgotten during training, using forgetting events as a data quality signal. Avoid it when you cannot train a model first to measure forgetting events.

Use this skill when you want to estimate the influence of individual training examples on model predictions for data debugging and selection. Avoid it when you cannot afford the compute for influence estimation or have too many training examples.

Use this skill when you want to understand how to optimally train when data is limited and must be repeated, and how to trade off epochs vs model size. Avoid it when you have unlimited unique data and do not need to repeat data.

Use this skill when you want to understand power-law relationships between model size, data size, and compute for optimal resource allocation. Avoid it when you are not making scaling decisions.

Use this skill when you want to know the compute-optimal ratio of model size to training data based on Chinchilla scaling laws. Avoid it when you are not making model/data scaling decisions.

Use this skill when you want to efficiently attribute model behavior to training data at scale using random projections and ensembling. Avoid it when you do not need per-example attribution or simple quality metrics suffice.

Use this skill when you want to create 3D-language instruction data for training LLMs to understand and reason about 3D environments. Avoid it when you only work with 2D images.

Use this skill when you want to synthesize high-quality instruction data using GPT-4V on diverse image sources for training small VLMs. Avoid it when you cannot afford GPT-4V API calls or are training a large-scale model.

Use this skill when you want to distill instruction-following capability from a strong LLM to a weaker one using 52K generated instruction samples. Avoid it when you can directly use the strong LLM or have human-written instructions.

Use this skill when you want to combine random augmentations with adversarial training for improved robustness to distribution shifts. Avoid it when standard augmentation provides sufficient robustness.

Use this skill when you want to learn optimal data augmentation policies using reinforcement learning to search over augmentation transforms. Avoid it when RandAugment's simpler approach is sufficient.

Use this skill when you need to bridge a frozen image encoder to a frozen LLM for VLM training with limited paired data. Avoid it when you can afford end-to-end training of the full model or have abundant paired data.

Use this skill when you want to fuse noisy web captions with synthetic model-generated captions using an LLM to get the best of both. Avoid it when you have only one caption source or cannot afford LLM-based fusion.

Use this skill when you want to generate diagnostic visual reasoning data with programmatic scene and question generation for compositional reasoning evaluation. Avoid it when you need real-world images rather than synthetic scenes.

Use this skill when you want to create training data for a VLM that can understand and interact with graphical user interfaces (GUIs). Avoid it when you do not need GUI understanding or agent capability.

Use this skill when you want to apply photometric augmentations (color jitter, brightness, contrast) to CLIP training images for improved robustness. Avoid it when standard random crop and flip are sufficient.

Use this skill when you want to augment CLIP training data with knowledge graph information for improved conceptual understanding. Avoid it when you do not have access to knowledge graphs or standard CLIP training is sufficient.

Use this skill when you want to augment instance segmentation data by copying object instances from one image and pasting them onto another. Avoid it when standard augmentation is sufficient or you do not need instance segmentation.

Use this skill when you want to augment training images by cutting and pasting rectangular patches between images with mixed labels. Avoid it when your task does not benefit from spatial regularization.

Use this skill when you want to generate images conditioned on CLIP embeddings for training data augmentation or dataset expansion. Avoid it when you do not need image generation or CLIP-guided generation.

Use this skill when you want to improve image generation by training a detailed captioner and recaptioning training data. Avoid it when you are not building an image generation model or already have detailed captions.

Use this skill when you want to generate images autoregressively from text for dataset augmentation using a discrete VAE tokenizer. Avoid it when diffusion models provide better generation quality for your needs.

Use this skill when you want to create instruction data for an autonomous driving VLM using BDD-X driving dataset with explanations. Avoid it when you do not work with autonomous driving.

Use this skill when you want to pre-train a document understanding model on synthetic document images without OCR using a vision encoder-decoder. Avoid it when you have good OCR and only need text extraction without visual understanding.

Use this skill when you want to generate long, detailed captions for CLIP training by prompting VLMs with sub-image descriptions. Avoid it when you are satisfied with short captions or your CLIP text encoder cannot handle long text.