Claude Skills by rchaz

Build a training dataset for fine-tuning, distillation, or continued pretraining. Use when the user wants to turn logs/CSV/JSONL into fine-tuning data, label data with an LLM, distill a teacher model's outputs, generate synthetic training examples from nothing, chunk raw domain text for CPT, deduplicate a dataset, make train/valid/test splits, convert to MLX chat format, write a datacard, or asks "how much data do I need to fine-tune?".

The tunelab front door — decides whether a task needs fine-tuning at all, by running EXPERIMENTS on the user's data, not just interviewing. Use whenever the user wants to fine-tune, distill, or train a small/local model, cut their LLM API bill, replace frontier calls with something cheaper or faster, build a router/classifier/triage/cascade, asks "is fine-tuning worth it?" or "which architecture?", or wants to learn fine-tuning by experiment. Runs cheap probes + a frontier ceiling probe (head...

Evaluate a fine-tuned, distilled, or continued-pretrained model with held-out test discipline — the honest scoreboard at the end of the tunelab pipeline. Pre-registers the acceptance bar and metric set BEFORE results exist, runs the untouched test split through base and tuned models, scores classification (accuracy, macro-F1, per-class precision/recall, confusion matrix, hallucinated-label flagging) or generative output (blinded pairwise LLM-as-judge — session-native or API), measures CPT per...

The tunelab capstone — drives a self-improving AI system. Use when the user wants a deployed model/cascade/workflow to keep getting better from feedback, run champion/challenger experiments to discover the best architecture, set up a data flywheel with retrain triggers, or automate "is a new model better than what we ship?" decisions. Orchestrates tune-decide/tune-data/tune-train/tune-eval as a Monitor-Analyze-Plan-Execute loop. The system may be one model, a cascade, an agent, or a determini...

Drive a local MLX-LM training run on Apple Silicon (LoRA/QLoRA, full fine-tuning, CPT) after tune-decide has validated a Level 2-3 plan. Use to pick a base model and hyperparameters, launch/monitor/resume a detached mlx_lm.lora run, diagnose loss curves, run continued pretraining on a validated corpus, or fuse adapters / export GGUF. Also the re-entry point when a training run was interrupted or a session died mid-run. Assumes tune-decide already validated the level — routes there first if no...