All authors
Claude Skills by Calibre-Labs
github.com/Calibre-Labs8 skills0 installs0 views
- Eval CodeWrite deterministic, code-based evaluator functions for an AI product. Use this skill whenever you need to write evaluators that check structural properties of AI outputs (format, count, presence, schema compliance) without calling another LLM. Also use when auditing existing code-based evaluators for brittleness, coverage gaps, or false positives.Votes: 0GitHub stars: 74
- Eval GuideEntry point for eval work. Use when someone asks for help with evals in general, does not know where to start, asks "what should I do next", or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches the request. Load that skill directly.Votes: 0GitHub stars: 74
- Eval Llm JudgeWrite LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner function, saved to the project's evaluators file.Votes: 0GitHub stars: 74
- Eval RubricWrite or update the eval rubric for an AI agent or AI feature: the pass/fail criteria that define good behavior, in four groups (outcome, trajectory, governance, experience), plus the release thresholds that belong in the PRD. Use this skill whenever someone asks to "write an eval rubric", "define what good looks like", "add evals to my PRD", "what should we evaluate", "turn this PRD into a rubric", "update the rubric from these traces", or shares a PRD, agent spec, system prompt or trace cod...Votes: 0GitHub stars: 74
- Llm AlignAnalyze alignment between LLM judge scores and human labels in an eval dataset. Use this skill whenever someone wants to evaluate how well an LLM judge agrees with human reviewers, calculate TPR/TNR, investigate disagreements, or improve a scorer prompt. Trigger on phrases like: "calculate TPR/TNR", "judge alignment", "where does the LLM disagree with humans", "improve my scorer", "why is the LLM getting these wrong", "compare human labels to LLM scores", "evaluate my eval", or any time someo...Votes: 0GitHub stars: 74
- Ticket To EvalStrip PII from a customer support ticket or eval trace and convert it into eval dataset rows. Produces two outputs: a regression dataset row (close to the original input, tagged with failure_mode) and a generalized dataset row (abstracted for broader coverage). Works with local CSV files or any eval platform. Use this skill whenever someone shares a support ticket, user complaint, or raw trace and wants to turn it into eval data. Trigger on: "add this ticket to the dataset", "convert this tra...Votes: 0GitHub stars: 74
- Trace CodesTurn free-form notes from reading agent traces into trace codes: named, yes/no failure or quality patterns with definitions, examples and counts. Use this skill whenever someone has annotated traces or review notes and asks to "cluster my notes", "create trace codes", "find the failure modes", "do error analysis on these annotations", "what patterns are in these traces", or "what should we fix first". Also use to merge new notes into an existing set of codes. Output is a trace codes file, a l...Votes: 0GitHub stars: 74
- UigBuild a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query matrix" or "SQM", improve dataset coverage, or assess whether their offline evals reflect what real users will actually send. Also trigger when they share a product domain + sample dataset and want to know what the data...Votes: 0GitHub stars: 74