All authors
Claude Skills by ContextJet-ai
github.com/ContextJet-ai26 skills0 installs67 views
- Add Llm EvalsUse this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI) and online (production LLM-as-a-judge) evaluation.Votes: 0GitHub stars: 40
- Add Llm FallbacksUse this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages. Trigger on "handle LLM API errors", "add retries/fallbacks", "the app breaks when OpenAI is down", "rate limit errors", "make my LLM calls reliable", "timeout handling". Add retries, timeouts, and model/provider fallbacks, and observe them so failures are visible.Votes: 0GitHub stars: 40
- Add Llm GuardrailsUse this to add safety and security guardrails to an LLM/agent app - blocking prompt injection, PII leakage, jailbreaks, toxic output, off-topic responses, or invalid structured output. Trigger on "add guardrails", "prevent prompt injection", "stop PII leaks", "validate the model's output", "make this safe for production", especially for regulated/finance/enterprise use.Votes: 0GitHub stars: 40
- Annotate Traces For ReviewUse this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy golden dataset. Trigger on "review my LLM outputs", "have an expert label these", "error analysis", "annotate traces", "build a golden dataset", or when automated evals are not enough for a high-stakes or specialized domain. Looking at your data is the highest-ROI thing you can do.Votes: 0GitHub stars: 40
- Build Eval DatasetUse this to build a good evaluation dataset for an LLM app, the part everyone underestimates. Trigger on "make an eval set", "what should I test my LLM on", "I don't have test data for my prompt", "build a golden dataset", or before setting up evals. A great eval set beats a great metric; garbage-in means your evals lie to you.Votes: 0GitHub stars: 40
- Check Answer ConsistencyUse this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-consistency check". Ships a runnable, tested scorer you can put inline or on sampled traffic.Votes: 0GitHub stars: 40
- Choose Observability StackUse this to recommend an LLM observability / evaluation tool or stack for a specific situation. Trigger on "which observability tool should I use", "compare Langfuse vs Phoenix vs LangSmith", "what's the best LLM monitoring for us", or picking an eval/tracing/gateway tool given constraints (self-hosting, budget, compliance, existing stack). Ask about constraints, then recommend from the curated list - don't just name the most popular tool.Votes: 0GitHub stars: 40
- Collect User FeedbackUse this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real user signal into your best source of eval data.Votes: 0GitHub stars: 40
- Compare Llm ModelsUse this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release. Trigger on "which model should I use", "is GPT/Claude/Gemini/Llama better for this", "should I switch models", "can a cheaper model do this", "compare models for my use case". Evaluate on YOUR task, not on leaderboards alone.Votes: 0GitHub stars: 40
- Debug Agent From TracesUse this to diagnose WHY an LLM agent or chain produced a wrong, empty, slow, or expensive result, by reading its observability trace. Trigger on "my agent gave the wrong answer", "the chain returned nothing", "why is this so slow/expensive", "debug this trace/run", or when a trace tree is available. Walk the span tree systematically instead of guessing.Votes: 0GitHub stars: 40
- Detect HallucinationsUse this to detect when an LLM is making things up, so you can flag or block confident-but-wrong answers before users see them. Trigger on "detect hallucinations", "is the model making this up", "flag unreliable answers", "hallucination check", "confidence scoring for LLM output", or hardening a RAG/QA system. Pick a method that matches whether you have reference context or not.Votes: 0GitHub stars: 40
- Estimate Llm CostUse this to estimate what an LLM call or feature will cost, and to compare models on price, before or after shipping. Trigger on "how much will this cost", "estimate my OpenAI/Anthropic bill", "is a cheaper model worth it", "cost of this prompt", "project my LLM spend". Ships a runnable, tested calculator so the numbers are real, not hand-waved.Votes: 0GitHub stars: 40
- Eval Driven DevelopmentUse this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.Votes: 0GitHub stars: 40
- Instrument Llm ObservabilityUse this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.Votes: 0GitHub stars: 40
- Measure Agent Task SuccessUse this to measure whether an AI agent actually completed its task end to end, not just whether individual LLM calls looked fine. Trigger on "is my agent working", "measure agent success rate", "evaluate my agent", "how good is my agent", "agent completion rate", or evaluating a multi-step/tool-using agent. Score the outcome of the whole task, plus the path it took.Votes: 0GitHub stars: 40
- Monitor Rag QualityUse this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG", "improve RAG accuracy". Diagnose whether the failure is in retrieval or generation - they need different fixes.Votes: 0GitHub stars: 40
- Optimize PromptsUse this to improve a prompt systematically instead of hand-tweaking it by feel. Trigger on "optimize my prompt", "make this prompt better", "the prompt isn't working well", "auto-tune my prompt", "few-shot example selection", or when prompt quality has plateaued. Optimize against an eval set with a method, and let the numbers pick the winner.Votes: 0GitHub stars: 40
- Red Team Llm AppUse this to adversarially test an LLM/agent app before attackers do - prompt injection, jailbreaks, data exfiltration, tool misuse, and unsafe output. Trigger on "red team my LLM", "test for prompt injection", "is my agent secure", "jailbreak testing", "security review of my AI app", especially before shipping anything customer-facing or with tools/data access. Test systematically against the known attack classes, not ad-hoc.Votes: 0GitHub stars: 40
- Redact Pii For TracingUse this when adding LLM observability to an app that handles sensitive data (finance, healthcare, PII) and you must NOT ship raw prompts/PII to a third-party tracing backend. Trigger on "redact traces", "PII in observability", "can we self-host tracing for compliance", "GDPR/HIPAA/SOC2 and LLM logging", or instrumenting a regulated app. Get observability without creating a data-leak.Votes: 0GitHub stars: 40
- Reduce Llm CostUse this to cut the cost of an LLM app using observability data. Trigger on "my OpenAI/Anthropic bill is too high", "reduce token usage", "the app is expensive", "optimize LLM cost", "why am I spending so much on the API". Find the expensive spans first (measure), then apply the cheapest wins. Don't guess - the trace tells you where the money goes.Votes: 0GitHub stars: 40
- Scrub Pii From TextUse this to strip PII (emails, credit cards, SSNs, phone numbers, IPs, IBANs) out of text before it is logged to a tracing backend or sent to a third-party model. Trigger on "redact PII", "scrub sensitive data", "mask PII before logging", "don't send customer data to the tracing tool", especially for finance/healthcare/regulated apps. Ships a runnable, tested scrubber with a Luhn check to cut false positives.Votes: 0GitHub stars: 40
- Set Up Ab TestingUse this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Measure impact on real users, gated, before a full switch.Votes: 0GitHub stars: 40
- Set Up Drift AlertsUse this to catch an LLM app silently getting worse in production - quality dropping, cost creeping up, inputs shifting away from what you tested. Trigger on "monitor my LLM in production", "alert me when quality drops", "detect drift", "my app got worse and I didn't notice", "set up monitoring/alerting for my AI app". Alert on the signals that actually move, not vanity metrics.Votes: 0GitHub stars: 40
- Trace Based TestingUse this to turn real production traces into a regression test suite, so bugs you saw once never come back. Trigger on "turn traces into tests", "capture this bug as a test", "build tests from production data", "my eval set is out of date", or after debugging a production incident. This is the loop that connects observability to evaluation.Votes: 0GitHub stars: 40
- Trace Multi Agent SystemUse this to add observability to a multi-agent or agentic system (multiple agents, sub-agents, tool loops, handoffs). Trigger on "trace my agents", "my agent crew is a black box", "which agent failed", "debug my LangGraph/CrewAI/AutoGen/agent workflow", "the agents loop forever". Get a clear span tree across agents and tool calls so you can see who did what.Votes: 0GitHub stars: 40
- Validate Genai SpansUse this to check that your LLM tracing actually emits complete, spec-compliant spans, so cost/latency/model dashboards downstream are not full of holes. Trigger on "are my traces complete", "validate my instrumentation", "my spans are missing fields", "lint my OTel GenAI spans", "test my tracing". Ships a runnable, tested validator you can drop into your instrumentation tests.Votes: 0GitHub stars: 40