All authors
Claude Skills by Mahalanobis-Systems
github.com/Mahalanobis-Systems8 skills0 installs11 views
- BuildCommands C5-C8 of aot-evals - size the error-finding and quality-estimate case sets with the arithmetic printed, author frozen and scrubbed cases from real traces, screen each case (human read, reference passes, do-nothing fails, no check trusts the agent's claim), and implement binary code checks and judge specs. Use when status reports C5-C8 unmet, or when writing eval cases or checks for an agent.Votes: 0GitHub stars: 5
- CalibrateCommands C9-C11 of aot-evals - collect stratified human labels per judge (spreadsheet or one at a time), calibrate each binary judge by TPR/TNR with thresholds chosen on labels (never raw agreement), compare System One, generative and custom backends on the same labels (E5), and measure self-consistency and cross-family agreement (C11). Use when an agent's evals include judge checks, when choosing between a System One model such as Jev and an LLM judge, or when status reports C9-C11 unmet.Votes: 0GitHub stars: 5
- DiscoverPhase 1 of aot-evals, usually reached through start. Runs when a team wants evals for an agent and no discovery report exists yet (evals/agents/<agent>/discovery.md), about 5 minutes. Shows the AOT EVALS banner and the two phases, scans the repository for every agent (each skill of a multi-skill system counts as its own agent), asks which agent to test, asks three questions about it, runs discovery in parallel, and writes a plain-English discovery report: an overview of the four things good e...Votes: 0GitHub stars: 5
- ReportCommand C18 of aot-evals - regenerate <suite>/report/report.json (fixed schema) and write one self-contained local HTML diagnostic from it, shaped around the user's question (why did the candidate regress? is this suite any good? what does it cost?). Use when someone asks for an eval report, dashboard, scorecard or diagnostic for an agent that has an evals/ directory.Votes: 0GitHub stars: 5
- RunCommands C12-C17 of aot-evals - run the baseline k=3-5 times with per-trial outcomes and plane-B counters, read the variance band, run the operational sweep (E14) against budgets, compare a candidate with a paired test (gains, losses, McNemar, trade verdicts), audit coverage against the external taxonomy, and measure redundancy with a prune proposal; drift and online evaluation are guidance only for now. Use when status reports C12 unmet, after any prompt/skill/tool/model/harness change, or w...Votes: 0GitHub stars: 5
- ScopeCommands C0-C4 of aot-evals - import the team's trace export, inventory the agent (agent.yaml, model per call site, stable window, invocation), build the external coverage taxonomy, categorize traces with worth/cost matrix/observed share, and decompose each category into failure modes with check types and plane-B budgets. Use when starting evals for an agent, or when status reports C0-C4 unmet.Votes: 0GitHub stars: 5
- StartThe one command to remember for aot-evals. Starts Phase 1 (discover) when this repository has no discovery report yet (evals/agents/<agent>/discovery.md); otherwise says where the evals stand and carries on with Phase 2 (status). Use for "start aot-evals", "set up evals", "build evals for my agent", "where are we with our evals", or "what's next".Votes: 0GitHub stars: 5
- StatusPhase 2 entry point, after discover has written a discovery report (evals/agents/<agent>/discovery.md); with no discovery report, load discover instead. Designing, building, measuring and iterating a production agent's evals. Reads the state of the repo's evals/ directory, reports which command gate (C0-C18) is unmet, and routes to the skill that clears it (scope, build, calibrate, run, report). Use when a team wants evals for an agent, asks "where are we with our evals", "what's next", or "i...Votes: 0GitHub stars: 5