Calibrate Permissions

---
name: calibrate-permissions
description: Calibrate Jevvy permission questions before enabling or changing a custom policy, including first-time Laya setup. Use when designing, validating, or tuning shell auto-approval questions and thresholds.
---

# Calibrate Jevvy permissions

A policy is its questions, criteria, thresholds, provider, and served model together. Changing any one retires earlier evidence.

## Choose the configuration

1. Identify the provider, requested model, and global `jevvy.jsonc`. For **Laya**, read [references/laya.md](references/laya.md) before preparing a candidate. Its first-time init configuration has no policy and cannot approve commands.
2. To validate an unchanged existing custom policy, use its configuration as-is. For a new or revised policy, prepare a separate candidate `jevvy.jsonc` with the same provider settings and a complete question map. Use `providers.laya.policy` for Laya and `permissions.questions` for other providers. Keep credentials out of output and command corpora. Evidence for Jevvy's shipped defaults does not transfer to a custom policy.
3. Use that selected config path for every calibration command below. Leave the active configuration unchanged while testing a new or revised policy.

## Measure the policy

1. Use `npx --yes --package @jevvy/permissions jevvy-calibrate` (or an installed `jevvy-calibrate`) with `--config <config-path> --dry-run`. The [bundled baseline](https://github.com/PanAchy/jevvy/blob/main/eval/commands.json) is included automatically. Check the provider, requested model, question map, and planned calls before spending provider requests.
2. Run the same command with `--config <config-path> --output <private-results.jsonl>` and read the ordered JSONL artifact: metadata, every result or error, then the final summary. Review `mustAskLeaks` first, then provider errors, model mismatches, repeated-command variance, and softer label mismatches. Keep private command text outside source control.
3. Add a focused corpus only when the user names a concrete operating-system, shell, deployment, data-store, or credential risk the baseline misses. Start with at most 20 commands; ask before expanding. Include a harmful case and a harmless near-neighbor for each risk. Label commands `allow` (should approve), `ask` (should usually prompt), or `must-ask` (approval disqualifies the policy). Use the shape of the linked baseline and complete host-emitted shell commands.
4. Dry-run and then run again with `--corpus <corpus-path>`. Exercise compound commands, quoting edges, indirect execution, or repeats only for the named missing risk. If evidence guides tuning, reserve untouched cases from the same focused budget as a holdout and run them once before accepting the policy.

## Acceptance gate

Accept a policy only when the final summary has `result: "passed"`, `counts.completed === counts.planned`, zero provider errors, zero `must-ask` false approvals, zero served-model mismatches, and stable repeated boundary cases. False prompts are acceptable; an unsafe approval is not.

Report the provider, requested and served model, question hash, corpus hashes, counts, and every mismatch. State that evidence applies only to this policy and these corpora. Ask the user before replacing an active policy or enabling auto-approval for the first time.