Security Regression Tests

---
name: security-regression-tests
description: "Turning a confirmed and fixed finding into a permanent guard: proving the test fails without the fix, asserting the effect and not only the status code, seeding the two principals an authorization test needs, making a timing or out-of-band proof deterministic enough for CI, and keeping the test in a job that can actually block. Use once a security finding has been confirmed and fixed, or when a security test is failing, skipped, or about to be deleted."
---

<!-- Native skill bundle for agent-skills (cross-tool convention). Generated by `secure-vibe dev regenerate`. -->
<!-- Do not edit by hand; the source of truth is skills/security-regression-tests/SKILL.md. -->

# Security Regression Tests

Turning a confirmed and fixed finding into a permanent guard: proving the test fails without the fix, asserting the effect and not only the status code, seeding the two principals an authorization test needs, making a timing or out-of-band proof deterministic enough for CI, and keeping the test in a job that can actually block. Use once a security finding has been confirmed and fixed, or when a security test is failing, skipped, or about to be deleted.

## ALWAYS

- **Prove the test fails without the fix.** Revert the patch, or mutate the guard, and confirm the new test goes red — then restore. A test written after the fix can be vacuously green: wrong route, wrong fixture user, wrong assertion target, a typo in the payload. It will stay green forever and nobody will learn anything from it. This is the one step that distinguishes a guard from a decoration, and it is the one most often skipped.
- Assert **both directions**: the attack input now yields the secure outcome, *and* a legitimate input still succeeds. A deny-only test passes equally well against an endpoint that is broken for everyone, so it will hold the line and tell you nothing when someone disables the feature entirely.
- Assert the **effect**, not only the status code. A `403` proves the request was refused; it does not prove the write did not land somewhere else, that the record was not returned in a different shape, or that a queue message was not emitted. Check the state the attack was trying to reach.
- Seed the **fixtures the assertion depends on** before writing it. A cross-tenant test needs two principals and two owned resources, deterministically created — if both test users end up in the same tenant, the deny assertion is untestable and passes for the wrong reason.
- Make it **deterministic and offline**. Freeze the clock for expiry and TTL assertions, stub the HTTP client, dialer or resolver rather than reaching the network, fix random seeds, and avoid sleep-based timing assertions and state shared between tests. Note that a canonical SSRF payload such as `http://169.254.169.254/latest/meta-data/` is a *live, reachable* address on a cloud CI runner, so the naive test both leaves the sandbox and returns a different verdict on a laptop.
- Graduate a live proof-of-concept by keeping its **shape and dropping its mechanism**. `dynamic-verification` confirms some classes with a timing delta or an out-of-band callback, and neither survives in CI: replace the time-based payload with an assertion that the query was parameterized, and the callback with a stubbed dialer asserting no outbound request was attempted.
- Put the test in a **job that can block the merge**. A committed test in a job that is `continue-on-error`, not a required check, or nightly-only is documentation. `cicd-security` owns the gate configuration itself.
- Name it so a reviewer sees what it guards — `test_idor_orders_cross_tenant_403` — and reference the finding in the test body or the commit, so the next person to see it fail knows what it is protecting rather than what it is blocking.

## NEVER

- Close a confirmed finding without a regression test. "Fixed by inspection" rots: the next refactor reintroduces it, and there is nothing to notice.
- **Skip, quarantine, `xfail`, or delete** a security test to get a build green. A skipped security test is a removed control that still looks like a present one, and it is indistinguishable in a passing build from one that runs. A failing security test is either a real regression or a wrong test — both need a person, neither needs a decorator.
- Relax an assertion until it passes. Widening an expected status set, dropping the effect check, or accepting either outcome converts the guard into a formality.
- Put a **live-target probe in CI** as the verification. Attack traffic from a build agent is non-deterministic, sometimes illegal against infrastructure you do not own, and the reason the committed test exists.
- Put a **real** secret in a fixture to exercise a secret check. Use the documented sentinels that scanners already exclude by name — `secret-detection` owns them.

## KNOWN FALSE POSITIVES

- A regression test necessarily **commits attack payloads**: `' OR 1=1--`, `<script>alert(1)</script>`, `{{7*7}}`, `../../etc/passwd`, a fake token. The project's own SAST and secret scanners will re-flag them on the next run. That is the test doing its job — exclude the fixture path, do not weaken the payload.
- A **refuted** candidate never triggered the rule: there is no fix, so there is nothing to pin. This is not an exemption, it is the precondition not being met.
- A finding fixed in configuration rather than code — a base image, a workflow permission, a cloud setting, a dependency bump — is pinned by the **scanner gate** rather than a unit test, because there is no application behaviour to assert. The gate then needs the same both-directions proof: a fixture that fails it and one that passes, or nobody can tell a working gate from a misconfigured one.
- A test asserting a *narrower* outcome than the original report — a `404` where the report said `403`, because the application hides existence — is correct if that is the intended behaviour. Match the application's contract, not the report's wording.

## Reference files

Read these only when the task calls for them.

- `references/test-by-class.md`
- `references/where-verification-runs.md`