Harness Supply Chain Audit

# Harness Supply Chain Audit

> 6-factor dependency risk evaluation adapted from Trail of Bits security skill patterns. Surfaces dependency risk flags for human review — not automated verdicts.

## When to Use

- Before a major release to assess dependency risk
- After adding new dependencies
- During security audits or compliance reviews
- When `on_milestone` trigger fires (part of release gate)
- NOT as a replacement for `npm audit` — this complements it with risk signals beyond CVEs
- NOT for license compliance (separate concern)

## Iron Law

**Present findings as flags for human review, never as verdicts.** A dependency flagged as "high risk" may be entirely appropriate for a project. The skill surfaces signals; humans decide whether to act.

---

## Process

### Phase 1: INVENTORY — Build Dependency List

1. **Resolve project root.** Use the path argument or default to the current directory.

2. **Detect lockfile.** Check for the following in order:
   - `package-lock.json` (npm)
   - `pnpm-lock.yaml` (pnpm)
   - `yarn.lock` (yarn)
   - If none found: report "No lockfile detected. Run `npm install` first." and stop.

3. **Parse direct dependencies** from `package.json`:
   - Read `dependencies` and `devDependencies`
   - Build a list: `{ name, version, isDev }`

4. **Parse transitive depth** from lockfile:
   - For `package-lock.json`: read `packages` keys to extract the dependency tree. Nesting depth of `node_modules/` segments in keys indicates transitive depth.
   - For `pnpm-lock.yaml`: read `importers` section for direct dependencies (keyed by workspace path, e.g., `.` for root). Each importer lists `dependencies` and `devDependencies` with version specifiers. Read `packages` section for resolved versions — keys are package identifiers (e.g., `/@scope/pkg@1.2.3`) with `resolution` (tarball URL + integrity hash) and `dependencies` sub-map for transitives.
   - For `yarn.lock`: parse block-format entries. Each block header is `"pkg@version-range":` followed by indented fields: `version` (resolved), `resolved` (tarball URL), `integrity` (hash), and `dependencies` sub-block listing transitive deps as `"name" "version-range"` pairs.
   - Assign each package a depth (0 = direct, 1 = first-level transitive, etc.)
   - Flag packages with depth > 5 for transitive risk evaluation

5. **Build inventory table:**

   ```
   INVENTORY: <project-name>
   Direct dependencies: N
   Dev dependencies: N
   Total packages (including transitives): N
   Deep transitive packages (depth > 5): N
   ```

6. Proceed to EVALUATE.

---

### Phase 2: EVALUATE — Score Dependencies on 6 Factors

For each **direct dependency** (and any transitive with depth > 5), score on 6 factors:

> Network access required: npm registry (`https://registry.npmjs.org/<pkg>`) and GitHub API (`https://api.github.com/repos/<owner>/<repo>`).
>
> - If npm registry returns 404: mark as "unresolvable", flag for manual review, skip remaining factors
> - If GitHub API rate limits hit: score `maintenance-status` as "unknown", continue with other factors
> - If no GitHub repo link in package metadata: skip `maintenance-status` factor, note in report

#### Factor 1: Maintainer Concentration

- Fetch: `GET https://registry.npmjs.org/<pkg>`
- Check: `maintainers` array length
- Score:
  - **High risk:** 1 maintainer (bus factor = 1)
  - **Medium risk:** 2-3 maintainers
  - **Low risk:** 4+ maintainers

#### Factor 2: Maintenance Status

- Source: npm `time` field (last publish date) + GitHub API commit activity
- npm: `GET https://registry.npmjs.org/<pkg>` → `time.modified`
- GitHub: `GET https://api.github.com/repos/<owner>/<repo>/commits?per_page=1` → latest commit date
- Score:
  - **High risk:** Last publish > 12 months ago AND no GitHub commits in 6 months
  - **Medium risk:** Last publish > 12 months ago OR no commits in 6 months (not both)
  - **Low risk:** Active in both dimensions

#### Factor 3: Popularity Signal

- Fetch: `GET https://api.npmjs.org/downloads/point/last-week/<pkg>`
- Score:
  - **High risk:** < 1,000 weekly downloads
  - **Medium risk:** 1,000–10,000 weekly downloads
  - **Low risk:** > 10,000 weekly downloads
  - **Note:** Low popularity is a signal, not a verdict — internal/niche packages are expected to be low

#### Factor 4: Install Scripts

- Read: `node_modules/<pkg>/package.json` (or lockfile-resolved path) → `scripts` field
- Check for: `preinstall`, `postinstall`, `install`, `preuninstall`, `postuninstall`
- Score:
  - **High risk:** Any install script present
  - **Low risk:** No install scripts
  - **Note:** Some install scripts are legitimate (native addon compilation). Flag for review.

#### Factor 5: Known CVEs

- Run: `npm audit --json` or `pnpm audit --json`
- Parse: map findings to their package name
- Score:
  - **Critical:** Any high/critical severity CVE
  - **Medium risk:** Moderate severity CVE
  - **Low risk:** No CVEs or low severity only

#### Factor 6: Transitive Risk

- Source: Lockfile depth analysis from INVENTORY phase
- Score:
  - **High risk:** Depth > 5 AND subtree size > 20 transitive packages
  - **Medium risk:** Depth > 5 OR subtree size > 20
  - **Low risk:** Depth ≤ 5 and subtree size ≤ 20

#### Risk Scoring

Combine factor scores into an overall risk level:

| Overall Risk | Condition                                                      |
| ------------ | -------------------------------------------------------------- |
| **Critical** | Factor 5 is Critical (any high/critical CVE)                   |
| **High**     | 2+ factors scored High, OR Factor 1 is High + Factor 2 is High |
| **Medium**   | 1 factor scored High, OR 3+ factors scored Medium              |
| **Low**      | All factors Low or at most 1 Medium                            |

---

### Phase 3: REPORT — Generate Risk Summary

1. **Produce risk summary table** sorted by overall risk (Critical first):

   ```
   Supply Chain Audit: <project-name>
   Date: <ISO date>
   Packages evaluated: N direct + M deep transitives

   ┌─────────────────────┬──────────┬────────────┬─────────────┬────────────┬──────┬─────────────┐
   │ Package             │ Version  │ Maintainers│ Last Publish│ Downloads  │ CVEs │ Overall Risk│
   ├─────────────────────┼──────────┼────────────┼─────────────┼────────────┼──────┼─────────────┤
   │ example-pkg         │ 1.2.3    │ 1 (HIGH)   │ 18mo (HIGH) │ 500 (MED)  │ none │ HIGH        │
   │ another-pkg         │ 2.0.0    │ 12         │ 2mo         │ 50k        │ 1 mod│ MEDIUM      │
   └─────────────────────┴──────────┴────────────┴─────────────┴────────────┴──────┴─────────────┘
   ```

2. **Detail section for Critical and High risk packages:**

   ```
   HIGH RISK: example-pkg@1.2.3
   ├── Maintainer concentration: 1 maintainer (bus factor = 1)
   ├── Maintenance status: Last publish 18 months ago, no commits in 12 months
   ├── Popularity: 500 weekly downloads
   ├── Install scripts: none
   ├── Known CVEs: none
   └── Transitive risk: depth 2, subtree 4 packages
   Recommendation: Consider replacing with a well-maintained alternative,
   or pin the version and monitor for abandonment.
   ```

3. **Install script warnings** (any package with install scripts):

   ```
   INSTALL SCRIPTS DETECTED:
   - node-gyp@9.4.0: postinstall — native addon compilation (likely legitimate)
   - suspicious-pkg@1.0.0: postinstall — review script contents before trusting
   ```

4. **Summary line:**

   ```
   RESULT: 1 Critical, 2 High, 3 Medium, N Low — Review flagged items before release
   ```

5. **Output:** Print report to stdout. If `--output <file>` was passed, also write to that file.

---

## Gates

- **Stop if no lockfile.** Do not evaluate without a lockfile — results will be unreliable.
- **Present as flags, not verdicts.** Never state "this package is unsafe." State "this package has signals that warrant review."
- **Do not block on API failures.** If npm registry or GitHub API is unavailable, note which factors were skipped and continue with available data.

## Harness Integration

- **`harness validate`** — Run after creating the skill files to verify they are properly placed.
- **Triggers:** `on_milestone` fires this skill as part of the milestone completion checklist.
- **Depends on:** `harness-security-scan` — run after mechanical scanning to complete the security picture.
- **Output:** Stdout report, optionally written to file via `--output`. No state files written.

## Evidence Requirements

When reporting findings, cite the source for each factor:

- Maintainer data: `registry.npmjs.org/<pkg>` → `maintainers` field
- Publish date: `registry.npmjs.org/<pkg>` → `time.modified`
- Downloads: `api.npmjs.org/downloads/point/last-week/<pkg>`
- Install scripts: `node_modules/<pkg>/package.json` → `scripts`
- CVEs: `npm audit --json` output
- Depth: lockfile analysis

Do not assert risk scores without citing the specific data point that generated the score.

## Success Criteria

- Running `/harness:supply-chain-audit` on a project with dependencies outputs a risk table with all 6 factors scored
- A dependency with a sole maintainer and no commits in 12 months scores "high risk"
- A dependency with a `postinstall` script is flagged in the install scripts section
- API failures produce "unknown" scores with a note, not errors that stop the audit
- All findings are framed as flags for human review, not automated verdicts

## Escalation

- **If a critical CVE is found:** Surface immediately — do not bury it in the table. Recommend blocking the dependency update or requiring an immediate patch before merge.
- **If all maintainers are unresponsive:** Flag the package as abandoned and recommend finding an alternative. Include download counts to help the user assess how widely adopted the package is.
- **If an install script has unknown behavior:** Do not guess. State that the script requires manual review and link to the script source.
- **If the npm or GitHub API is unavailable:** Note which factors were skipped with "unknown" scores. Do not fail the audit — partial results are better than none.
- **If the user asks for a verdict ("is this safe?"):** Decline to give a binary answer. Supply chain risk is probabilistic. Present the risk signals and let the human decide.

## Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

| Rationalization                                                                             | Why It Is Wrong                                                                                                                                        |
| ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| "This package has high risk signals but it is widely used, so it must be safe"              | The Iron Law: present findings as flags for human review, never as verdicts. Popularity does not eliminate bus-factor risk or maintenance abandonment. |
| "The npm API returned an error for this package, so I will skip it and move on"             | API failures produce "unknown" scores with a note, not skips. Partial results with noted gaps are always better than incomplete audits.                |
| "The install script is probably just native addon compilation, so I do not need to flag it" | Every install script must be flagged in the report. "Probably legitimate" is exactly the assumption that supply chain attacks exploit.                 |

## Examples

```
Supply Chain Audit: my-project
Date: 2026-03-31
Packages evaluated: 24 direct + 3 deep transitives (depth > 5)

CRITICAL (1):
  lodash@4.17.20 — CVE-2021-23337 (high severity, unpatched)

HIGH (2):
  abandoned-util@0.9.1 — sole maintainer, last publish 22 months ago
  sketchy-helper@2.1.0 — sole maintainer, postinstall script detected

MEDIUM (3):
  small-lib@1.0.0 — 800 weekly downloads (low popularity signal)
  ...

LOW (18): no significant risk signals

INSTALL SCRIPTS:
  node-gyp@9.4.0 — postinstall (native compilation, likely legitimate)
  sketchy-helper@2.1.0 — postinstall (REVIEW: contents unknown)

RESULT: 1 Critical, 2 High, 3 Medium, 18 Low
Next steps: Update lodash to patch CVE. Review sketchy-helper postinstall script.
Consider alternatives to abandoned-util.
```

## Example Output

```
Supply Chain Audit: my-project
Date: 2026-03-31
Packages evaluated: 24 direct + 3 deep transitives (depth > 5)

CRITICAL (1):
  lodash@4.17.20 — CVE-2021-23337 (high severity, unpatched)

HIGH (2):
  abandoned-util@0.9.1 — sole maintainer, last publish 22 months ago
  sketchy-helper@2.1.0 — sole maintainer, postinstall script detected

MEDIUM (3):
  small-lib@1.0.0 — 800 weekly downloads (low popularity signal)
  ...

LOW (18): no significant risk signals

INSTALL SCRIPTS:
  node-gyp@9.4.0 — postinstall (native compilation, likely legitimate)
  sketchy-helper@2.1.0 — postinstall (REVIEW: contents unknown)

RESULT: 1 Critical, 2 High, 3 Medium, 18 Low
Next steps: Update lodash to patch CVE. Review sketchy-helper postinstall script.
Consider alternatives to abandoned-util.
```