Harness Test Advisor

# Harness Test Advisor

> Graph-based test selection. Answers: "I changed these files — what tests should I run?"

## When to Use

- Before pushing code — run only the tests that matter
- In CI — optimize test suite execution order
- When a test fails — understand which changes could have caused it
- When `on_pr` triggers fire
- **Coverage Audit mode** — when no diff is available and the user asks for "coverage gaps", "deep dive", "coverage plan", or "what's untested", run project-wide gap analysis instead of test selection
- NOT for writing tests (use harness-tdd, or `canary:canary-write-test` for uncovered files surfaced by Coverage Audit)
- NOT for test quality analysis at the test-selection level (Coverage Audit mode does include a capped quality review via `canary:canary-review-test`)

## Prerequisites

A knowledge graph at `.harness/graph/` enables full analysis. If no graph exists,
the skill uses static analysis fallbacks (see Graph Availability section).
Run `harness scan` to enable graph-enhanced analysis.

### Graph Availability

Before starting, check if `.harness/graph/graph.json` exists.

**If graph exists:** Check staleness — compare `.harness/graph/metadata.json`
scanTimestamp against `git log -1 --format=%ct` (latest commit timestamp).
If graph is more than 10 commits behind (`git log --oneline <scanTimestamp>..HEAD | wc -l`),
run `harness scan` to refresh before proceeding. (Staleness sensitivity: **Medium**)

**If graph exists and is fresh (or refreshed):** Use graph tools as primary strategy.

**If no graph exists:** Output "Running without graph (run `harness scan` to
enable full analysis)" and use fallback strategies for all subsequent steps.

## Process

### Phase 1: PARSE — Identify Changed Files

1. **From diff**: Parse `git diff --name-only` to get changed file paths.
2. **From input**: Accept comma-separated file paths.
3. **Filter**: Only consider `.ts`, `.tsx`, `.js`, `.jsx` files (skip docs, config).

### Phase 2: DISCOVER — Find Related Tests via Graph

For each changed file, use graph traversal to find test files:

1. **Direct test coverage**: Use `get_impact` to find test files that import the changed file.

   ```
   get_impact(filePath="src/services/auth.ts")
   → tests: ["tests/services/auth.test.ts", "tests/integration/auth-flow.test.ts"]
   ```

2. **Transitive test coverage**: Use `query_graph` with depth 2 to find tests that import files that import the changed file.

   ```
   query_graph(rootNodeIds=["file:src/services/auth.ts"], maxDepth=2, includeEdges=["imports"], bidirectional=true)
   ```

3. **Co-change tests**: Check `co_changes_with` edges for test files that historically change alongside the modified files.

#### Fallback (without graph)

When no graph is available, use naming conventions, import parsing, and git history:

1. **Tier 1 — Filename convention matching**: For each changed file `foo.ts`, search for:
   - `foo.test.ts`, `foo.spec.ts` (same directory)
   - `__tests__/foo.ts`, `__tests__/foo.test.ts`
   - Test files in a parallel `tests/` directory mirroring the source path
2. **Tier 2 — Import-linked tests**: Parse test files' import statements (grep for `import.*from` in `*.test.*` and `*.spec.*` files). If a test file imports the changed file, it belongs in Tier 2 (if not already in Tier 1).
3. **Tier 3 — Co-change correlated tests**: Use `git log --format="%H" --name-only` to find test files that frequently change in the same commit as the target file. Files that co-change in >2 commits are co-change correlated.
4. **Rank**: Tier 1 = direct filename match, Tier 2 = import-linked tests, Tier 3 = co-change correlated tests. Output the same tiered format as the graph version.

> Fallback completeness: ~80% — naming conventions and imports catch most mappings; misses dynamic imports and indirect coverage.

### Phase 3: PRIORITIZE — Rank and Generate Commands

Organize tests into three tiers:

**Tier 1 — Must Run** (direct coverage):
Tests that directly import or test the changed files. These are most likely to catch regressions.

**Tier 2 — Should Run** (transitive coverage):
Tests that cover code one hop away from the changed files. These catch indirect breakage.

**Tier 3 — Could Run** (related):
Tests in the same module or that co-change with the modified files. Lower probability of failure but worth running if time permits.

### Output

```
## Test Advisor Report

### Changed Files
- src/services/auth.ts (modified)
- src/types/user.ts (modified)

### Tier 1 — Must Run (direct coverage)
1. tests/services/auth.test.ts — imports auth.ts
2. tests/types/user.test.ts — imports user.ts

### Tier 2 — Should Run (transitive)
3. tests/routes/login.test.ts — imports routes/login.ts → imports auth.ts
4. tests/middleware/verify.test.ts — imports middleware/verify.ts → imports auth.ts

### Tier 3 — Could Run (related)
5. tests/integration/auth-flow.test.ts — same module, co-changes with auth.ts

### Quick Run Command
npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes/login.test.ts tests/middleware/verify.test.ts

### Full Run Command (all tiers)
npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes/login.test.ts tests/middleware/verify.test.ts tests/integration/auth-flow.test.ts
```

## Coverage Audit Mode

Activates when `--audit` is passed, OR when no diff is available AND the user's
language matches audit intent ("coverage gaps", "deep dive", "coverage plan",
"what's untested"). When this mode is selected, skip Phases 1–3 above and run
the audit phases below instead.

### Audit Phase 0: PROBE — Detect canary CLI availability

Call the `canary_probe` MCP tool once at the start of the audit. It returns
`{ status: "available" | "degraded", version?, reason? }` and never errors.

- **`available`** — the deterministic canary CLI is usable; use `canary_recommend_framework`
  for framework selection in Phase 3 (GAP REPORT).
- **`degraded`** — the CLI is not usable (reason: `not-installed`, `binary-missing`,
  `exec-failed`, or `bad-output`). Print one line:

  > canary CLI unavailable (`<reason>`) — install `canary-test-cli` for deterministic
  > framework recommendations. Proceeding; framework picks fall back to the plugin.

  Then skip `canary_recommend_framework` calls for the rest of the run. This does **not**
  affect the plugin-based Quality Review in Phase 2 (`canary:canary-review-test` is a
  separate Claude Code plugin, installed independently of the CLI).

### Audit Phase 1: INVENTORY — Build the Source-to-Test Map

1. **Enumerate source files**: glob for `.ts`, `.tsx`, `.js`, `.jsx` under the
   project's source roots (e.g., `src/`, `packages/*/src/`). Skip `node_modules`,
   `dist`, build outputs, and fixtures.
2. **Enumerate test files**: glob for `*.test.*`, `*.spec.*`, and files under
   `__tests__/` and parallel `tests/` directories.
3. **Map source → test**: for each source file, find its test file using the
   Tier 1 / Tier 2 / Tier 3 strategies described in `Phase 2: DISCOVER`. With a
   graph, prefer `get_impact`; without one, use naming conventions and import
   parsing.
4. **Split**: produce two lists — `covered` (source files with at least one
   matching test) and `uncovered` (source files with no matching test).

### Audit Phase 2: QUALITY REVIEW — Critique Covered Test Files

1. **Cap at 10 files per run**: a deep quality review is expensive. Pick the
   top 10 covered files prioritized by size (lines of code) and criticality
   (depth in the graph / co-change frequency). The remaining covered files are
   accepted as-is for this run.
2. **For each of the 10**: dispatch `canary:canary-review-test` against the
   test file. Collect its findings (missing edge cases, anti-patterns,
   brittleness, oracle gaps).
3. **Aggregate**: bin findings by severity (high / medium / low) and by file.

### Audit Phase 3: GAP REPORT — Synthesize a Unified Coverage Plan

Emit a single report with three sections:

```
## Coverage Audit Report

### Uncovered Files (no test found)
| File | Lines | Priority | Suggested Action |
|------|-------|----------|------------------|
| src/services/billing.ts | 412 | high | canary:canary-write-test |
| src/utils/format.ts     | 38  | low  | canary:canary-write-test |

### Quality Gaps (from Quality Review, capped at 10 reviewed)
| Test File | Severity | Top Gap |
|-----------|----------|---------|
| tests/services/auth.test.ts | high | no failure-path assertions |
| tests/utils/date.test.ts    | med  | mock leaks across cases |

### Recommended Next Steps
- Generate tests for high-priority uncovered files via `canary:canary-write-test`.
- Resolve high-severity quality gaps via `canary:canary-review-test` follow-ups.
- For uncovered files, pick a framework first: when Phase 0 reported `available`, call
  the `canary_recommend_framework` MCP tool with a prompt describing the file's purpose
  (deterministic, no LLM round-trip) and put its `framework` in the Suggested Action.
  When `degraded`, fall back to the `canary:canary-pick-framework` plugin.
```

## Harness Integration

- **`harness scan`** — Recommended before this skill for full graph-enhanced analysis. If graph is missing, skill uses naming convention and import parsing fallbacks.
- **`harness validate`** — Run after acting on findings to verify project health.
- **Graph tools** — This skill uses `query_graph`, `get_impact`, and `get_relationships` MCP tools.

## Success Criteria

- Tests prioritized into 3 tiers (Must Run, Should Run, Could Run)
- Executable run commands generated for quick and full test runs
- Coverage gaps flagged for changed files with no test coverage
- Report follows the structured output format
- All findings are backed by graph query evidence (with graph) or systematic static analysis (without graph)

## Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

| Rationalization                                                                                    | Why It Is Wrong                                                                                                                                                                     |
| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "Only the Tier 1 direct tests matter -- Tier 2 and Tier 3 are probably unnecessary"                | Tier 2 tests catch indirect breakage one hop away. A change to auth.ts breaks login.ts which breaks login.test.ts. Skipping Tier 2 misses exactly the regressions hardest to debug. |
| "The changed file has no tests, but that is not my concern -- I just advise on which tests to run" | Coverage gaps must be flagged. When a changed file has no test coverage, the advisor reports it. Silently producing an empty test list gives false confidence.                      |
| "The graph is stale but I will use it anyway since some data is better than no data"               | If the graph is more than 10 commits behind, refresh before proceeding. Staleness sensitivity is Medium for test advisor.                                                           |

## Examples

### Example: Selecting Tests for a Services Change

```
Input: git diff shows src/services/auth.ts and src/types/user.ts modified

1. PARSE    — 2 changed files identified (both .ts)
2. DISCOVER — get_impact(filePath="src/services/auth.ts")
              query_graph with depth 2 for transitive tests
              Tier 1: auth.test.ts, user.test.ts (direct imports)
              Tier 2: login.test.ts, verify.test.ts (one hop away)
              Tier 3: auth-flow.test.ts (co-change history)
3. PRIORITIZE — 5 tests across 3 tiers

Output:
  Tier 1 (must run): 2 tests
  Tier 2 (should run): 2 tests
  Tier 3 (could run): 1 test
  Quick command: npx vitest run auth.test.ts user.test.ts login.test.ts verify.test.ts
  Coverage gaps: none
```

## Gates

- **Graph preferred, fallback available.** If no graph exists, use naming conventions, import parsing, and git co-change analysis to identify relevant tests. Do not stop — produce the best test selection possible.
- **Always include Tier 1.** Direct test coverage is non-negotiable — always recommend running these (whether found via graph or naming conventions).

## Escalation

- **When changed file has no test coverage**: Flag as a gap: "No tests found for src/services/auth.ts — consider adding tests before merging."
- **When Tier 1 has >20 tests**: The changed file may be a hub. Suggest running Tier 1 in parallel or splitting the file.