Project State Graph

---
name: project-state-graph
description: Use when setting up a new project, creating a new app, kicking off a new initiative, onboarding an existing repo, or whenever you need a repo graph / code graph / project state graph / map of a codebase before working in it. Keywords - set up project, new project, new app, new initiative, project state graph, repo graph, code graph, codebase map, architecture overview.
---

# Project State Graph

## Overview

Initialize a complete, two-layer **state graph** for ANY repo with one command, so future work has an accurate map of the codebase before touching it.

- **Shallow layer** — `ARCHITECTURE.md`: human-readable overview, file list, subsystem grouping, pointers into the deep layer.
- **Deep layer** — `<name>-state-graph.db`: a SQLite graph for precise impact analysis — `node` / `edge` rows typed by `node_type` / `edge_type` (`function`, `class`, `data_var`, `column`, …), a `consistency_card` + `symbol_card` per callable, the `analysis_run` log, and the append-only `node_snapshot` / `node_event` history.

A global **registry** (`projects.json` + `PROJECT-STATE-GRAPHS.md` index) tracks every project graph on record.

**Phase A (provLedger) — stored assumed-schema:** when code reads an external
source, the analyzer records the columns/keys the code *expects* it to return as
`metadata_json["assumed_schema"]` on `sql_table` / `bq_dataset` nodes (SQL SELECT
projection) and `api_source` nodes (response subscript keys). This feeds the
reviewer's contract-drift gates and Phase D pre-flight. Purely additive — no
schema migration (generic property graph).

**v2 (data-science aligned):** beyond code structure, the graph now models data as
first-class citizens — `dataframe`/`column`/`dataset` nodes with **dtypes + provenance**,
`api_source` nodes, ML overlay (`split`/`model`/`hyperparameter`), DE `lineage`
(upstream→op→downstream), and sub-flow `profile` tags (function-calling, pipeline,
data-flow, ml-training, data-engineering, endpoint). `data-flow` requires **real
flow** (a consumed produced value, or a consumed input) — merely returning a value
nobody uses does NOT qualify; route handlers are tagged `endpoint` instead. The
visualization shows the full graph by
default with a **sub-flow lens** and **click-a-node end-to-end focus**. New selfcheck
gates enforce dtype presence (warn), end-to-end dtype consistency (error), and lineage
integrity (error).

**Boundary:** This skill covers *building and registering* a project state graph and stops at *querying it for a specific task* (that's normal analysis work against the produced DB).

## When to Use

- Starting a new project / app / initiative and want a baseline map.
- Onboarding an unfamiliar existing repo before making changes.
- Refreshing the graph after significant code changes.

**When NOT to use:** day-to-day querying of an already-built graph — just `SELECT` from the deep DB. Building skills themselves → `writing-skills`.

## The 5-Stage Flow

Decide a unique `--name` and the `--repo` path; `init_project.sh` then runs five
stages, in this order, deterministically:

1. **Deep layer** — the analyzer produces `<name>-state-graph.db`.
2. **Shallow layer** — `ARCHITECTURE.md` is generated from the DB.
3. **Registry** — `projects.json` and the global index `PROJECT-STATE-GRAPHS.md` are updated.
4. **Verify** — every error-severity `selfcheck` invariant must PASS (see *Self-check severity model*); a failure exits non-zero, though stage 3 has already registered the graph.
5. **Slices** — the DataFrame-aware slices HTML (non-fatal; see Phase B below).

```bash
bash ${CLAUDE_PLUGIN_ROOT}/skills/project-state-graph/scripts/init_project.sh \
  --name <project-name> --repo <repo-path> [--out-dir <dir>]
```

Default `--out-dir` is `~/skill-workspace/project-graphs/<name>/`; the header of
`init_project.sh` lists the environment overrides (`PSG_REGISTRY_ROOT`,
`PSG_REGISTRY_PATH`, `PSG_INDEX_PATH`).
On success you get the deep DB + `ARCHITECTURE.md` in the out dir, a registry entry,
the regenerated index, and a `Self-check: PASS` line. Non-zero exit on any failure.

## Agent profiling phase (data-science routing + gap-only dtype probes)

`init_project.sh` is fully deterministic and **always runs every analyzer**
(code, data-flow, data-model, API, pipeline, profiles, and the ML/DE overlays).
On top of that, the agent performs a lightweight **profiling phase** before/around
the build to maximise data coverage:

1. **Inspect repo settings** — read `pyproject.toml`/`requirements`, imports, and
   a sample of modules to decide which sub-flows are present (pandas, pyspark,
   sklearn/torch, SQL/BigQuery, HTTP APIs).
2. **Static first.** The analyzers capture dtypes from annotations, `.astype()`,
   `dtypes=`, `StructType`, pandera/pydantic schemas, and literal `df["col"]`
   access. Every typed node records a `dtype_provenance` on the ladder
   `declared-schema -> annotation -> static-inference -> runtime-probe -> unknown`.
3. **Probe only the gaps.** For `column`/`data_var` nodes left `dtype = "unknown"`,
   and ONLY those, the agent may emit a tiny throwaway probe script **in the
   project's own venv** that imports the code, builds/inspects a small sample
   frame, prints the dtype, then records it back onto the node with
   `dtype_provenance = "runtime-probe"`. Never run more than needed; never probe
   what static analysis already resolved.
4. **Audit.** Because every node says HOW its type is known, a reviewer can trust
   `declared-schema`/`annotation` immediately and scrutinise `runtime-probe` /
   `unknown`.

This keeps determinism (the build never depends on probes) while letting the
agent enrich coverage where it matters.

## Self-check severity model

`selfcheck.py` runs deterministic invariants in two tiers. A failing **error**
check makes it exit 1, which aborts `init_project.sh` (under `set -e`) and fails a
review; a failing **warning** prints `[WARN]` and never changes the exit code.
The error-severity checks — the ones that can block a build:

| Invariant (error) | Fails when |
|---|---|
| `node_types_nonempty` | the graph is empty |
| `no_dangling_edges` | an edge endpoint references no node |
| `cards_match_callables` | function/method nodes and their consistency + symbol cards do not match one to one |
| `commit_sha_set` | the latest `analysis_run` recorded no git SHA |
| `no_undefined_symbols` | an `unresolved_call` node exists: a bare-name call (`foo()`, never `obj.foo()`) that resolves to no project callable, import, builtin or local. Conservative — a rename or typo trips it; legitimate imports, builtins and methods never do |
| `dtype_consistency_e2e` | a produced dtype disagrees with the dtype a consumer declares (high-confidence `produces`/`consumes` edges only) — an end-to-end data break |
| `lineage_no_dangling` | a `derives`/`transforms`/`feeds`/`lineage` edge endpoint references no node |
| `no_data_leakage` | a model fits and evaluates on same-source data without a split (a `leakage` node) |
| `history_append_only` | an append-only trigger on `node_snapshot` / `node_event` is missing |
| `history_key_coverage` | an identity-bearing node of the latest run has no `node_key` |

Every other check is a warning. The full list, with each check's severity, is
`_CHECKS` in `scripts/selfcheck.py`; `uv run python selfcheck.py <db>` prints
every check's result.

## DataFrame-aware slices (Phase B)

Beyond the whole-graph `visualize.py`, a peer renderer `viz_slices.py`
(`analyzer/slice_viz.py`) emits **four per-perspective slices** into ONE
self-contained, `file://`-safe HTML file so a human can *eyeball-validate* the
graph (a hairball of hundreds of nodes is unreviewable). Stdlib-only; no analyzer
imports (byte-portable into the reviewer skill, like `graph_viz.py`).

| # | Slice | What it shows |
|---|---|---|
| 1 | **Dataflow + datatype** | data_var/column nodes along produces/consumes/feeds, each labeled with dtype. **Unknown dtypes render gray (`#9aa3af`) with a `?`** so coverage holes are visible at a glance. A dtype-coverage badge (`N% typed`) sits in the header. |
| 2 | **Function call chain** | a focus function's callers (above) + callees (below) within N hops — the neighborhood, not the whole graph. Interactive focus box. |
| 3 | **Pipeline view** | `pipeline_step` edges grouped by pipeline, in execution order (hierarchical LR). |
| 4 | **API surface** | a **table** (not a diagram): path · method · handler · what the handler calls. |

`init_project.sh` runs this automatically as stage **[5/5]** (additive,
non-fatal) -> `<out-dir>/<name>-slices.html`. The unknown=gray convention also
feeds the Phase C-2 dtype-coverage metric (gate strength == dtype coverage).

## Cold backup + dtype coverage (Phase C)

Two low-cost protections run automatically during `init_project.sh`:

- **C-1 · Cold snapshot.** Before the deep-layer rebuild resets the graph rows
  (nodes, edges and cards; the `analysis_run` log and the node history persist),
  `archive_db.sh` copies the DB to `provledger.<old_sha>.db` (sha from the prior
  `analysis_run`, timestamp fallback). It is a **cold archive — not in any query
  path** — the raw material for future version-over-version provenance ("how did
  this symbol change across versions?") and the ledger's fuzzy search. The delta
  logic can come later; the snapshots **cannot be captured retroactively**.
  Non-fatal: a failed archive never blocks the rebuild.
- **C-2 · dtype coverage as a tracked metric.** Every data gate is exactly as
  strong as dtype coverage — a `data_var` left `dtype=unknown` makes its check
  pass through silently (green where it is actually blind). `selfcheck.py` now
  prints a headline **`dtype coverage: N% (typed/total typed)`** line (warning
  severity, never blocks) alongside the per-node `dtype_present` detail. The same
  number shows on the Phase B dataflow slice badge.

## Quick Reference

| Action | Command |
|---|---|
| Init / refresh a project graph | `init_project.sh --name N --repo R` |
| Build deep layer only | `uv run python -m analyzer <repo> --project N --db-path P` |
| Generate ARCHITECTURE.md only | `uv run python architecture_md.py <db> <name> <out>` |
| Render the 4 DataFrame-aware slices | `uv run python viz_slices.py <db> <out.html> [--title N] [--focus QNAME]` |
| Cold-snapshot a DB before overwrite | `bash archive_db.sh <db>` |
| Verify a built DB (+ dtype coverage %) | `uv run python selfcheck.py <db>` |
| History of one node (events + run attribution) | `uv run python -m analyzer history <db> <qualified_name\|node_key>` |
| Replay past commits into a FRESH graph (phase 7) | `uv run python -m analyzer backfill <repo> --project N --db-path P --since SHA [--until HEAD] [--every N] [--subdir DIR] [--fresh-db]` |
| Export identity ambiguities for calibration (phase 7) | `uv run python -m analyzer ambiguities <db> --export calib.json --repo R` |
| Build a calibration set by construction, or print its distribution (phase 8) | `uv run python -m analyzer calibration generate --out calib.json [--corpus DIR] [--include-live DB [--repo R]] [--merge F ...]` · `uv run python -m analyzer calibration stats calib.json` |
| Evaluate an arbiter against a calibration file (phase 7) | `uv run python -m analyzer arbiter-eval calib.json --arbiter pkg.mod:Class` — see `docs/arbitration.md` |
| List projects on record | read `~/skill-workspace/project-graphs/projects.json` |

All Python entry points run inside the skill's `uv` env (`cd scripts && uv run ...`).

## Common Mistakes

- **Skipping verification.** Always confirm `Self-check: PASS` — a graph that fails invariants is not trustworthy.
- **Committing generated graphs.** `*.db` and `*-state-graph.*` are git-ignored in the skill dir; keep graphs under `~/skill-workspace/project-graphs/`, never in the skill or the target repo.
- **Re-running on a dirty tree.** The analyzer warns when the repo has uncommitted changes — the graph may not match committed code. It reads the working tree, untracked files included; directories are skipped by name (`.venv`, `.venv-*`, `node_modules`, `build`, … — see `analyzer/walker.py`), not by `.gitignore`.