Kg Schema Maintenance

---
name: kg-schema-maintenance
description: Library-level procedures for maintaining a genai-graph schema across factories — assess whether a Pydantic field is dead code, merge/unify two node types across factories, add a node type or field, and diagnose schema warnings ("Class X referenced but has no GraphNode", label collisions, orphaned nodes, duplicate relationships). Use when tidying a multi-factory KG, interpreting a warnings report, or generalizing an Agents_Skills.md procedure past a single project.
---

# GenAI Graph Schema Maintenance

Reusable procedures for common schema upkeep. These generalize the project-specific
procedures in `Agents_Skills.md` to the library level. Reference `kg-schema` for the
schema model and `kg-export` for the warnings report these procedures act on.

## Read First

- `Agents_Skills.md` — the project-specific version of these procedures
- `docs/schema-compilation.md` — field-path deduction, `table_name`, exclusion mechanics
- `docs/graph_construction.md` — canonical types, schema merging
- `docs/baml_extraction_guide.md` — Pattern 4 (cross-factory node unification)
- `genai_graph/kg/schema/core.py` — `GraphNode`, `GraphRelation`, `GraphSchema`
- `genai_graph/kg/schema/compiler.py` — `validate_schema_coherence`, `compute_excluded_fields`
- `genai_graph/kg/schema/registry.py` — `GraphRegistry.build_combined_schema` (label dedup)
- `genai_graph/kg/factories/neo4j_factory.py` — auto-populates `id` from `key_field`
- `genai_graph/kg/ingest/extract.py` — `metadata_field_names` (fields skipped from columns)

---

## Procedure 1 — Assess whether a Pydantic field is dead code

**When**: the user asks whether a field can be removed from a node model.

1. **Is it a `property_mappings` target?** (Neo4j factories) If the field is not a target in
   any `Neo4jNodeMapping.property_mappings`, it is never populated from source data.
2. **Is it `key_field`?** If so, it is critical — do not remove.
3. **Is it in `index_fields`?** If so, removing it breaks embedding/vector indexing.
4. **Is it in `metadata_field_names`** (`kg/ingest/extract.py`)? Fields like `"id"` are
   auto-generated by the factory, not read from the Pydantic model.
5. **Grep for `.field_name`** across the package and tests — any code accessing it on model
   instances?
6. **Is it a relation endpoint / edge property?** Relation endpoint fields are excluded from
   the node and stored on the relation; `p_`-prefixed fields are edge properties. These are
   not dead — they live on the relation.
7. **Verdict**: not mapped, not key/index, not accessed, not an edge property → safe to remove.

### Common case: the `id` field

`Neo4jImportFactory` auto-generates `mapped_props["id"]` from `key_field`
(see `neo4j_factory.py`), and `"id"` is in `metadata_field_names` (excluded from schema
column generation). Unless `key_field="id"`, an `id: str` field on a Neo4j-mapped Pydantic
model is dead code — remove it.

---

## Procedure 2 — Merge / unify two node types across factories

**When**: two factories define semantically identical node types under different names
(e.g. `TechnologyPartner` in one factory and `Partner` from a BAML extraction).

### Decide the canonical name

- The canonical class `__name__` determines the Ladybug table name (`label`).
- If one type comes from BAML (auto-generated), the canonical class **must keep the same
  `__name__` as the BAML type**, because you extend it (`class Partner(BamlPartner): ...`).
  The table name follows the class name; renaming would create a second table.
- So if BAML defines `Partner`, the canonical name is `Partner`.

### Steps

1. Create the canonical class in a shared `canonical_nodes.py` (extend the BAML type when
   relevant): `class Partner(BamlPartner): ...`.
2. Define a `GraphNode` singleton there: `PartnerNode = GraphNode(node_class=Partner,
   name_from="name", key_from="name", description="...")` and export it.
3. In the factory that owned the old type, delete the old class and import the canonical
   `GraphNode` singleton; keep `neo4j_label` (Neo4j) matching the source data but point
   `node_class` at the canonical class.
4. Update `GraphRelation` references to pass the canonical `GraphNode` (or its class).
5. In other factories using the type, switch from importing the raw BAML class to importing
   the canonical `GraphNode` singleton.
6. `GraphRegistry.build_combined_schema()` dedups by `label`, so both factories now produce
   one `Partner` table. Verify with `validate_schema_coherence`.
7. Update docs (`docs/graph_construction.md` canonical types table,
   `docs/baml_extraction_guide.md` Pattern 4) and run tests.

---

## Procedure 3 — Add a new node type to a factory

1. **Check if the type already exists** in a shared `canonical_nodes.py` or BAML types. If
   yes, import and reuse the `GraphNode` singleton.
2. **Define the Pydantic model** in the factory file with `Field(description=...)` per
   property. Use `str | None = None` for optionals. Do **not** add an `id: str` field unless
   `key_field="id"` (Neo4j/Json factories generate `id`).
3. **Add the `GraphNode`** (or `Neo4jNodeMapping`) with `name_from`, `key_from`, and
   `index_fields` (for embedding fields).
4. **Add relationships** (`GraphRelation` / `Neo4jRelationMapping`) if the node participates,
   and the rel type string to `get_included_rel_types()` (Neo4j).
5. **Run `validate_schema_coherence`** and fix warnings (orphan, label collision,
   referenced-but-no-GraphNode).
6. **Re-ingest** with `--force graph` so the new table is created.

---

## Procedure 4 — Add a field to an existing node type

### Neo4j factories
1. Add the field to the Pydantic model (`| None = None` if optional).
2. Add the mapping in `property_mappings`: `{"neo4jProp": "pydantic_field"}`.
3. If it should be searchable, add to `index_fields`.
4. Re-ingest — Ladybug `ALTER TABLE ADD`s the new column automatically on next `kg create`.

### BAML factories
1. Add the field to the `.baml` schema file.
2. Run `baml-cli generate` to regenerate `baml_client/`.
3. If the type is extended in `canonical_nodes.py`, add the field there (not in generated code).
4. Re-ingest.

### JSON factories
1. Add the field to the Pydantic model.
2. Re-ingest (`--force parquet` if the JSON cache must reflect new columns).

---

## Procedure 5 — Diagnose schema warnings

Run `validate_schema_coherence(schema)` / `schema.get_warnings()`, or read the consolidated
`{profile}-{tag}-warnings.md` report (see `kg-export`). Categories and fixes:

| Warning | Root cause | Fix |
|---|---|---|
| `Class X is referenced in relationships but has no GraphNode` | A `GraphRelation` references a class with no `GraphNode`/`Neo4jNodeMapping` | Add a `GraphNode(node_class=X, ...)`; or unify the type (Procedure 2); or embed the struct on the parent instead of relating |
| `Two different node classes share the label 'X'` | Two classes resolve to the same `label` | Set `table_name` on one (and update relations that reference it — relations dedup by label) |
| `No field paths found for X` | Node not reachable from `root_model_class` | Add a path from the root model, set `field_paths` explicitly, or mark `explicitly_defined=True` (mapping-defined nodes) |
| 🔄 Duplicate relationships | Multiple rel types between the same node pair (e.g. `HAS_CUSTOMER` and `FOR_CUSTOMER`) | Consolidate to one semantically clear rel type |
| 🔗 Orphaned nodes | Node not reachable from any root | Wire it into a relation path or drop it |
| ❌ Schema creation failures | A subgraph schema raised | Inspect the per-factory error in the warnings log; usually a bad `key_from`/`table_name` |

### Quick check after any fix

```python
from genai_graph.kg.schema import validate_schema_coherence

print(validate_schema_coherence(schema))  # expect []
```

```bash
cli kg schema --regen --kg <profile>
cli kg create <profile> --force graph
```

---

## General rules

- Use absolute imports; keep canonical types in one module imported by all factories.
- Prefer removing dead/compat code over keeping it (per `Agents.md`).
- After any schema change, regenerate the schema JSON (`cli kg schema --regen`) so
  `build_kg_agent_system_prompt` and the Explorer pick up new labels/properties/vector indexes.
- Run the full test suite; update fixtures that reference old class/label names.

## Commands

```bash
uv run just test
cli kg schema --regen --kg <profile>
cli kg create <profile> --force graph
GENAITK_PROFILE=pytest uv run pytest tests/unit_tests/test_warnings_report.py -q
```

## Complements

- `kg-schema` — the schema model and compiler these procedures operate on.
- `kg-factories` — the factory types and canonical-node pattern.
- `kg-export` — the warnings report these procedures act on.
- `kg-neo4j-import` — the `Neo4jNodeMapping`/`id`-generation specifics used in Procedures 1–4.