Enrich Stagingdb

---
name: enrich-stagingdb
description: Manage, enrich, consolidate, and validate Onto2AI schema content in Neo4j stagingdb. Use for ontology staging, duplicate consolidation, materialization, artifact regeneration, constraints, and domain-package smoke workflows.
---
# Staging Database Expert Instructions

You are a master of the Staging Database (typically `stagingdb`). Use this skill to move data from the primary ontology to staging, enrich classes with properties, create custom classes, manage named individuals, and clean up/flatten the staging schema for production use.

## Operating Boundary
- Use `stagingdb` for schema/model work: staged ontology subsets, class enrichment, enumeration discovery, generated schema artifacts, and validation before packaging.
- Use dataset databases such as `testdb` for sample data or runtime-style smoke tests. Dataset databases must not contain ontology schema nodes such as `owl__Class`, `owl__Ontology`, `owl__Restriction`, or ontology-only relationships such as `rdf__type` and `rdfs__subClassOf` unless the task explicitly requires schema validation there.
- Treat RDF ontology files as the source of truth for finalized ontology meaning. Staging edits are review/prototyping work until reflected back into the RDF and package artifacts.
- Package finalized domain artifacts independently from the core toolset. Examples include `onto2ai_entitlement/` and `onto2ai_parcel/`.

## URI Convention For Local Staging Additions
When creating local classes, relationships, or datatypes that do not come from FIBO or another source ontology, use the Onto2AI URI convention unless the user gives a domain-specific URI:

```text
http://www.onto2ai-toolset.com/ontology/<domain>/<OntologyName>/<Fragment>
```

Use source ontology URIs for copied FIBO/LCC/standards concepts and Onto2AI URIs only for local target ontology additions.

## Core Operations

### Staging Data
Use `staging_materialized_schema` to copy classes and relationships to the staging database.
- **Goal**: Create a self-contained subset of the ontology.
- **Options**: Set `flatten_inheritance=True` if you want to copy ancestor relationships directly to the child classes during extraction.

### Enriching Classes from FIBO Ontology
Use `get_materialized_schema` to discover available properties for a class, then write them to staging with Cypher.

**Workflow:**
1. Query FIBO ontology: `get_materialized_schema(class_names=["person"])` to see all available relationships
2. Check what already exists in staging: `MATCH (c:owl__Class {rdfs__label: 'person'})-[r]->(t) RETURN type(r), t.rdfs__label`
3. Write enrichment via idempotent Cypher: `MERGE` target nodes by URI and `MERGE` relationships by stable type, endpoints, and URI.

**Example — Enriching Person:**
```cypher
MATCH (person:owl__Class {rdfs__label: 'person'})
SET person.skos__definition = 'individual human being, with consciousness of self',
    person.uri = 'https://spec.edmcouncil.org/fibo/ontology/FND/AgentsAndPeople/People/Person'

MERGE (personName:owl__Class {uri: 'https://spec.edmcouncil.org/fibo/ontology/FND/AgentsAndPeople/People/PersonName'})
  ON CREATE SET personName.rdfs__label = 'person name',
               personName.skos__definition = 'designation by which someone is known in some context'

MERGE (person)-[hasName:hasName {uri: 'https://www.omg.org/spec/Commons/Designators/hasName'}]->(personName)
SET hasName.materialized = true,
    hasName.skos__definition = 'is known by',
    hasName.cardinality = '0..*'
```

**Key rules for enrichment relationships:**
- Always set `materialized: true` on relationship properties
- Include `uri`, `skos__definition`, and `cardinality` on each relationship
- Use `MERGE` for nodes and relationships so repeated enrichment does not create duplicates.
- Standard cardinality values: `1`, `0..1`, `0..*`, `1..*`

### Creating Custom Classes
When FIBO or the source ontology does not have a class you need, create it in staging with the target ontology URI namespace.

**Example — Creating Tax Payer:**
```cypher
MERGE (tp:owl__Class {uri: 'http://www.onto2ai-toolset.com/ontology/tax/Tax/TaxPayer'})
SET tp.rdfs__label = 'tax payer',
    tp.skos__definition = 'A person who is obligated to pay taxes and is identified by a tax identifier.'

// Inheritance
WITH tp
MATCH (person:owl__Class {rdfs__label: 'person'})
MERGE (tp)-[isPerson:rdfs__subClassOf]->(person)
SET isPerson.materialized = true,
    isPerson.skos__definition = 'A tax payer is a person.'

// Associations
WITH tp
MATCH (taxId:owl__Class {rdfs__label: 'tax identifier'})
MERGE (tp)-[hasTaxId:hasTaxId {uri: 'http://www.onto2ai-toolset.com/ontology/tax/Tax/hasTaxId'}]->(taxId)
SET hasTaxId.materialized = true,
    hasTaxId.skos__definition = 'The tax identifier assigned to a tax payer.',
    hasTaxId.cardinality = '1..*'
```

**Rules for custom classes:**
- Use the target ontology URI namespace, normally `http://www.onto2ai-toolset.com/ontology/<domain>/<OntologyName>/`
- Use lowercase with spaces for `rdfs__label` (e.g., `'tax payer'`)
- Use PascalCase for the URI fragment (e.g., `TaxPayer`)
- Always include `skos__definition`
- Use `rdfs__subClassOf` for inheritance relationships

### Materializing Datatype Properties
For simple value-type properties, create `rdfs__Datatype` nodes directly instead of full classes.

**Example — Enriching Conventional Street Address as US Physical Address:**
```cypher
MATCH (addr:owl__Class {rdfs__label: 'conventional street address'})

MERGE (sa:rdfs__Datatype {uri: '...'})
  ON CREATE SET sa.rdfs__label = 'streetAddress', sa.xsd__type = 'xsd:string',
               sa.skos__definition = 'primary address number, street name, suffix'

MERGE (zip:rdfs__Datatype {uri: '...'})
  ON CREATE SET zip.rdfs__label = 'zipCode', zip.xsd__type = 'xsd:string',
               zip.skos__definition = 'US postal ZIP code'

MERGE (city:rdfs__Datatype {uri: '...'})
  ON CREATE SET city.rdfs__label = 'city', city.xsd__type = 'xsd:string'

MERGE (state:rdfs__Datatype {uri: '...'})
  ON CREATE SET state.rdfs__label = 'state', state.xsd__type = 'xsd:string'

MERGE (addr)-[street:hasStreetAddress]->(sa)
SET street.materialized = true, street.cardinality = '1'
MERGE (addr)-[postal:hasZipCode]->(zip)
SET postal.materialized = true, postal.cardinality = '1'
MERGE (addr)-[locality:hasCity]->(city)
SET locality.materialized = true, locality.cardinality = '1'
MERGE (addr)-[region:hasState]->(state)
SET region.materialized = true, region.cardinality = '1'
```

**Common XSD types:**
- `xsd:string` — names, codes, identifiers, addresses
- `xsd:date` — dates (dateOfBirth, dateOfDeath)
- `xsd:integer` — whole numbers (age)
- `xsd:decimal` — monetary amounts
- `xsd:boolean` — true/false flags

**Primitive XSD staging rule:**
- If a staging class resource uses a primitive XSD URI such as `http://www.w3.org/2001/XMLSchema#string`, `#integer`, `#boolean`, `#date`, or similar, normalize that node as `:rdfs__Datatype`.
- Keep the node URI equal to the XSD URI.
- Set `rdfs__label` to the primitive local name only, such as `string`, `integer`, `boolean`, or `date`.
- Do not leave primitive XSD targets as bare `:Resource` nodes.
- Apply this normalization consistently so primitive XSD targets behave the same way as other staged datatype nodes.

### Creating Named Individuals
Create instances of classes using `owl__NamedIndividual` nodes with `rdf__type` links.

**Example — Creating United States of America:**
```cypher
MATCH (country:owl__Class {rdfs__label: 'country'})

MERGE (usa:owl__NamedIndividual {uri: 'https://www.omg.org/spec/LCC/Countries/ISO3166-1-CountryCodes/UnitedStatesOfAmerica'})
SET usa.rdfs__label = 'United States of America',
    usa.skos__definition = 'country in North America'
MERGE (usa)-[countryType:rdf__type]->(country)
SET countryType.materialized = true

// Link to a class that uses this individual
WITH usa
MATCH (addr:owl__Class {rdfs__label: 'conventional street address'})
MERGE (addr)-[defaultCountry:defaultCountry]->(usa)
SET defaultCountry.materialized = true, defaultCountry.cardinality = '1'
```

If you need instance data attributes (for example, ISO codes), model them as relationships to `rdfs__Datatype` nodes instead of inline properties.

**Rules for named individuals:**
- Label: `owl__NamedIndividual`
- Must have `rdf__type` relationship to its class
- Keep only metadata properties on the node (`rdfs__label`, `uri`, `skos__definition`)
- Model all domain data attributes as relationships to `rdfs__Datatype` nodes
- Use official URIs (e.g., FIBO/LCC) when available

### Consolidating Inheritance
If data is already in staging but still has parent-child links, use `consolidate_inheritance`.
- **Purpose**: Flatten the hierarchy so each class is fully descriptive on its own.
- **When**: Use this after staging classes if you didn't use `flatten_inheritance` during the initial copy.

### Structural Consolidation (Class → Datatype)
Use `consolidate_staging_db` to convert complex classes into simpler datatypes.

**Example — Converting date classes to datatypes:**
```python
consolidate_staging_db(transformations=[
    {"old_label": "date of birth", "new_label": "dateOfBirth", "xsd_type": "xsd:date"},
    {"old_label": "date of death", "new_label": "dateOfDeath", "xsd_type": "xsd:date"},
    {"old_label": "person name",   "new_label": "personName",   "xsd_type": "xsd:string"},
    {"old_label": "age",           "new_label": "age",           "xsd_type": "xsd:integer"}
])
```

**When to consolidate:**
- Class acts purely as a value container (no outgoing relationships of its own)
- Class represents a simple scalar type (date, string, number)
- You want to simplify the UML diagram by reducing class boxes

### Enriching Location Classes
For location-type classes (place of birth, headquarters, etc.), use a mix of datatypes and class references.

**Example — Enriching Place of Birth:**
```cypher
MATCH (pob:owl__Class {rdfs__label: 'place of birth'})

MERGE (city:rdfs__Datatype {rdfs__label: 'city', xsd__type: 'xsd:string'})
MERGE (state:rdfs__Datatype {rdfs__label: 'stateOrProvince', xsd__type: 'xsd:string'})

MATCH (country:owl__Class {rdfs__label: 'country'})

MERGE (pob)-[cityRel:hasCity]->(city)
SET cityRel.materialized = true, cityRel.cardinality = '0..1'
MERGE (pob)-[stateRel:hasStateOrProvince]->(state)
SET stateRel.materialized = true, stateRel.cardinality = '0..1'
MERGE (pob)-[countryRel:hasCountry]->(country)
SET countryRel.materialized = true, countryRel.cardinality = '1'
```

**Pattern**: Use datatypes for simple text fields (city, state names) and class references for complex objects (country with its own properties).

### Metadata Enrichment (AI-Driven)
For nodes (Classes) and relationships in `stagingdb` that are missing semantic documentation, use the LLM to generate `skos__definition` based on the context.

**Class Enrichment Workflow:**
1. Identify missing class definitions: `MATCH (n:owl__Class) WHERE n.skos__definition IS NULL RETURN n.rdfs__label, n.uri`
2. Generate with AI: Provide the class label and URI to the LLM.
3. Update Staging: `MATCH (n:owl__Class {uri: $uri}) SET n.skos__definition = $definition`

**Relationship Enrichment Workflow:**
1. Identify missing rel definitions: `MATCH (n:owl__Class)-[r]->(m) WHERE r.skos__definition IS NULL RETURN n.rdfs__label, type(r), m.rdfs__label, r.uri`
2. Generate with AI: Provide the relationship URI and the source/target labels to the LLM.
3. Update Staging: Set the generated definition on the relationship property in `stagingdb`.

### Full Schema Documentation
Maintain a textual representation of the entire graph schema for easy reference and LLM context.

**Tool:** `generate_neo4j_schema_description(database='stagingdb')`
**Purpose**: Generates a structured Markdown/text description:
1. **Node Labels**: URI and semantic definition. Subclass nodes are shown with **multi-label notation** (e.g., `TaxPayer:Person`, `Form1040_2025:IndividualTaxReturn`).
2. **Relationship Types**: URI, definition, source/target, and cardinality. `rdfs__subClassOf` is **excluded** — inheritance is encoded in multi-label notation instead.
3. **Node Properties**: Deduplicated by `(label, property, type, mandatory)`. Subclass label column uses multi-label notation. Includes **Data Type** and **Mandatory** status.
4. **Graph Topology**: Node patterns use multi-label notation. `rdfs__subClassOf` edges are omitted.
5. **Enumeration Members**: Explicit table of `owl__NamedIndividual` members grouped by class.

**Enum Visibility Standard**:
- Ensure `owl__NamedIndividual` members and `rdf__type` links are represented in schema artifacts.
- Ensure the schema description includes an explicit enumeration members section for review.

### Data Schema Constraints (Archival)
To ensure data integrity, maintain a Cypher constraints file for the finalized domain deliverable (for example, `onto2ai_entitlement/staging/neo4j_constraint.cypher` or `onto2ai_parcel/staging/neo4j_constraint.cypher`) that defines the physical constraints of the Neo4j database.

**Core Principles:**
1. **Separate Metadata**: Metadata properties like `uri`, `skos__definition`, and `rdfs__label` should NOT have constraints or persistent indexes in the archival script (keep them as comments only).
2. **Enforce Structural Schema**: Mandatory properties (cardinality starting with `1`) MUST have existence constraints (`IS NOT NULL`), and ontology properties marked unique MUST have `IS UNIQUE` constraints.
3. **Keep in Sync**: Generate or update the constraints file from current graph metadata as part of your release workflow (scripted or manual), and verify it against `stagingdb` before applying.
4. **Enum-Aware Notes**: Keep mandatory enum/class relationships documented as comments in the generated constraints output (while reserving physical `IS NOT NULL` constraints for datatype-backed node properties).

### Regeneration Workflow (After Enum or Relationship Updates)
After changing enum classes, named individuals, subclass relationships, or mandatory relationships, regenerate in this order. Prefer a checked-in package generator when one exists; it avoids manual copy drift.

For the entitlement package, run from the repository root:

```bash
venv/bin/python scripts/regenerate_entitlement_artifacts.py --database stagingdb
```

For domains without a package generator, call the MCP tools in this order and explicitly serialize each returned object or string to the reviewed package path; these tools do not write files themselves:
1. `extract_data_model(database='stagingdb')` → serialize `DataModel.model_dump_json(indent=2)` to the model JSON path.
   - Note: `rdfs__subClassOf` relationships are automatically included in the extracted model.
2. `generate_schema_code(target_type='pydantic', database='stagingdb')` → write the returned code string to the application-model path.
   - Child classes inherit from their parent Pydantic class; inherited fields are not redeclared.
   - Pydantic is one supported application model target. Keep schema decisions generic enough for other target generators.
3. `generate_neo4j_schema_description(database='stagingdb')` → write the returned Markdown string to the query-context path.
   - Subclass nodes appear as `Child:Parent` multi-label in all five sections.
4. `generate_neo4j_schema_constraint(database='stagingdb')` → write the returned Cypher string to the constraint path.
5. Write or copy finalized release artifacts into the relevant domain package staging folder only after reviewing the transient output.
6. Run the domain workflow validation test, for example:
   - `python -m onto2ai_entitlement.staging.schema_to_data_flow_smoke_test`
   - or the matching smoke test for the active domain package.
   - Use a dataset-oriented database. The entitlement smoke test creates a unique `entitlement-smoke-*` database by default and does not drop an existing database unless `--reset-database` is explicit.
   - Use `--cleanup` for release verification; omit it only when retained sample data is needed for manual review.
   - Review the printed summary before considering finalization complete
7. Build from the canonical domain package directory, inspect the wheel/source archive, and run the smoke workflow from an isolated wheel installation before publishing.
8. Publish the ontology package only after the smoke test passes.
9. Ensure workflow semantics are covered by test data. For entitlement, validate user-to-policy-group membership, policy-group inclusion, rule-to-column targets, and the database/schema/table/column containment chain.

### Domain Model Consistency
To ensure generated application code models are fully compatible with the graph, follow these modeling standards. Pydantic is one supported target; the underlying contract should also support future application code model targets.

**Key Patterns:**
1. **Ontology Identity**: Preserve class and property URIs in `full_schema_model.json` and query context. Do not assume generated classes inherit from a shared semantic base unless that base is implemented by the selected target generator.
2. **Field Aliases**: For Pydantic output, use `Field(alias="...")` to map Python field names to their ontological property or relationship names.
   - Example: `taxableIncome: Optional[MonetaryAmount] = Field(alias="hasTaxableIncome", ...)`
3. **Persistence Adapter**: Use an implemented domain loader or write an explicit adapter that separates scalar node properties from object relationships. Do not reference a bridge utility unless it exists in the repository and is covered by tests.

**Why**: This 1:1 parity between the domain model and the graph schema enables type-safe, automated data ingestion and extraction without manual Cypher mapping.

### Strict Schema Description Parity
`generate_neo4j_schema_description` currently ignores its legacy `use_heuristics` parameter and formats the complete result of `extract_data_model`. Do not filter classes by instance counts or leaf status in package artifacts. If a curated view is needed, generate it as a separate review artifact and keep the full schema description authoritative.

### ⚠️ CRITICAL: No Inline Properties on Named Individuals
**NEVER store data attributes as inline properties on `owl__NamedIndividual` nodes.** All attributes must be modeled as relationships to `rdfs__Datatype` nodes.

❌ **WRONG** — inline properties:
```cypher
CREATE (w2:owl__NamedIndividual {
  rdfs__label: 'W-2',
  box1_wages: 'decimal',      // WRONG: inline property
  box2_taxWithheld: 'decimal'  // WRONG: inline property
})
```

✅ **CORRECT** — relationships to datatypes:
```cypher
MERGE (w2:owl__NamedIndividual {uri: '...'})
SET w2.rdfs__label = 'W-2',
    w2.skos__definition = '...'

MERGE (wages:rdfs__Datatype {rdfs__label: 'wagesTipsOtherComp'})
  ON CREATE SET wages.xsd__type = 'xsd:decimal',
               wages.skos__definition = 'Box 1: Total wages, tips, and other compensation'

MERGE (w2)-[wagesRel:hasWagesTipsOtherComp]->(wages)
SET wagesRel.materialized = true, wagesRel.cardinality = '1'
```

**Why**: Named individuals should only have metadata properties (`rdfs__label`, `uri`, `skos__definition`). All domain attributes must be expressed as graph relationships so they appear correctly in UML/Pydantic visualizations and can be properly queried.

### ⚠️ CRITICAL: Promote Schema Types to Classes, Not Named Individuals
**When a concept represents a type/template (e.g., a form type, document type), model it as `owl__Class` with `rdfs__subClassOf`, NOT as `owl__NamedIndividual` with `rdf__type`.**

Use `owl__NamedIndividual` ONLY for true singleton instances (e.g., "United States of America", "US Dollar").

❌ **WRONG** — form type as named individual:
```cypher
CREATE (f:owl__NamedIndividual {rdfs__label: 'Form 1040'})
CREATE (f)-[:rdf__type]->(report)  // WRONG: rdf__type implies instance
```

✅ **CORRECT** — form type as class:
```cypher
MERGE (f:owl__Class {uri: '...'})
SET f.rdfs__label = 'Form 1040', f.skos__definition = '...'
MERGE (f)-[formType:rdfs__subClassOf]->(report)
SET formType.materialized = true  // Subclass hierarchy
```

**To convert existing named individuals to classes:**
```cypher
MATCH (f:owl__NamedIndividual {rdfs__label: 'Form 1040'})
REMOVE f:owl__NamedIndividual
SET f:owl__Class
WITH f
MATCH (f)-[oldType:rdf__type]->(c)
DELETE oldType
WITH DISTINCT f
MATCH (parent:owl__Class {rdfs__label: 'report'})
MERGE (f)-[formType:rdfs__subClassOf]->(parent)
SET formType.materialized = true
```

**When to use which:**
| Concept | Node Type | Relationship |
|---|---|---|
| W-2 form, Form 1040, Form 1120 | `owl__Class` | `rdfs__subClassOf → report` |
| United States of America | `owl__NamedIndividual` | `rdf__type → country` |
| US Dollar, Euro | `owl__NamedIndividual` | `rdf__type → currency` |

## Best Practices
- **Isolation**: Always work on `stagingdb` to avoid polluting the main ontology.
- **Consistency**: Use camelCase for relationship types and lowercase with spaces for class labels.
- **Integrity**: Verify results after each enrichment: `MATCH (c {rdfs__label: '...'})-[r]->(t) RETURN type(r), t.rdfs__label`
- **Reuse nodes**: Always `MERGE` target nodes by URI to avoid duplicates (e.g., `country`, `city` datatypes).
- **FIBO first**: Check the FIBO ontology for existing definitions before creating custom classes.
- **Deduplication**: The `consolidate_staging_db` tool automatically de-duplicates URIs, labels, and relationships.
- **Consolidation cleanup**: `consolidate_staging_db` automatically deletes named individuals linked via `rdf__type` when converting a class to a datatype.
- **RDF sync**: Before release, reflect accepted staging semantics back into the RDF source and validate with `xmllint`.