Regulatory Templates Gate1

---
name: ring:regulatory-templates-gate1
description: |
  Gate 1 sub-skill - performs regulatory compliance analysis and field mapping
  from template specifications.

trigger: |
  - regulatory-templates-setup completed
  - Need to analyze regulatory specification and map fields

skip_when: |
  - Setup not complete → run setup first
  - Gate 1 already passed → proceed to Gate 2

sequence:
  after: [regulatory-templates-setup]
  before: [regulatory-templates-gate2]
---

# Regulatory Templates - Gate 1: Placeholder Mapping (Post Gate 0)

## Overview

**UPDATED: Gate 1 now maps placeholders from Gate 0 template to data sources. NO structure creation, NO logic addition.**

**Parent skill:** `regulatory-templates`

**Prerequisites:**
- Context from `regulatory-templates-setup`
- Template base from `regulatory-templates-setup`

**Output:** Mapping of placeholders to backend data sources

---

## Foundational Principle

**Field mapping errors compound through Gates 2-3 and into production.**

Gate 1 is the foundation of regulatory template accuracy:
- **snake_case conversion**: Python/Django ecosystem standard (PEP 8) - mixed conventions cause maintenance nightmares
- **Data source prefixes**: BACEN audits require data lineage traceability - "where did this value come from?"
- **Interactive validation**: No dictionary = no ground truth - user approval prevents assumption errors
- **Confidence thresholds**: Quality gates prevent low-confidence mappings from reaching production
- **Dictionary checks**: Consistency across team, audit trail for regulatory reviews

**Every "shortcut" in Gate 1 multiplies through downstream gates:**
- Skip snake_case → Gate 3 templates have mixed conventions → maintenance debt
- Skip prefixes → Gate 2 cannot trace data sources → debugging nightmares
- Auto-approve mappings → Gate 2 validates wrong assumptions → compliance violations
- Skip optional fields → Gate 1 fails confidence threshold → rework loops
- Lower thresholds → Low-confidence fields reach Gate 3 → production errors

**Technical correctness in Gate 1 = foundation for compliance in production.**

---

## When to Use

**Called by:** `regulatory-templates` skill after Gate 0 template structure copy

**Purpose:** Map each placeholder to its data source - structure already defined in Gate 0

---

## NO EXCEPTIONS - Technical Requirements Are Mandatory

**Gate 1 field mapping requirements have ZERO exceptions.** Every requirement exists to prevent specific failure modes.

### Common Pressures You Must Resist

| Pressure | Your Thought | Reality |
|----------|--------------|---------|
| **Speed** | "camelCase works, skip conversion" | PEP 8 violation creates maintenance debt. 30 min now vs 75+ min debugging later |
| **Simplicity** | "Prefix is verbose, omit it" | BACEN audits require data lineage. Implicit resolution = debugging nightmares |
| **Efficiency** | "AUTO-approve obvious mappings" | No dictionary = no ground truth. "Obvious" assumptions cause compliance violations |
| **Pragmatism** | "Skip optional fields" | Confidence calculated across ALL fields. 64% coverage = FAIL |
| **Authority** | "75% confidence is enough" | Threshold erosion: 75% → 70% → 60%. LOW confidence fields = high-risk mappings |
| **Experience** | "I memorized these, skip dictionary" | Memory is fallible. 1-min check prevents 20-40 min error correction |

### Technical Requirements (Non-Negotiable)

**snake_case Conversion:**
- ✅ REQUIRED: Convert ALL field names to snake_case
- ❌ FORBIDDEN: Use camelCase, PascalCase, or mixed conventions
- Why: Python/Django PEP 8 standard, grep-able patterns, maintenance

**Data Source Prefixes:**
- ✅ REQUIRED: `{{ midaz_onboarding.organization.0.legal_document }}`
- ❌ FORBIDDEN: `{{ organization.legal_document }}`
- Why: Data lineage traceability, multi-source disambiguation, audit compliance

**Interactive Validation:**
- ✅ REQUIRED: AskUserQuestion for EACH field mapping
- ❌ FORBIDDEN: Auto-approve HIGH confidence fields
- Why: No dictionary = no ground truth, user provides domain knowledge

**Confidence Threshold:**
- ✅ REQUIRED: Overall confidence ≥ 80%
- ❌ FORBIDDEN: Lower threshold or skip fields
- Why: Quality gate for Gate 2/3, prevents low-confidence mappings in production

**Dictionary Check:**
- ✅ REQUIRED: Check `~/.claude/docs/regulatory/dictionaries/` first
- ❌ FORBIDDEN: Skip check and use memory
- Why: Consistency, audit trail, error prevention

### The Bottom Line

**Shortcuts in field mapping = errors in production regulatory submissions.**

Gate 1 creates the foundation for Gates 2-3. Technical correctness here prevents compliance violations downstream.

**If you're tempted to skip ANY requirement, ask yourself: Am I willing to debug production BACEN submission failures caused by this shortcut?**

---

## Rationalization Table - Know the Excuses

Every rationalization below has been used to justify skipping requirements. **ALL are invalid.**

| Excuse | Why It's Wrong | Correct Response |
|--------|---------------|------------------|
| "camelCase works fine in Django" | PEP 8 violation, maintenance debt, inconsistent conventions | Convert ALL to snake_case |
| "Prefix is verbose and ugly" | Audit trail required, multi-source disambiguation critical | Prefix ALL fields |
| "HIGH confidence = obvious, no approval needed" | No dictionary = no ground truth, assumptions fail | Ask approval for EACH field |
| "Optional fields don't affect compliance" | Confidence calculated across ALL fields, 64% = FAIL | Map ALL fields |
| "75% is close to 80%, good enough" | Threshold erosion, LOW confidence = high risk | Research to ≥80% |
| "I know these mappings by heart" | Memory fallible, experience creates overconfidence | Check dictionary first |
| "Everyone knows where organization comes from" | Implicit tribal knowledge, new team members lost | Explicit beats implicit |
| "User approval wastes their time" | User provides domain knowledge we lack | Interactive validation mandatory |
| "Conversion is unnecessary busywork" | Dismissing requirements without understanding cost | Technical correctness prevents debt |
| "This is simple, process is overkill" | Simple tasks accumulate into complex problems | Follow workflow completely |

### If You Find Yourself Making These Excuses

**STOP. You are rationalizing.**

The requirements exist to prevent these exact thoughts from causing errors. If a requirement seems "unnecessary," that's evidence it's working - preventing shortcuts that seem reasonable but create risk.

---

## CRITICAL CHANGE

### ❌ OLD Gate 1 (Over-engineering)
- Created complex field mappings
- Added transformation logic
- Built nested structures
- Result: 90+ line templates

### ✅ NEW Gate 1 (Simple)
- Takes template from Gate 0
- Maps placeholders to single data source
- NO structural changes
- Result: <20 line templates

### 🔴 CRITICAL: NAMING CONVENTION - SNAKE_CASE STANDARD
**ALL field names MUST be converted to snake_case:**
- ✅ If API returns `legalDocument` → convert to `legal_document`
- ✅ If API returns `taxId` → convert to `tax_id`
- ✅ If API returns `openingDate` → convert to `opening_date`
- ✅ If API returns `naturalPerson` → convert to `natural_person`
- ✅ If API returns `tax_id` → keep as `tax_id` (already snake_case)

**ALWAYS convert camelCase, PascalCase, or any other convention to snake_case.**

### 🔴 CRITICAL: DATA SOURCES - ALWAYS USE CORRECT DOMAIN PREFIX

**REFERENCE:** See `/docs/regulatory/DATA_SOURCES.md` for complete documentation.

**Available Data Sources (Reporter Platform):**

| Data Source | Descrição | Entidades Principais |
|-------------|-----------|---------------------|
| `midaz_onboarding` | Dados cadastrais | organization, account |
| `midaz_transaction` | Dados transacionais | operation_route, balance, operation |
| `midaz_onboarding_metadata` | Metadados cadastro | custom fields |
| `midaz_transaction_metadata` | Metadados transações | custom fields |

**Field Path Format:** `{data_source}.{entity}.{index?}.{field}`

**Examples:** `{{ midaz_onboarding.organization.0.legal_document }}` | `{{ midaz_transaction.operation_route.code }}` | `{{ midaz_transaction.balance.available }}`

**Common Mappings:** CNPJ→`organization.0.legal_document`, COSIF→`operation_route.code`, Saldo→`balance.available`

**RULE:** Always prefix with data source! ❌ `{{ organization.legal_document }}` → ✅ `{{ midaz_onboarding.organization.0.legal_document }}`

---

## Gate 1 Process

### STEP 1: Check for Data Dictionary (FROM/TO Mappings)

**HIERARCHICAL SEARCH - Dictionary first, Interactive Validation second:**

**Dictionary Path:** `~/.claude/docs/regulatory/dictionaries/{category}-{code}.yaml`

| Step | If Dictionary EXISTS | If Dictionary NOT EXISTS |
|------|---------------------|--------------------------|
| 1 | Load YAML, use field_mappings | Query MCP: `mcp__apidog_midaz/crm__read_project_oas()` |
| 2 | Apply transformations | Analyze schemas, SUGGEST mappings (preserve casing) |
| 3 | Use existing mappings | **AskUserQuestion** for EACH field (user approval required) |
| 4 | Return | Create dictionary with APPROVED mappings only |
| 5 | — | Save to dictionary path for future use |

**Dictionary contains:** field_mappings (FROM→TO), transformations, pitfalls, validation_rules

---

## 🔴 CRITICAL: INTERACTIVE VALIDATION FOR TEMPLATES WITHOUT DICTIONARY

### Data Dictionaries Location

**Dicionários de dados disponíveis em:** `~/.claude/docs/regulatory/dictionaries/`

Consulte os dicionários existentes antes de iniciar o mapeamento de campos.

---

### Interactive Validation Process (MANDATORY for templates without dictionary)

| Step | Action | Details |
|------|--------|---------|
| **A** | Discover Fields | Read regulatory spec (XSD/PDF) → Extract ALL required fields + types + formats |
| **B** | Query API Schemas | `mcp__apidog-midaz/crm__read_project_oas()` → Extract available fields from both systems |
| **C** | Interactive Validation | For EACH field: AskUserQuestion with top 3-4 suggestions + "Skip" + "Other" (custom path) |
| **D** | Validate Transformations | If field needs transform: AskUserQuestion with options (e.g., `slice:':8'`, "No transformation") |
| **E** | Generate Dictionary | Create YAML with APPROVED mappings only → Save to `DICTIONARY_BASE_PATH/[template].yaml`

---

### AskUserQuestion Implementation for Field Mapping

**CRITICAL: Use AskUserQuestion tool with these patterns:**

| Pattern | Question Format | Options |
|---------|----------------|---------|
| **Field Source** | `Map '${field.name}' (${type}, ${required})?` | Top 3 suggestions (with confidence %) + "Skip for now" + "Other" (auto) |
| **Transformation** | `Transformation for '${field.name}'?` | Suggested filters (with examples) + "No transformation" + "Other" (auto) |
| **Batch Approval** | `Approve mapping for '${name}'? Suggested: ${path}, Confidence: ${%}` | "Approve ✓" / "Reject ✗" (max 4 questions per batch) |

**Note:** "Other" option automatically added by AskUserQuestion for custom input.

---

### Complete Interactive Validation Flow

**Process:** Read spec → Query MCP schemas → For EACH field: AskUserQuestion → Process response → Track approved/skipped

| Response Type | Action |
|--------------|--------|
| "Skip" | Add to skippedFields (resolve in Gate 2) |
| Custom input ("Other") | Add with `approved_by: "user_custom_input"`, confidence: 100 |
| Suggested option | If needs transformation: ask for filter (slice, floatformat, etc.) → Add with `approved_by: "user_selection"` |

**Output:** `{ approvedMappings: [...], skippedFields: [...] }`

---

### Validation Rules for User Input

**Valid path patterns for custom input:**

| Source | Pattern | Example |
|--------|---------|---------|
| `midaz_onboarding` | `midaz_onboarding.(organization\|account).N.field` | `midaz_onboarding.organization.0.legal_document` |
| `midaz_transaction` | `midaz_transaction.(operation_route\|balance\|operation).field` | `midaz_transaction.balance.available` |
| `crm` | `crm.(holder\|alias).field` | `crm.holder.document` |
| `metadata` | `(midaz\|crm).entity.metadata.field` | `midaz.account.metadata.branch` |

**Validation:** If path doesn't match patterns → warn user but allow (may be valid custom path).

### NAMING CONVENTION IN FIELD DISCOVERY

**CRITICAL: ALWAYS CONVERT TO SNAKE_CASE!**

| API Returns | Map As | ✅/❌ |
|-------------|--------|------|
| `legalDocument` | `organization.legal_document` | ✅ |
| `taxId` / `TaxID` | `organization.tax_id` | ✅ |
| `openingDate` | `organization.opening_date` | ✅ |
| `legalDocument` | `organization.legalDocument` | ❌ NEVER |

**Search patterns help FIND fields. Once found, CONVERT TO SNAKE_CASE!**

### Hierarchical Search Strategy

**CRITICAL: Convert ALL discovered fields to snake_case!**

| Step | Action | Priority Paths |
|------|--------|----------------|
| **1** | Query MCP schemas | `mcp__apidog_crm/midaz__read_project_oas()` |
| **2** | Search CRM first | holder.document, holder.name, holder.type, holder.addresses.*, holder.contact.*, holder.naturalPerson.*, holder.legalPerson.*, alias.bankingDetails.*, alias.metadata.* |
| **3** | Search Midaz second | account.name, account.alias, account.metadata.*, account.status, transaction.metadata.*, balance.amount, organization.legalDocument |
| **4** | Check metadata | crm.holder/alias.metadata.*, midaz.account/transaction.metadata.* |
| **5** | Mark as uncertain | If not found → document searched locations + suggest closest matches + indicate confidence |

### Confidence Scoring System

| Level | Score | Criteria |
|-------|-------|----------|
| **HIGH** (90-100%) | Base(30) + Name(25) + System(25) + Type(20) + Validation(20) | Exact name match, type matches, primary system, validation passes, simple/no transform |
| **MEDIUM** (60-89%) | Base(30) + partial matches | Partial name or pattern match, compatible type needs transform, secondary system, some uncertainty |
| **LOW** (30-59%) | Base(30) only | Synonym/fuzzy match, significant transform, metadata only, cannot validate |

**Formula:** `Score = Base(30) + NameMatch(0-25) + SystemMatch(0-25) + TypeMatch(0-20) + ValidationMatch(0-20)`

| Component | Values |
|-----------|--------|
| NameMatch | exact=25, partial=15, pattern=5 |
| SystemMatch | primary=25, secondary=15, metadata=5 |
| TypeMatch | exact=20, compatible=10, needs_transform=5 |
| ValidationMatch | validated=20, partial=10, cannot_validate=0 |

### Validation with Examples

**Process:** Fetch sample → Apply transformation → Validate format

| Pattern | Regex |
|---------|-------|
| CPF | `/^\d{11}$/` |
| CNPJ | `/^\d{14}$/` |
| CNPJ_BASE | `/^\d{8}$/` |
| DATE_BR | `/^\d{2}\/\d{2}\/\d{4}$/` |
| DATE_ISO | `/^\d{4}-\d{2}-\d{2}$/` |
| PHONE_BR | `/^\+?55?\s?\(?\d{2}\)?\s?\d{4,5}-?\d{4}$/` |
| CEP | `/^\d{5}-?\d{3}$/` |

**Example:** CNPJ Base: `"12345678000190"` → `slice:':8'` → `"12345678"` → `/^\d{8}$/` → ✓ valid (+20 confidence)

### Agent Dispatch

**Dispatch:** `Task(subagent_type: "ring:finops-analyzer", model: "opus")`

**Pre-dispatch:** Check dictionary at `~/.claude/docs/regulatory/dictionaries/{category}-{code}.yaml`

| Mode | Condition | Instructions |
|------|-----------|--------------|
| **Dictionary Mode** | File exists | USE dictionary data ONLY. NO MCP calls. Validate mappings. |
| **MCP Discovery Mode** | File missing | Query MCP APIs → Suggest mappings → AskUserQuestion for EACH → Create dictionary with APPROVED only |

**Prompt includes:** Template info, dictionary status/content (if exists), snake_case requirement, validation steps, output format

**CRITICAL REQUIREMENTS:**

| ✅ DO | ❌ NEVER |
|-------|---------|
| Check dictionary FIRST | Skip dictionary check |
| MCP only if no dictionary | Call MCP when dictionary exists |
| AskUserQuestion for ALL mappings | Auto-approve without asking |
| Save APPROVED mappings only | Save unapproved guesses |
| Validate all transformations | Guess field mappings |

**Report Output:** dictionary_status, field_mappings (code, name, required, source, transformation, confidence, validated, examples), validation_summary (total, mapped, coverage%, avg_confidence)

**COMPLETION STATUS:** COMPLETE, INCOMPLETE, or NEEDS_DISCUSSION

---

## Capture Gate 1 Response

**Response structure:**

| Section | Fields |
|---------|--------|
| **Template Info** | template_name, regulatory_standard, authority, submission_frequency, submission_deadline |
| **Field Counts** | total_fields, mandatory_fields, optional_fields |
| **Discovery Summary** | crm_fields_available, midaz_fields_available, metadata_fields_used, unmapped_fields |
| **Field Mappings** (per field) | field_code, field_name, required, type, format, mappings_found[], selected_mapping, confidence_score, confidence_level, reasoning, transformation, validation_passed, status |
| **Uncertainties** (per field) | field_code, field_name, mappings_attempted[], best_match, doubt, suggested_resolution |
| **Confidence Summary** | high/medium/low_confidence_fields, overall_confidence |
| **Compliance Risk** | LOW/MEDIUM/HIGH (based on confidence levels) |
| **Documentation Used** | official_regulatory URL, implementation_reference URL, regulatory_framework |

---

## Documentation Sources

### Official Regulatory Sources (SOURCE OF TRUTH)

---

## Red Flags - STOP Immediately

If you catch yourself thinking ANY of these, STOP and re-read the NO EXCEPTIONS section:

### Skip Patterns
- "Skip snake_case conversion for..."
- "Omit prefix for obvious fields"
- "Use camelCase this time"
- "Mixed conventions are fine"
- "Dictionary check is ceremony"

### Partial Compliance
- "Convert only mandatory fields"
- "Prefix only ambiguous fields"
- "Auto-approve HIGH confidence"
- "Map only mandatory fields"
- "75% is close enough to 80%"

### Experience-Based Shortcuts
- "I memorized these mappings"
- "I know where this comes from"
- "We've done this 50 times"
- "The pattern is obvious"
- "Dictionary won't exist anyway"

### Justification Language
- "Unnecessary busywork"
- "Verbose and ugly"
- "Wasting user time"
- "Process over outcome"
- "Being pragmatic"
- "Close enough"
- "Everyone knows"

### If You See These Red Flags

1. **Acknowledge the rationalization** ("I'm trying to skip snake_case")
2. **Read the NO EXCEPTIONS section** (understand why it's required)
3. **Read the Rationalization Table** (see your exact excuse refuted)
4. **Follow the requirement completely** (no modifications)

**Technical requirements are not negotiable. Field mapping errors compound through Gates 2-3.**

---

## Pass/Fail Criteria

### PASS Criteria
- ✅ `COMPLETION STATUS: COMPLETE`
- ✅ 0 Critical gaps (unmapped mandatory fields)
- ✅ Overall confidence score ≥ 80%
- ✅ All mandatory fields mapped (even if LOW confidence)
- ✅ < 10% of fields with LOW confidence
- ✅ Dynamic discovery via MCP executed
- ✅ Documentation was consulted (both official and implementation)
- ✅ CRM checked first for banking/personal data

### FAIL Criteria
- ❌ `COMPLETION STATUS: INCOMPLETE`
- ❌ Critical gaps exist (mandatory fields unmapped)
- ❌ Overall confidence score < 60%
- ❌ > 20% fields with LOW confidence
- ❌ Documentation not consulted
- ❌ MCP discovery not performed
- ❌ Only checked one system (didn't check CRM + Midaz)

---

## State Tracking

| Status | Output Fields |
|--------|--------------|
| **PASS** | STATUS: PASSED, FIELDS: total/mandatory, UNCERTAINTIES: count, COMPLIANCE_RISK, NEXT: Gate 2, EVIDENCE: docs consulted + all mandatory mapped |
| **FAIL** | STATUS: FAILED, CRITICAL_GAPS: count, HIGH_UNCERTAINTIES: count, NEXT: Fix gaps, BLOCKERS: Critical mapping gaps |

---

## Critical Validations

Ensure these patterns are followed:
- Use EXACT patterns from Lerian documentation
- Apply filters like `slice`, `floatformat` as shown in docs
- Follow tipoRemessa rules: "I" for new/rejected, "S" for approved only
- Date formats must match regulatory requirements (YYYY/MM, YYYY-MM-DD)
- CNPJ/CPF formatting rules must be exact

---

## Output to Parent Skill

**Return to `regulatory-templates`:** `{ gate1_passed: bool, gate1_context: {...}, uncertainties_count: N, critical_gaps: [], next_action: "proceed_to_gate2" | "fix_gaps_and_retry" }`

---

## Common Issues and Solutions

| Issue | Solution |
|-------|----------|
| Documentation not accessible | Try alternative URLs or cached versions |
| Field names don't match Midaz | Mark as uncertain for Gate 2 validation |
| Missing mandatory fields | Mark as Critical gap, must resolve |
| Format specifications unclear | Consult both Lerian docs and government specs |

---

## Dynamic Discovery Example

**Finding "Agência" field for CADOC 4010:**

| Step | Action | Result |
|------|--------|--------|
| 1 | Pattern search | `["branch", "agency", "agencia", "branch_code"]` |
| 2 | Query CRM first | `crm.alias.bankingDetails.branch` ✓ (exact, 95%) |
| 3 | Query Midaz fallback | `midaz.account.metadata.branch_code` ⚠ (metadata, 45%) |
| 4 | Select highest | `crm.alias.bankingDetails.branch` (HIGH confidence) |

## Remember

1. **CONVERT TO SNAKE_CASE** - All fields must be snake_case (legal_document not legalDocument)
2. **Use MCP for dynamic discovery** - Never hardcode field paths
3. **CRM first for banking/personal data** - It has the most complete holder info
4. **Official specs are SOURCE OF TRUTH** - Regulatory requirements from government
5. **Lerian docs show IMPLEMENTATION** - How to create templates in their system
6. **Template-specific knowledge is valuable** - Always check for existing sub-skills
7. **Confidence scoring is key** - Always calculate and document confidence
8. **Be conservative with mappings** - Mark uncertain rather than guess
9. **Capture everything** - Gate 2 needs complete context with all attempted mappings
10. **Reference both sources** - Note official specs AND implementation examples
11. **Risk assessment based on confidence** - Low confidence = higher compliance risk

## Important Distinction

⚠️ **Regulatory Compliance vs Implementation**
- **WHAT** (Requirements) = Official government documentation
- **HOW** (Implementation) = Lerian documentation examples
- When validating compliance → Use official specs
- When creating templates → Use Lerian patterns
- Never confuse implementation examples with regulatory requirements