Monitor Sparta

---
name: monitor-sparta
description: >
  Continuous SPARTA quality monitor. Runs the 3-tier validation cascade
  alongside Stage 12 QRA generation — Tier 0 deterministic, Tier 1.5 trained GPT
  (via /assistant), Tier 2 Brandon (edge cases). Accumulates labels, auto-triggers
  /create-gpt training at 2000 labels, and tracks convergence.
allowed-tools:
  - Bash
  - Read
triggers:
  - monitor sparta
  - sparta quality
  - pipeline health
  - sparta convergence
  - qra quality monitor
metadata:
  short-description: Continuous 3-tier SPARTA quality monitor with auto-training

provides:
  - monitor-sparta
composes:
  - create-figure
  - task-monitor
  - agentic-evals
disciplines:
  - observability-operations
  - compliance-security
---

# Monitor SPARTA

**Continuous quality monitor for the SPARTA QRA pipeline.** Runs alongside Stage 12
and validates at configurable checkpoints (default: every 10K QRAs).

## Full Coverage Definition for SPARTA QRAs

`/monitor-sparta` must report SPARTA QRA coverage by semantic bucket, not as a
single LLM-call count. Full coverage means:

1. Canonical SPARTA QRAs exist for every in-scope/actionable SPARTA technique,
   tactic, and countermeasure.
2. Technique-context relationship QRAs exist for valid tactic→technique and
   technique→related-control prompts.
3. Technique-related control-to-control QRAs exist only for pairs that pass
   `/create-evidence-case` evidence-case adjudication inside technique `T`; raw related
   control pairs are diagnostic candidates, not coverage obligations.
4. Non-SPARTA native QRAs exist for every source framework present in
   `sparta_controls`, enumerated FROM the corpus rather than from a list written
   here. A literal list in prose goes stale silently: this item previously named
   seven frameworks (ATT&CK, CWE, NIST, CAPEC, D3FEND, ESA, NVD) while the
   corpus held twenty, so `MITRE_ATLAS`, `CISA_KEV`, `NVD_CVE`, `EMB3D`, `ISO`
   and `NASA` — 96.5% of all controls — were outside the coverage definition and
   therefore unmonitored. Derive the framework set with a `COLLECT` over
   `source_framework`; treat any framework absent from the monitor's own
   reporting as a finding, not as out of scope.

Health reporting must distinguish:

- `raw_candidates`: diagnostic universe, never equivalent to missing QRAs
- `gated_runnable`: evidence-gated jobs that should be run
- `stored_qras`: schema-valid generated artifacts
- `deterministic_skips`: valid no-call or no-store outcomes such as
  `no_grounded_relationship`
- `failures`: transport/schema/runtime defects

A monitor status that reports only "remaining calls" is incomplete unless it
identifies which bucket those calls belong to and whether control-to-control jobs
were `/create-evidence-case` gated.


## Architecture

```
Stage 12 (QRA generation, running)
    │
    ├── QRA count: 75K → 80K → 90K → 100K
    │
monitor-sparta (this skill)
    │
    ├── Poll QRA count every 60s
    ├── At checkpoint (every 10K):
    │   ├── Tier 0: Deterministic checks (free, instant)
    │   ├── Tier 1.5: GPT via /assistant (free, ~200ms)
    │   │   └── Low confidence → escalate to Tier 2
    │   ├── Tier 2: Brandon via /scillm (edge cases only)
    │   │   └── Labels accumulate toward training threshold
    │   ├── Training trigger: ~2000 labels → /create-gpt iterate
    │   ├── Repair plans: write review-required manifests for failed controls
    │   └── Convergence: issue count must decrease
    │
    └── When GPT promoted (>90% agreement):
        └── Brandon only called for edge cases
        └── Monitor runs essentially for free
```

## Purpose

**Runs continuously for multiple days** until ALL QRAs and data integrity
reach A+ quality AND Margaret/Jennifer can consistently ask Brandon SPARTA
questions with 95%+ accurate results. Stops automatically when:
- 0 upstream deficiencies across all 5 dimensions
- Brandon says ACCEPTABLE with 0 FAIL and 0 WARN on QRA impact
- Persona test: 95%+ A+ accuracy (Margaret/Jennifer → Brandon → verify vs spreadsheet)
- Or: human explicitly stops the monitor

## Sparta Explorer Coverage UX

`http://localhost:3002/#sparta-explorer/coverage` is the UX projection of this
skill. The page must not be a passive report. Each Coverage section is a lane
subagent under `/monitor-sparta`:

| Coverage section | Lane subagent | Safe automatic behavior | Human-gated behavior |
|------------------|---------------|--------------------------|----------------------|
| QRA Generation | `qra_generation` | observe backlog, write manifests, rerun audits, launch reviewed/runnable native + SPARTA v2 `/create-qras` backlog | launch or override blocked control-to-control comparison jobs without accepted `/create-evidence-case` responses |
| Prompt Health / Prompt Inventory | `prompt_health` | run `/best-practices-prompt`, classify prompt-unit failures, generate `/review-prompt` payloads | apply non-mechanical prompt rewrites or promote prompt changes |
| Monitor Health | `monitor_health` | run health scanners, publish heartbeat/state | change thresholds or suppress dimensions |
| UX Coverage | `ux_coverage` | scan data-qid/test-interaction coverage | edit React routes/components |
| Python Fallbacks | `python_fallbacks` | scan silent fallback patterns | remove or rewrite fallback behavior |
| Source/Text/QRA Coverage | `source_text_qra_coverage` | observe control/URL text and QRA gaps, write manifests | fetch URLs or generate QRAs |
| Source/Embedding Coverage | `source_embedding_coverage` | observe Arango/Qdrant coverage, write manifests | mutate Arango/Qdrant |

Trigger model:

- push-triggered: UI, CLI, Discord, Slack, and voice commands enqueue lane work
  immediately;
- scheduled: nightly or supervisor-managed runs catch drift when no human is
  present;
- browser: WebSocket/SSE pushes state to Sparta Explorer, with polling fallback.

Expensive health-check rule:

- Sparta Explorer may poll or subscribe continuously, but browser reads must
  return the last durable snapshot unless a cheap change signature says the
  SPARTA corpus/supervisor state changed.
- `run_all_checks()` and other full Arango health scans are allowed only for an
  explicit human/operator force refresh, a nightly/checkpoint run, or a detected
  change signature. They must not be launched by ordinary page refreshes.
- The Coverage API must expose whether a response is `change_gated`,
  `change_detected`, `force_requested`, or `snapshot_only` so agents can tell
  whether they are looking at a durable snapshot or an active audit.

Notification contract:

- `/monitor-sparta` health checks must emit operator-friendly notification copy
  next to raw machine fields. UI surfaces must not invent coverage-gap wording
  from dimensions, counts, or generic gate states.
- Each failed check should keep `dimension`, `message`, and structured detail
  fields for diagnostics, and may add:

```json
{
  "notification": {
    "title": "Invalid QRA reasoning",
    "body": "4,703 QRAs have invalid reasoning",
    "severity": "degraded",
    "actionLabel": "Open QRA reasoning gaps",
    "target": "coverage:qra_reasoning_coverage"
  }
}
```

- `notification.body` is the preferred high-density billboard text. The UI may
  fall back to `message` only when `notification` is absent.

Prompt Health lane rule:

1. Run the local prompt-health scanner on prompt units, not isolated user
   fragments or README files.
2. Apply `/best-practices-prompt` first.
3. Run `/review-prompt` for non-mechanical rewrites, low-confidence edits, or
   prompts that affect generation semantics.
4. Auto-apply only mechanical edits that pass the scanner and prompt canary.
5. Escalate any semantic rewrite, schema change, or production-promotion
   decision to the human with exact files, failure codes, and resume command.

### Persona-Driven Self-Improvement Loop

The key differentiator: **Margaret Chen and Jennifer Cheung ARE the test harness.**
They ask Brandon Bailey real F-36 LEO questions. When Brandon can't answer correctly,
the failure traces back to a bad QRA. That QRA is quarantined and regenerated.
The loop runs until 95% A+ accuracy.

```
Margaret/Jennifer ask F-36 question
  → Brandon answers using QRAs from ArangoDB
    → Answer verified against SPARTA spreadsheet ground truth
      → A+: correct, grounded, complete
      → F: wrong, hallucinated, or missing
        → Trace to source QRA keys
              → Stage review-gated QRA repair candidates
            → Approved apply regenerates affected controls
              → Retest. Loop until 95% A+.
```

## Quick Start

```bash
cd ${HOME}/workspace/experiments/memory

# Start the continuous monitor (runs for days until A+)
python scripts/validation/monitor_sparta.py start \
    --run-id run-recovery-verify

# Custom intervals (checkpoint every 5K QRAs or 30 minutes)
python scripts/validation/monitor_sparta.py start \
    --run-id run-recovery-verify \
    --checkpoint-interval 5000 --time-interval 1800

# Force a checkpoint NOW (one-shot, no daemon)
python scripts/validation/monitor_sparta.py checkpoint \
    --run-id run-recovery-verify

# Check status
python scripts/validation/monitor_sparta.py status

# Gate a worker/supervisor receipt before treating an issue as closed
skills/monitor-sparta/run.sh receipt-gate /path/to/receipt.json --require-closure --json

# Gate closure with separate verifier and reviewer receipts
skills/monitor-sparta/run.sh closure-review verify \
  --closure-receipt /path/to/worker-or-supervisor-receipt.json \
  --output /path/to/verifier-receipt.json \
  --json
skills/monitor-sparta/run.sh closure-review review \
  --closure-receipt /path/to/worker-or-supervisor-receipt.json \
  --verifier-receipt /path/to/verifier-receipt.json \
  --output /path/to/reviewer-receipt.json \
  --json
skills/monitor-sparta/run.sh closure-bundle-gate \
  --closure-receipt /path/to/worker-or-supervisor-receipt.json \
  --verifier-receipt /path/to/verifier-receipt.json \
  --reviewer-receipt /path/to/reviewer-receipt.json \
  --json

# Show convergence history
python scripts/validation/monitor_sparta.py convergence

# Stop the monitor
python scripts/validation/monitor_sparta.py stop
```

## Commands

| Command | Description |
|---------|-------------|
| `start` | Start continuous monitor (foreground by default) |
| `checkpoint` | Force one checkpoint now (one-shot) |
| `persona-test` | Run persona test loop (Margaret/Jennifer → Brandon → verify → backfill) |
| `status` | Show monitor status, label count, GPT readiness |
| `convergence` | Show convergence history across checkpoints |
| `stop` | Gracefully stop the running monitor |
| `receipt-gate` | Validate that a worker or supervisor receipt does not claim closure without Tau handoff proof |
| `closure-review` | Produce separate verifier and reviewer receipts for a closure candidate |
| `closure-bundle-gate` | Validate closure receipt plus separate verifier and reviewer receipts before reconciliation |

## Three-Tier Cascade

| Tier | Source | Cost | When Used |
|------|--------|------|-----------|
| **0** | Deterministic checks | Free | Always — every checkpoint |
| **1.5** | Trained GPT via /assistant | Free | After training + deployment |
| **2** | Brandon via /scillm (DeepSeek-V3) | ~$0.12/1K | Edge cases + label accumulation |

### Lifecycle

1. **Pre-GPT** (~0-2000 labels): Tier 0 + Tier 2 Brandon at every checkpoint
2. **Training** (2000+ labels): `/create-gpt iterate` auto-triggered
3. **Shadow** (GPT deployed, <90% agreement): Tier 1.5 runs alongside Tier 2, logs disagreements
4. **Production** (>90% agreement): Tier 1.5 handles bulk, Brandon only for low-confidence escalations

## Training Pipeline

```
Brandon labels (25 per dim per checkpoint)
    │
    ├── Accumulate in STATE_DIR/*/bootstrap_labels.jsonl
    │   (currently: 735 labels)
    │
    ├── At 2000 labels:
    │   ├── Export to pipeline-validator.jsonl
    │   └── /create-gpt iterate --task sparta-pipeline-validator
    │
    └── Deploy trained model:
        ├── ./run.sh register --task sparta-pipeline-validator \
        │     --model-path <path> --type gpt --threshold 0.85
        ├── Shadow mode: always escalates to Tier 2, logs both
        └── Promote when agreement > 90%
```

## Convergence Tracking

The monitor tracks quality metrics across checkpoints:

```
  CP    QRAs   Defic       Verdict   Labels    GPT  Agree
  ---   -----  -----       -------   ------    ---  -----
    1  75,724  5,169    ACCEPTABLE      735      N     0%
    2  85,000  4,800    ACCEPTABLE    1,035      N     0%
    3  90,000  4,200    ACCEPTABLE    1,335      N     0%
    4  95,000  3,900    ACCEPTABLE    1,635      N     0%
    5 100,000  3,500    ACCEPTABLE    1,935      N     0%
    6 105,000  3,200    ACCEPTABLE    2,235      Y    72%
    7 110,000  2,800    ACCEPTABLE    2,535      Y    85%
    8 120,000  2,400    ACCEPTABLE    2,835      Y    92%  ← promote
```

**Rules:**
- Issue count MUST decrease cycle over cycle
- 3 consecutive regressions → STALLED alert, manual intervention
- NEVER lower thresholds to game the curve

### Agent Behavior Rule (Non-Negotiable)

- The agent MUST NOT stop and wait for the human to ask for status or remember to check
- If a cycle fails, diagnose the failure, attempt auto-repair, and continue
- Only escalate to the human if genuinely blocked after exhausting /dogpile research

**Anti-pattern**: Reporting status and waiting for the human to ask "what next?" is UNACCEPTABLE. The agent must proactively fix issues and continue the monitoring loop.

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `MONITOR_SPARTA_STATE_DIR` | `/mnt/storage12tb/.../monitor-sparta` | State directory |
| `TEACHER_MODEL` | `deepseek-ai/DeepSeek-V3` | Brandon's LLM |
| Checkpoint interval | 10,000 QRAs | `--checkpoint-interval` |
| Poll interval | 60s | `--poll-interval` |
| Training threshold | 2,000 labels | Hardcoded in script |
| Shadow agreement target | 90% | Hardcoded in script |
| Brandon samples per checkpoint | 25 per dim | Hardcoded in script |
| Brandon impact samples | 50 | Hardcoded in script |

## State Files

```
/mnt/storage12tb/media/agents/shared/monitor-sparta/
├── state.json           # Current monitor state
├── convergence.jsonl    # One entry per checkpoint
├── monitor.pid          # PID file for stop command
├── task_state.json      # For /task-monitor integration
├── training.log         # /create-gpt output when triggered
└── checkpoint_N.json    # Validation report per checkpoint
```

## Integration

| Skill | How It's Used |
|-------|---------------|
| `/sparta-pipeline-validator` | Tier 0 + Tier 2 validation |
| `/assistant` | Tier 1.5 GPT inference + shadow mode |
| `/create-gpt` | Train Tier 1.5 GPT when threshold reached |
| `/scillm` | All Brandon LLM calls |
| `/task-monitor` | Progress reporting |
| `/memory` | Store convergence findings as lessons |

## Relationship to Other Skills

| Skill | Role |
|-------|------|
| `/sparta-review` | One-shot quality assessment (used ad-hoc by humans) |
| `/monitor-sparta` | **Continuous autonomous monitoring** (this skill) |
| `regenerate_failed.py` | Regeneration action triggered by this monitor |
| `backfill_parent_context.py` | Upstream fix triggered by this monitor |

## Data Integrity (ArangoDB Only)

`/monitor-sparta` is ArangoDB-first. Do not use DuckDB in this workflow.

Health command coverage (`monitor-sparta health`):
1. Description audit
2. Upstream version
3. Source control parity (missing controls)
4. Control description coverage (missing/empty descriptions)
5. Relationship coverage
6. Taxonomy backfill coverage
7. Embedding gaps
8. Corpus completeness
9. Description preservation
10. Requirement quality
11. Formalizability coverage
12. QRA evidence coverage (missing/empty evidence_case + version)
13. QRA reasoning coverage
13b. QRA reasoning uniqueness
14. Crosswalk edge coverage
15. Crosswalk-chain schema discipline
16. Framework label alignment (check 30)
17. Framework label congruence (check 31)

### Framework label invariants

A framework is not ingested until it has BOTH nodes in `sparta_controls` and
edges in `sparta_relationships`. Nodes alone are retrievable by semantic and
BM25 search, which makes the gap invisible to every coverage count, while
contributing nothing to multi-hop traversal — the one retrieval mode that
requires anchored identifiers.

`check_framework_name_normalization` groups by `UPPER(source_framework)` over
`sparta_controls` alone, so it detects only pure case variants inside one
collection. `check_framework_label_alignment` adds three invariants derived
from the data, not from a maintained registry:

| Invariant | Catches |
|-----------|---------|
| `dangling_edge_labels` | a framework named on an edge that no control carries, e.g. edges labelled `nist`/`sparta`/`ATT&CK` |
| `case_collisions` | labels colliding under `UPPER()` across the union of BOTH collections |
| `orphan_frameworks` | a framework with nodes but no edge on either side |

Observed 2026-08-11: the normalization check returned ok while the corpus held
`NVD_CVE` (367,886) beside `NVD` (4,830), `ESA_Shield` beside `ESA`, `attack`
beside `ATT_CK_Enterprise`, and NIST's 121,957 edges labelled `nist` against
nodes labelled `NIST`. The alignment check reports 6 dangling edge labels, 2
case-colliding groups, and 15 frameworks with zero edges covering 378,696 of
382,700 controls. NIST is not missing edges — its edges exist and join nothing
because of the label split.

The check reports and does not collapse populations. Choosing a canonical label
rewrites data and stays review-gated under the read-only default below.

`check_framework_label_congruence` (check 31) covers the drift class alignment
cannot see: labels are compared against each edge's OWN endpoints, joined via
the unique `control_id` index, not against populations. An edge labelled NIST
passes alignment as long as NIST controls exist somewhere — even when its own
source control is SPARTA. Each edge side is classified `match` / `mismatch` /
`unlabelled` / `dangling`. The 2026-08-11 baseline found 840 contradicting
labels and 135,350 edges (51% of the graph) with no labels at all — the
unlabelled pool concealed most of the graph's real connectivity (93k
SPARTA→NIST, 15k →D3FEND, 16k →CWE). After the review-gated repairs, both
mismatch and unlabelled are 0; dangling endpoints (F36_* namespaces and
removed controls) remain surfaced as findings. Authority order for repairs,
per the 2026-08-11 external validation: endpoint identity outranks the stored
edge label; upstream identifiers (STIX bundles for ATT&CK domains) outrank
both. Node relabels that cannot be resolved deterministically are quarantined
with `attack_domain_unresolved`, never guessed.

### QRA reasoning invariants

Reasoning is the field that shows why an answer follows from its evidence. Two
invariants govern it, and neither is a coverage percentage to be traded off.

**No reasoning may restate its own source text.** A QRA whose reasoning is
copied from the source control's description is well-formed, long, and
grammatical. It passes every emptiness and length test while carrying no
reasoning at all. Length is not a proxy for reasoning. `qra_reasoning_coverage`
reports this as `duplicates_source` in its breakdown, compared
whitespace-normalized and case-insensitive against `evidence_case.glossary`.

**No two QRAs may share an identical reasoning string.** Distinct questions
require distinct reasoning; a string reused across QRAs is boilerplate or a
copied field, not reasoning about any of them. `qra_reasoning_uniqueness`
enforces this with no tolerance threshold — any duplicate group fails the check.
It catches reuse regardless of where the text came from, including QRAs with no
evidence case, so it is the broader of the two.

Observed 2026-07-30 on AC-11.1: three different questions carried one identical
16-token reasoning string, itself a verbatim copy of the control description.
Sixteen tokens cleared the previous 5-token floor with room to spare, so the
monitor reported the dimension healthy. Both invariants above exist because that
defect was structurally invisible to a length-and-emptiness test.

Repair belongs to the review-gated lane: backfilling or regenerating reasoning
is a mutating operation and requires approved patch artifacts and rollback
manifests, per the read-only default below.

Read-only default:

- `monitor-sparta` must not mutate SPARTA corpus data by default.
- `monitor-sparta health --fix` is blocked unless
  `SPARTA_MONITOR_MUTATION_ENABLED=1` is explicitly set for an approved,
  rollback-backed apply.
- Regeneration requests from checkpoint/persona paths must write
  review-required manifests by default, not upsert controls, synthesize
  descriptions, create relationships, or start Stage 12.
- Missing source text is not a QRA eligibility problem to pad around. Stub text
  such as "requires QRA generation" must be treated as missing/unsupported
  evidence until authoritative text is supplied.

Former automatic remediation lanes are now review-gated:

- Backfill missing source controls into `sparta_controls`
- Backfill missing/empty control descriptions
- Backfill missing mind taxonomy tags
- Backfill embedding gaps
- Backfill QRA `evidence_case` fields (spans/glossary/version)
- Backfill missing/invalid QRA reasoning, including reasoning that restates its
  source text or is shared verbatim with another QRA

Each mutating lane requires approved patch artifacts and rollback manifests
before `SPARTA_MONITOR_MUTATION_ENABLED=1` may be used.

## Shared Maintainer Architecture

`/monitor-sparta` follows the same reporter/fixer/reviewer split used by
`/monitor-skill-health`:

1. Monitor lanes observe live state and write normalized findings, manifests, or
   tickets.
2. Tau-backed maintainer lanes lease one work item at a time.
3. A bounded repair subagent changes only the leased target.
4. A separate verifier/reviewer attaches deterministic proof.
5. Tickets or manifests close only after proof is attached and reconciled.

The monitor must not treat a report, WebGPT review, or CI green status as
closure proof. Mutating SPARTA repairs remain review-gated and rollback-backed.
Use `skills/monitor-sparta/run.sh receipt-gate ... --require-closure` before
closing or reconciling any monitor-sparta queue item from a worker or supervisor
receipt.
Use `skills/monitor-sparta/run.sh closure-review verify ...` and
`skills/monitor-sparta/run.sh closure-review review ...` to produce separate
`monitor_sparta.verifier_receipt.v1` and `monitor_sparta.reviewer_receipt.v1`
artifacts. Use `skills/monitor-sparta/run.sh closure-bundle-gate ...` when a
queue item is ready to reconcile: it requires a closing worker/supervisor
receipt, the verifier receipt, and the reviewer receipt with no blocking
findings.