Verify System

---
name: verify-system
description: "End-to-end system health verification. Checks all containers, ports, endpoints, and pipelines (logging, email, auth, socket, worker). Proves the system is truly working, not just running."
argument-hint: ""
---

# /verify-system Skill

## Purpose

Proves that the entire system is **truly working** — not just "containers are up" but every pipeline, every connection, every integration is functional. Run after `/create-new-project` or anytime you need confidence that the system is healthy.

**Philosophy:** Don't say "it's ready" — prove it with evidence.

## Verification Levels

### Level 0: Configuration Sanity

Static checks against `docker-compose.yml` — catch blueprint drift before paying the cost of `docker compose up`. These verify that what infra-agent generated matches what `infra-agent/children/` documents.

#### 0.1 Health-gated dependencies

Every service that talks to a stateful infrastructure dependency (PostgreSQL, RabbitMQ, Redis, Elasticsearch) must wait on `service_healthy`, not `service_started`. `service_started` only proves the container's PID 1 exists — the broker/db may still be in its bootstrap loop, causing first-run connection errors that "auto-recover" but pollute logs and confuse new contributors.

```bash
# Find any depends_on blocks that target stateful services with service_started
grep -nE "(rabbitmq|postgres|db|redis|elasticsearch):\s*$" -A1 docker-compose.yml \
  | grep -B1 "condition: service_started"
```

**Pass:** No matches — every stateful dep uses `service_healthy`.
**Fail diagnostic:** Print the offending service+dependency pairs and recommend flipping to `service_healthy`. See [infra-agent/children/health-checks.md](../../agents/infra-agent/children/health-checks.md) for the canonical pattern.

#### 0.2 Shared .NET project shadow volumes

When the API/Worker/Socket/LogIngest services mount source from host, every referenced .NET project (`Domain`, `Application`, `Infrastructure`, `Logging`, plus any other `.csproj` referenced by a service) needs its `bin/` and `obj/` shadowed by named volumes. Otherwise host-side `dotnet build` (run by a developer on macOS) collides with container-side Linux artefacts and produces mysterious assembly-load failures.

```bash
# For each shared project under src/, both bin and obj volumes must exist
for proj in Domain Application Infrastructure Logging; do
  base=$(echo "$proj" | tr '[:upper:]' '[:lower:]')   # domain, application, infrastructure, logging
  short=${base%ation}                                  # application → applic; tweak as needed per project
  grep -qE "^\s*${base}_bin:|^\s*${short}_bin:" docker-compose.yml || echo "MISSING: ${base}_bin"
  grep -qE "^\s*${base}_obj:|^\s*${short}_obj:" docker-compose.yml || echo "MISSING: ${base}_obj"
done
```

(Exact short-name convention follows [infra-agent/children/volume-strategy.md](../../agents/infra-agent/children/volume-strategy.md): `domain_bin`, `app_bin`, `infra_bin`, `logging_bin` and their `_obj` siblings. Any project referenced by a `.csproj` ProjectReference inside a service must appear in that service's `volumes:` block AND in the top-level `volumes:` declaration.)

**Pass:** Every shared project has a `_bin` and `_obj` named volume, declared at the top level AND mounted into every service whose csproj references it.
**Fail diagnostic:** List the missing volumes and the affected services. See [infra-agent/children/volume-strategy.md](../../agents/infra-agent/children/volume-strategy.md) — "Type 1: Shadow Volumes".

**Report format:**
```
Level 0 — Configuration:
  ✅ Health-gated deps      All stateful depends_on use service_healthy
  ✅ Shadow volumes         domain/app/infra/logging bin+obj all declared and mounted
  Result: ✅ PASS (2/2 checks)
```

If Level 0 fails, **stop and fix the compose file** before running Level 1+ — runtime checks against a structurally drifted compose file produce noisy false-positives.

### Level 1: Containers Running

Check every container is up and healthy (not just "created" or "restarting"):

```bash
docker compose ps --format "table {{.Name}}\t{{.Status}}"
```

**Pass criteria:**
- All containers show "Up" status
- Health-checked services show "(healthy)"
- No container in "Restarting" or "Exited" state (except init containers like minio-init which are expected to exit)

**Report format:**
```
Level 1 — Containers:
  ✅ wfm-api          Up 2 minutes
  ✅ wfm-db           Up 2 minutes (healthy)
  ✅ wfm-rabbitmq     Up 2 minutes (healthy)
  ✅ wfm-redis        Up 2 minutes (healthy)
  ✅ wfm-elasticsearch Up 2 minutes (healthy)
  ⚪ wfm-minio-init   Exited (expected — one-shot init)
  ... (all containers)
  Result: ✅ PASS (14/14 running)
```

### Level 2: Ports Accessible

For every service with an exposed port, verify TCP connectivity:

```bash
curl -sf -o /dev/null -w "%{http_code}" http://localhost:{port}/
```

**Services to check:**

| Service | Port | Check Method |
|---------|------|-------------|
| API | {api_port} | HTTP GET /api/health/ping |
| Swagger | {api_port} | HTTP GET /swagger/index.html |
| Socket | {socket_port} | HTTP GET (SignalR negotiate) |
| PostgreSQL | {db_port} | docker compose exec db pg_isready |
| RabbitMQ Management | {rmq_mgmt_port} | HTTP GET / |
| Redis | {redis_port} | docker compose exec redis redis-cli ping |
| Elasticsearch | {es_port} | HTTP GET /_cluster/health |
| Kibana | {kibana_port} | HTTP GET /api/status |
| Mailpit | {mailpit_port} | HTTP GET / |
| MinIO Console | {minio_console_port} | HTTP GET / |
| Adminer | {adminer_port} | HTTP GET / |
| Redis Commander | {redis_ui_port} | HTTP GET / |

**Report format:**
```
Level 2 — Ports:
  ✅ API (13000)           HTTP 200
  ✅ Swagger (13000)       HTTP 200
  ✅ PostgreSQL (15432)    pg_isready OK
  ✅ RabbitMQ (25673)      HTTP 200
  ✅ Redis (16379)         PONG
  ✅ Elasticsearch (19200) green
  ✅ Kibana (15601)        HTTP 200
  ✅ Mailpit (18025)       HTTP 200
  ✅ MinIO (19001)         HTTP 200
  ✅ Adminer (18083)       HTTP 200
  ✅ Redis Commander (18082) HTTP 200
  Result: ✅ PASS (12/12 accessible)
```

### Level 3: Application Health

Beyond port accessibility — does the application return **meaningful responses**?

```bash
# API health
curl -sf http://localhost:{api_port}/api/health/ping
# Expected: {"message":"pong","timestamp":"..."}

# Elasticsearch cluster
curl -sf http://localhost:{es_port}/_cluster/health
# Expected: {"status":"green"} or {"status":"yellow"}

# RabbitMQ queues exist
curl -sf -u guest:guest http://localhost:{rmq_mgmt_port}/api/queues
# Expected: logs.elasticsearch and emails.smtp queues listed

# Redis connectivity
docker compose exec redis redis-cli INFO server | head -5
# Expected: redis_version, uptime_in_seconds

# PostgreSQL tables
docker compose exec db psql -U {user} -d {db} -c '\dt'
# Expected: migration history table + any seeded tables
```

**Report format:**
```
Level 3 — Application Health:
  ✅ API health           {"message":"pong"}
  ✅ Elasticsearch        cluster: green, nodes: 1
  ✅ RabbitMQ queues      logs.elasticsearch ✓, emails.smtp ✓
  ✅ Redis                v7.x, uptime: 120s
  ✅ PostgreSQL           tables: __EFMigrationsHistory, Users, AuditLogs
  ✅ Seed data            admin@test.com exists
  Result: ✅ PASS (6/6 healthy)
```

### Level 4: Pipeline Smoke Tests

The most important level. **End-to-end proof** that each pipeline works.

#### 4.1 Logging Pipeline

```
Test: API writes a log → RMQ → LogIngest → Elasticsearch

Steps:
1. Call API endpoint: GET /api/health/ping
2. Wait 5 seconds (batch flush)
3. Query Elasticsearch for the log:
   curl -sf http://localhost:{es_port}/{project}-logs-*/_search?q=ping
4. Verify: log entry exists with service:"api", message contains "ping"
```

**Pass:** Log entry found in Elasticsearch within 10 seconds of API call.

#### 4.2 Email Pipeline

```
Test: API triggers email → RMQ → MailSender → Mailpit

Steps:
1. Call API auth register endpoint (if available) or trigger a test email
   POST /api/auth/register { email: "test-verify@test.com", password: "Test1234!" }
   (This should trigger a welcome email)
2. Wait 5 seconds
3. Check Mailpit for the email:
   curl -sf http://localhost:{mailpit_port}/api/v1/messages
4. Verify: email to test-verify@test.com exists in Mailpit
```

**Pass:** Email visible in Mailpit. If no auth register endpoint yet, skip with note: "Email pipeline ready but no trigger endpoint available yet."

#### 4.3 Auth Pipeline

```
Test: Register → Login → Token → Protected endpoint

Steps:
1. Register: POST /api/auth/register { email, password }
   → Expected: 200 + accessToken + refreshToken
2. Login: POST /api/auth/login { email, password }
   → Expected: 200 + accessToken
3. Use token: GET /api/health/ping with Authorization: Bearer {token}
   → Expected: 200
4. Without token: GET /api/auth/me (if exists)
   → Expected: 401
```

**Pass:** Full auth cycle works. If auth endpoints not yet implemented, skip with note.

#### 4.4 Socket Pipeline

```
Test: Socket connects and API can broadcast

Steps:
1. Check Socket is accepting connections:
   curl -sf http://localhost:{socket_port}/health (if exists)
2. Check Socket logs for "connected to RabbitMQ" or startup messages
3. (Full WebSocket test requires a client — note as manual test)
```

**Pass:** Socket service running and RMQ connected.

#### 4.5 Worker Pipeline

```
Test: Worker job executes on schedule

Steps:
1. Check Worker logs for SampleJob execution:
   docker compose logs worker --tail 20 | grep "SampleJob"
2. Verify: at least one "SampleJob executing" log entry exists
3. If SampleJob calls API: verify the API received the call (check API logs)
```

**Pass:** SampleJob has executed at least once.

#### 4.6 Redis Integration

```
Test: Redis is being used (tokens, settings, etc.)

Steps:
1. Check Redis has keys:
   docker compose exec redis redis-cli DBSIZE
2. If auth exists: register a user, check refresh token in Redis:
   docker compose exec redis redis-cli KEYS "refresh_token:*"
```

**Pass:** Redis has entries (settings seed or tokens).

#### 4.7 MinIO/Storage (if enabled)

```
Test: Default bucket exists

Steps:
1. Check MinIO bucket:
   curl -sf http://localhost:{minio_api_port}/minio/health/ready
2. Or check via mc: docker compose exec minio mc ls local/
```

**Pass:** MinIO healthy and default bucket exists.

---

## Final Report

After all levels complete, produce a comprehensive report:

```
╔══════════════════════════════════════════════════════════════╗
║                    SYSTEM VERIFICATION                       ║
║                    {ProjectName} — {date}                    ║
╠══════════════════════════════════════════════════════════════╣
║                                                              ║
║  Level 0 — Configuration:  ✅ PASS  (2/2 static checks)     ║
║  Level 1 — Containers:     ✅ PASS  (14/14 running)         ║
║  Level 2 — Ports:          ✅ PASS  (12/12 accessible)      ║
║  Level 3 — App Health:     ✅ PASS  (6/6 healthy)           ║
║  Level 4 — Pipelines:                                        ║
║    4.1 Logging:            ✅ PASS  (API→RMQ→ES verified)   ║
║    4.2 Email:              ✅ PASS  (email in Mailpit)      ║
║    4.3 Auth:               ✅ PASS  (register→login→token)  ║
║    4.4 Socket:             ✅ PASS  (connected, RMQ linked) ║
║    4.5 Worker:             ✅ PASS  (SampleJob executed)    ║
║    4.6 Redis:              ✅ PASS  (keys present)          ║
║    4.7 Storage:            ✅ PASS  (MinIO ready, bucket OK)║
║                                                              ║
║  OVERALL: ✅ ALL SYSTEMS OPERATIONAL                         ║
║                                                              ║
║  The system is verified and ready for development.           ║
║                                                              ║
╠══════════════════════════════════════════════════════════════╣
║  Endpoints:                                                  ║
║    API:        http://localhost:13000                         ║
║    Swagger:    http://localhost:13000/swagger                 ║
║    Kibana:     http://localhost:15601                         ║
║    Mailpit:    http://localhost:18025                         ║
║    RabbitMQ:   http://localhost:25673                         ║
║    MinIO:      http://localhost:19001                         ║
║    Adminer:    http://localhost:18083                         ║
║    Redis UI:   http://localhost:18082                         ║
║                                                              ║
║  Test Credentials:                                           ║
║    Admin: admin@test.com / Admin123!                         ║
╚══════════════════════════════════════════════════════════════╝
```

If any test FAILS:
```
║  4.2 Email:              ❌ FAIL  (no email in Mailpit)     ║
║      → Check: docker compose logs mail-sender               ║
║      → Likely: RMQ topology not declared, or SMTP error     ║
```

Include diagnostic hints for each failure — what to check, what's likely wrong.

---

## Usage

**Standalone:**
```
/verify-system
```

**Called by /create-new-project:**
Phase 4 of project creation automatically runs this skill.

**After changes:**
Run anytime after infrastructure changes, compose updates, or if something feels wrong.

## Important Rules

1. **Wait for services to stabilize.** After `docker compose up`, wait 30-60 seconds before starting verification. Services need time to boot, migrate, connect.
2. **Skip gracefully.** If an endpoint doesn't exist yet (e.g., auth not implemented), skip that test with a note — don't fail the entire verification.
3. **Pipeline tests are the real proof.** Level 1-3 are basic checks. Level 4 proves the system actually works end-to-end.
4. **Log evidence.** For each pipeline test, show the actual log line or response that proves it worked.
5. **Ports from .env.** Read port numbers from `.env` file, don't hardcode. Every project might use different ports.
6. **Init containers are expected to exit.** minio-init, migration runners — these exit after their job. Don't flag them as failures.

## Accumulated Learnings

(Auto-rebuilt by /save-learnings from `learnings/*.md` frontmatter. Do not edit by hand. Initially empty — entries appear as the skill encounters reusable edge cases.)