Self Monitor

---
name: self-monitor
version: 1.1.0
author: Polycat
tags: [monitoring, health, infrastructure]
license: MIT
platform: universal
description: >
  Proactive self-monitoring of infrastructure, services, and health. Tracks
  disk/memory/load, service health, cron job status, recent errors. Auto-fixes safe
  issues. Triggers on: health check, heartbeat, monitor status, service status,
  infrastructure check.
---

> **Compatible with Claude Code, Codex CLI, Cursor, Windsurf, and any SKILL.md-compatible agent.**

# Self Monitor

Proactive self-monitoring: infrastructure, services, and health.

## Usage

Run during heartbeats or scheduled checks.

### 1. Infrastructure Health

```bash
# Disk usage
df -h / | awk 'NR==2 {print $5}' | tr -d '%'

# Memory usage  
free -m | awk 'NR==2 {printf "%.0f", $3/$2*100}'

# Load average
uptime | awk -F'load average:' '{print $2}' | awk -F',' '{print $1}'

# Top processes by memory
ps aux --sort=-%mem | head -10

# Top processes by CPU
ps aux --sort=-%cpu | head -10
```

**Thresholds:**
| Metric | Warning | Critical |
|--------|---------|----------|
| Disk | > 80% | > 90% |
| Memory | > 85% | > 95% |
| Load | > 2.0 | > 4.0 |

### 2. Service Health

```bash
# Check if a process is running
pgrep -f "your_process_name" >/dev/null && echo "OK" || echo "FAIL"

# Check HTTP endpoint
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health

# Check systemd service
systemctl is-active --quiet nginx && echo "OK" || echo "FAIL"

# Check Docker container
docker ps --filter "name=mycontainer" --filter "status=running" -q | grep -q . && echo "OK" || echo "FAIL"

# Tailscale (if using)
tailscale status --json 2>/dev/null | jq -r '.Self.Online' || echo "FAIL"
```

### 3. Cron Job Health

```bash
# Check recent cron executions
grep CRON /var/log/syslog | tail -20

# Count failures in last 24h
grep -c "CRON.*error\|CRON.*fail" /var/log/syslog

# List scheduled jobs
crontab -l
```

### 4. Recent Errors

```bash
# Check system logs for errors
journalctl -p err --since "1 hour ago" 2>/dev/null | tail -20

# Check application logs
tail -50 ~/workspace/projects/*/logs/*.log 2>/dev/null | grep -i "error"

# Check dmesg for hardware/kernel issues
dmesg | tail -20 | grep -i "error\|fail\|warn"
```

## Quick Health Check (for heartbeat)

```bash
#!/bin/bash
# Quick health snapshot

DISK=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
MEM=$(free -m | awk 'NR==2 {printf "%.0f", $3/$2*100}')
LOAD=$(uptime | awk -F'load average:' '{print $2}' | awk -F',' '{print $1}' | xargs)

echo "Disk: ${DISK}% | Mem: ${MEM}% | Load: ${LOAD}"

# Alert if thresholds exceeded
[ "$DISK" -gt 90 ] && echo "⚠️ Disk critical!"
[ "$MEM" -gt 95 ] && echo "⚠️ Memory critical!"
```

## Root Cause Iron Law (inspired by gstack /investigate)

**Never apply a fix without identifying the root cause first.**

When an issue is detected:
1. **Investigate** -- collect evidence (logs, metrics, timeline)
2. **Analyze** -- identify the root cause, not just the symptom
3. **Hypothesize** -- state the cause in one sentence before acting
4. **Fix** -- apply the smallest targeted fix
5. **Verify** -- confirm the fix resolved the root cause, not just the symptom

If root cause is unclear after 2 attempts:
- Log what you tried and what you observed
- Escalate to owner with evidence, not guesses
- Do NOT apply speculative fixes that could mask the real issue

**Anti-patterns:**
- Restarting a service without checking why it died
- Clearing disk without checking what filled it
- Killing a process without checking what caused high usage

## Destructive Command Guard (inspired by gstack /guard)

Before executing any of these commands, **stop and verify**:

| Pattern | Risk | Required Check |
|---------|------|----------------|
| `rm -rf` | Data loss | Verify path is correct, not `/`, not home |
| `DROP TABLE` / `DROP DATABASE` | Data loss | Confirm backup exists |
| `git push --force` | History loss | Confirm branch and remote |
| `kill -9` | Corruption | Try graceful stop first |
| `chmod 777` | Security | Use minimal permissions instead |
| `truncate` / `> file` | Data loss | Confirm file is a log, not config |
| `docker system prune -a` | Image loss | List what will be removed first |

Subagents MUST NOT execute destructive commands. Return the command to main session for approval.

## Proactive Actions

**When issues detected:**

| Issue | Auto-Action | Alert? |
|-------|------------|--------|
| Disk > 90% | Investigate what grew, then clean | Yes |
| Key process down | Check logs first, then restart | Yes |
| Cron 3+ failures | Root cause analysis, then report | Yes |
| Memory > 95% | Identify leak/offender, then act | Yes |

**Auto-fixable (safe, only after root cause check):**
```bash
# Clean old logs (> 7 days)
find /var/log -name "*.log" -mtime +7 -delete 2>/dev/null
find ~/.cache -type f -mtime +7 -delete 2>/dev/null

# Clean temp files
rm -f /tmp/agent-temp-* 2>/dev/null
rm -rf ~/.cache/pip 2>/dev/null
```

## Report Format

```markdown
## 🔍 Self-Monitor Report - [TIME]

### Health Summary
| Metric | Value | Status |
|--------|-------|--------|
| Disk | XX% | ✅/⚠️/🔴 |
| Memory | XX% | ✅/⚠️/🔴 |
| Load | X.X | ✅/⚠️/🔴 |
| Services | X/Y up | ✅/⚠️ |

### Issues Found
- [Issue 1]: [Action taken or recommended]

### Top Resource Consumers
| Process | CPU% | MEM% |
|---------|------|------|
| ... | ... | ... |
```

## Integration with Scheduled Tasks

Add to your crontab or task scheduler:
```cron
# Run health check every 30 minutes
*/30 * * * * /path/to/health-check.sh >> /var/log/health-check.log 2>&1
```

Or run manually as part of your workflow:
```bash
./health-check.sh
```

---

## Security Checks

Run daily (or alongside any health check). Uses only `sha256sum`/`shasum`, `grep`, `find`, `jq`. Designed to complete in under 5 seconds.

### Critical Files to Monitor

```
/path/to/workspace/SOUL.md
/path/to/workspace/IDENTITY.md
/path/to/workspace/MEMORY.md
/path/to/workspace/AGENTS.md
/path/to/workspace/TOOLS.md
```

### 1. File Integrity (SHA256 Baseline)

See `../security-guardian/scripts/check_integrity.sh` for the full script.

Logic:
- On first run: generate `data/security-baseline.json` with SHA256 of each critical file
- On subsequent runs: compare current hashes against baseline; report any drift
- Manual reset: "reset security baseline" → regenerate baseline (do this after intentional edits to SOUL.md/MEMORY.md)

Baseline format:
```json
{
  "generated_at": "2026-04-02T07:00:00Z",
  "files": {
    "/path/to/workspace/SOUL.md": "sha256:<hash>"
  }
}
```

### 2. Prompt Injection Scan

See `../security-guardian/scripts/scan_injections.sh` for the full script.

Scans last 3 days of `memory/*.md` files for patterns like:
- "ignore previous instructions"
- "you are now" / "pretend you are"
- "forget everything" / "override your"
- "new system prompt"

### 3. Credential Leak Scan

See `../security-guardian/scripts/scan_credentials.sh` for the full script.

Scans workspace `*.md`, `*.json`, `*.txt`, `*.yaml` (excluding `.git/`, `node_modules/`) for:
- `api_key = 'xxx'` / `token = 'xxx'`
- `sk-...` (OpenAI/Anthropic keys)
- `ghp_...` (GitHub tokens)
- Hardcoded passwords

### Security Report Format

```
🛡️ Security Report — YYYY-MM-DD HH:MM UTC

✅ File Integrity: OK (5/5 files match baseline)
⚠️  Injection Scan: 1 suspicious pattern in memory/2026-04-01.md
✅ Credential Scan: Clean

 Summary: 1 warning — review flagged file
```

### Severity & Alerting

| Level | Meaning | Action |
|---|---|---|
| ✅ OK | All clear | Log only |
| ⚠️ WARNING | Suspicious but not confirmed | Note in report |
| 🚨 CRITICAL | Confirmed drift or credential exposed | Alert owner immediately |

**CRITICAL triggers:**
- SOUL.md or IDENTITY.md hash changed without "reset baseline" command
- Credential pattern found in a committed/shared file
- Injection pattern found in a file that was then acted upon

**On CRITICAL:** alert owner via WhatsApp.

### Cron Schedule

Run daily at 06:00 UTC (before morning briefing). Silent unless CRITICAL found.

```json
{
  "id": "security-daily",
  "schedule": "0 6 * * *",
  "timezone": "UTC",
  "task": "Run self-monitor security checks: file integrity, injection scan, credential scan. Alert owner via WhatsApp if CRITICAL.",
  "delivery": { "mode": "silent" }
}
```

---

## 📊 Skill Analytics Section

Include this in the daily self-monitor report (after security checks).

```bash
LOG="/path/to/workspace/data/skill-analytics.jsonl"
TODAY=$(date -u +%Y-%m-%d)

if [ ! -f "$LOG" ] || [ $(wc -l < "$LOG") -eq 0 ]; then
  echo "### 📊 Skill Analytics\n_No data yet._"
else
  echo "### 📊 Skill Analytics (last 24h)"
  echo ""
  
  # Total today
  TOTAL=$(grep "$TODAY" "$LOG" | wc -l)
  echo "**Invocations:** $TOTAL"
  echo ""
  
  # Top 5 skills today
  echo "**Top skills:**"
  grep "$TODAY" "$LOG" \
    | jq -r '.skill' \
    | sort | uniq -c | sort -rn \
    | head -5 \
    | awk '{printf "- %s (%d)\n", $2, $1}'
  echo ""
  
  # Unused skills (never logged)
  KNOWN="ai-pa billing-monitor calendar-setup eval hebrew-nikud maintenance meetings memory-tiering monday-for-agents owner-briefing pa-onboarding self-learning self-monitor skill-master skill-scout supervisor whatsapp youtube-watcher skill-analytics"
  UNUSED=""
  for s in $KNOWN; do
    COUNT=$(jq -r '.skill' "$LOG" | grep -c "^${s}$" 2>/dev/null || echo 0)
    [ "$COUNT" -eq 0 ] && UNUSED="$UNUSED $s"
  done
  [ -n "$UNUSED" ] && echo "**Never used:**$UNUSED" || echo "**Never used:** none"
fi
```

### Output Example

```
### 📊 Skill Analytics (last 24h)

**Invocations:** 11

**Top skills:**
- supervisor (4)
- owner-briefing (2)
- meetings (2)
- whatsapp (2)
- self-monitor (1)

**Never used:** hebrew-nikud youtube-watcher
```