Full Stack Debugger

---
name: full-stack-debugger
description: This skill should be used when debugging full-stack issues that span UI, backend, and database layers. It provides a systematic workflow to detect errors, analyze root causes, apply fixes iteratively, and verify solutions through automated server restarts and browser-based testing. Ideal for scenarios like failing schedulers, import errors, database issues, or API payload problems where issues originate in backend code but manifest in the UI.
---

# Full Stack Debugger

## Overview

The Full Stack Debugger enables systematic debugging of issues across the entire application stack (UI/Frontend, Backend/API, Database/State). It combines browser testing, log analysis, code examination, and automated server restart/verification to iteratively identify and fix issues one at a time until the system is fully operational.

This skill uses a proven workflow: **Detection → Analysis → Fix → Restart → Verification → Iteration** to systematically resolve issues that developers encounter during development and testing.

## When to Use This Skill

Trigger this skill when observing:
- Error states in the UI (dashboard, buttons failing, status showing errors)
- Repeated failures in backend logs (task execution failures, import errors, database errors)
- Unexpected database state (rows showing failed status when they should succeed)
- API endpoints returning errors or unexpected responses
- Services failing to initialize or process tasks
- Cascading failures across multiple components

## Debugging Workflow

### Phase 1: Detection

Detect errors from multiple sources:

**Browser UI Detection:**
- Navigate to the affected page/feature in the browser
- Check for error messages, red warning states, or disabled functionality
- Read console error messages using DevTools
- Note the specific UI state and what action triggered the error

**Backend Log Detection:**
- Query recent error logs using `tail -200 /path/to/logs/errors.log`
- Search for error patterns related to the issue using `grep`
- Note error timestamps, error messages, and stack traces
- Look for repeated errors (indicates systemic issue)

**Database State Detection:**
- Query the database directly using sqlite3
- Check status of recent tasks, transactions, or records
- Look for failed, incomplete, or error states
- Note which records are affected and what their states are

**Example:** When debugging a scheduler failure:
1. Navigate to System Health dashboard
2. Observe scheduler showing "0 done" or "X failed"
3. Check `/logs/errors.log` for error messages
4. Query `queue_tasks` table to see failed task records

### Phase 2: Analysis

Analyze root causes by reading code and logs:

**Code Analysis:**
- Read the error file/module indicated in error stack traces
- Check imports - look for missing `from X import Y` statements
- Check class names - verify instantiation matches actual class names
- Look for syntax errors - unmatched quotes, unclosed parentheses
- Check function signatures - ensure payloads match expected parameters
- Read reference documentation (`references/common_errors.md`) for error patterns

**Log Analysis:**
- Extract error messages from logs
- Look for patterns like `'optional'` (missing import), `unterminated string` (syntax error), `'attribute'` (wrong class name)
- Trace error propagation backward to find the originating issue
- Check timestamps - multiple errors at same time indicate batch failure

**API/Payload Analysis:**
- Check what payload the API is sending to task handlers
- Read the task handler code to see what fields it expects
- Compare actual payload vs expected payload
- Look for missing required fields

**Example:** When debugging "name 'Optional' is not defined":
1. Find the file mentioned in error (`analysis_executor.py`)
2. Read the imports section
3. Notice `Optional` is used but not imported
4. Check line 14: `from typing import Dict, List, Any` - missing `Optional`
5. Fix: Add `Optional` to the import statement

### Phase 3: Fix (One Issue at a Time)

Apply fixes one issue per iteration:

**Before Fixing:**
- Verify this is the first/next issue to fix
- Read the relevant code section carefully
- Use the fix patterns from `references/fix_templates.md`

**Common Fix Patterns:**
- **Missing imports:** Add to import statement (e.g., `from typing import Optional`)
- **Wrong class name:** Update import and instantiation to match actual class
- **Missing docstring quotes:** Add opening `"""` to docstring
- **Wrong payload fields:** Add missing required fields to payload dictionary
- **Syntax errors:** Fix unmatched quotes, parentheses, brackets

**After Fixing:**
- Read back the changed code to verify syntax
- Check the edit was correct (line numbers, indentation)
- Only fix ONE issue, even if multiple exist - don't cascade fixes
- Document what was changed in a clear comment

**Example Fix:**
```python
# BEFORE
from typing import Dict, List, Any

# AFTER
from typing import Dict, List, Any, Optional
```

### Phase 4: Restart (Automated)

Restart the backend server after each fix:

```bash
# Kill existing processes
lsof -ti:8000 | xargs kill -9 2>/dev/null

# Clear Python bytecode cache
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null
find . -type f -name "*.pyc" -delete 2>/dev/null

# Restart backend
sleep 3 && python -m src.main --command web > /tmp/backend_restart.log 2>&1 &
sleep 10  # Wait for startup

# Verify health
curl -m 5 http://localhost:8000/api/health
```

### Phase 5: Verification

Verify the fix worked through multiple checks:

**Health Check:**
- Call `/api/health` endpoint
- Verify `"status": "healthy"`
- If still failing, check logs for new errors

**Browser Verification:**
- Navigate to the affected UI page
- Trigger the action that previously failed
- Verify the error is gone
- Check for new errors in console

**Database Verification:**
- Query the affected records/tasks
- Verify status changed from failed/error to success/completed
- Check that metrics updated (e.g., scheduler shows "1 done" instead of "0 done")

**Log Verification:**
- Check recent logs for the same error
- Verify no new errors appeared
- Look for success messages or "completed" status

**Example:**
- Scheduler should show "1 done" instead of "0 done"
- Task record should show status="completed" instead of "failed"
- No error messages in logs
- WebSocket shows healthy status in UI

### Phase 6: Iteration

If issues remain, repeat the cycle:

1. **Continue if more issues exist:**
   - Check logs for remaining errors
   - If yes, return to Phase 2 (Analysis)
   - Fix the next issue (Phase 3)
   - Restart (Phase 4)
   - Verify (Phase 5)

2. **Stop when all issues fixed:**
   - All schedulers show completed execution counts
   - UI shows no error states
   - Logs show no error patterns
   - Tasks/records show success status
   - Full verification complete

## Common Error Patterns

See `references/common_errors.md` for patterns to recognize:
- Python syntax errors (unterminated strings, missing quotes)
- Import errors (`name 'X' is not defined`, `cannot import name 'Y'`)
- Class/attribute errors (`'dict' object has no attribute 'symbol'`)
- Type errors (passing wrong data type)
- Payload/configuration errors (missing required fields)

## Fix Templates

See `references/fix_templates.md` for ready-to-use fix patterns:
- How to add missing imports
- How to fix class name mismatches
- How to fix docstring syntax
- How to add missing payload fields
- How to fix type errors

## Tools Used

- **Playwright Browser Tools:** Navigate UI, verify changes
- **Read/Grep Tools:** Examine code and logs
- **Bash:** Server restart, cache clearing, health checks
- **Edit Tool:** Apply code fixes
- **Database Queries:** Verify task/record state

## MCP Tools Integration

Use robo-trader-dev MCP tools for 95%+ token-efficient debugging:

| Task | MCP Tool | Token Savings | Usage |
|------|----------|---------------|-------|
| Analyze error logs | `mcp__robo-trader-dev__analyze_logs` | 98% | Pattern detection with time windows |
| System health check | `mcp__robo-trader-dev__check_system_health` | 97% | Database, queues, API, disk status |
| Diagnose DB locks | `mcp__robo-trader-dev__diagnose_database_locks` | 95% | Correlate logs with code patterns |
| Queue monitoring | `mcp__robo-trader-dev__queue_status` | 96% | Real-time queue backlog analysis |
| Coordinator status | `mcp__robo-trader-dev__coordinator_status` | 94% | Init status, error details |
| Error pattern fix | `mcp__robo-trader-dev__suggest_fix` | 90% | Known pattern matching with examples |
| Read code files | `mcp__robo-trader-dev__smart_file_read` | 85% | Progressive context (summary/targeted/full) |
| Find related files | `mcp__robo-trader-dev__find_related_files` | 88% | Import/git/similarity analysis |

**Example debugging workflow**:
```python
# 1. Detect errors (MCP instead of tail/grep)
mcp__robo-trader-dev__analyze_logs(patterns=["ERROR", "TIMEOUT"], time_window="1h")

# 2. Check system health (MCP instead of curl loops)
mcp__robo-trader-dev__check_system_health(components=["database", "queues", "api_endpoints"])

# 3. Diagnose specific issue (MCP instead of sqlite3 + code reading)
mcp__robo-trader-dev__diagnose_database_locks(time_window="24h", include_code_references=True)

# 4. Get fix suggestions (MCP instead of manual pattern matching)
mcp__robo-trader-dev__suggest_fix(error_message="name 'Optional' is not defined", context_file="src/services/analyzer.py")
```

**Integration with robo-trader architecture**:
- Queue operations: Use `queue_status` to monitor PORTFOLIO_SYNC, DATA_FETCHER, AI_ANALYSIS
- Coordinator debugging: Use `coordinator_status` for BroadcastCoordinator, AIChatCoordinator init issues
- Database access: Use `query_portfolio` or `diagnose_database_locks` instead of direct sqlite3 connections

## Key Principles

1. **One issue at a time** - Fix one problem per iteration to prevent cascading failures
2. **Verify immediately** - Always restart and verify after each fix
3. **Multi-layer detection** - Check UI, logs, and database for clues
4. **Iterative refinement** - Continue until all issues resolved
5. **Automated restart** - Always use clean restart (kill + cache clear + restart)
6. **Browser verification** - Always test in actual UI, not just logs