Auto Verify

---
name: auto-verify
description: Attempt automated verification of criteria before falling back to manual. Parses criterion text for automation hints and executes appropriate tool (curl, browser, file check).
---

# Auto-Verify Skill

Attempt automated verification of criteria that would otherwise require manual review.
This skill parses criterion text for automation hints, checks tool availability, and
executes verification commands before marking items as truly manual.

## When This Skill Runs

- Invoked by `/verify-task` when processing MANUAL type criteria
- Invoked by `/phase-checkpoint` for Manual Local Verification items
- Invoked by `code-verification` skill before browser fallback
- Invoked by `browser-verification` skill for HTTP-first optimization

## Workflow Overview

```
1. Parse criterion for automation hints (keywords, patterns)
2. Check tool availability (Bash always, browser conditional)
3. Select best available tool and generate command
4. Execute with appropriate timeout
5. Interpret result and return structured output
```

## Step 1: Pattern Detection

Analyze criterion text and optional `Verify:` line for automation keywords.
Patterns are checked in priority order — first match wins.

### Pattern Matching Table

| Priority | Pattern Keywords | Tool | Command Template |
|----------|-----------------|------|------------------|
| 1 | `curl`, `endpoint`, `API`, `/api/`, `returns`, `status code`, `HTTP` | curl | Status/response check |
| 2 | `response contains`, `JSON contains`, `body includes` | curl+jq/grep | Content verification |
| 3 | `redirect`, `redirects to`, `Location header`, `302`, `301` | curl | Header inspection |
| 4 | `health`, `running`, `reachable`, `alive`, `up`, `accessible` | curl | Health check |
| 5 | `page loads`, `loads at`, `visit`, `navigate to` | curl first | HTTP status (skip browser if sufficient) |
| 6 | `visible`, `shows`, `displays`, `element`, `selector`, `DOM` | browser | Browser snapshot required |
| 7 | `console`, `no errors`, `warnings`, `logs` | browser | Console inspection |
| 8 | `file exists`, `created`, `generated`, `written` | bash | File existence check |
| 9 | `directory exists`, `folder`, `path` | bash | Directory check |
| 10 | `env var`, `environment variable`, `\$`, `set` | bash | Environment check |
| 11 | `looks`, `feels`, `UX`, `intuitive`, `user experience`, `brand`, `tone` | NONE | Truly manual |

### Pattern Detection Algorithm

```
function detectPattern(criterion_text, verify_line):
  text = lowercase(criterion_text + " " + verify_line)

  # Check patterns in priority order
  for pattern in PATTERN_TABLE:
    if any(keyword in text for keyword in pattern.keywords):
      return pattern

  # No pattern matched - truly manual
  return { tool: NONE, reason: "No automation pattern detected" }
```

## Step 2: Tool Availability Check

### Always Available

These tools are always present and can be used without checking:

| Tool | Capabilities |
|------|--------------|
| **Bash** | curl, grep, jq, test, file operations, environment checks |
| **Read** | File content inspection |
| **Glob** | File pattern matching |
| **Grep** | Content search |

### Conditionally Available

Check availability before use:

| Tool | Check Method | Fallback |
|------|--------------|----------|
| ExecuteAutomation Playwright | `mcp__playwright__*` or `mcp__executeautomation__*` tools exist | Browser MCP |
| Browser MCP | `mcp__browsermcp__*` tools exist | Microsoft Playwright |
| Microsoft Playwright MCP | `mcp__playwright__*` tools exist | Chrome DevTools |
| Chrome DevTools MCP | `mcp__chrome-devtools__list_pages` responds | Manual |

### Availability Matrix

```
TOOL_AVAILABILITY = {
  "curl": ALWAYS,
  "bash": ALWAYS,
  "file_ops": ALWAYS,
  "browser": CHECK_REQUIRED
}
```

## Step 3: Command Generation

Based on detected pattern, generate the appropriate verification command.

### HTTP Patterns (curl-based)

**Status Check:**
```bash
# Check if endpoint returns success (2xx/3xx)
curl -sf "{url}" -o /dev/null && echo "PASS:status_ok" || echo "FAIL:status_{http_code}"
```

**Response Contains Text:**
```bash
# Check if response body contains expected text
curl -s "{url}" | grep -q "{expected_text}" && echo "PASS:text_found" || echo "FAIL:text_missing"
```

**JSON Field Exists:**
```bash
# Check if JSON response has expected field
curl -s "{url}" | jq -e '.{json_path}' > /dev/null 2>&1 && echo "PASS:field_exists" || echo "FAIL:field_missing"
```

**Redirect Check:**
```bash
# Check if redirect goes to expected location
curl -sI "{url}" | grep -i "location:" | grep -qi "{expected_location}" && echo "PASS:redirect_ok" || echo "FAIL:redirect_wrong"
```

**Health Check:**
```bash
# Check service health (try /health first, then root)
curl -sf "{url}/health" -o /dev/null 2>/dev/null || curl -sf "{url}" -o /dev/null && echo "PASS:service_healthy" || echo "FAIL:service_unreachable"
```

### File Patterns (bash-based)

**File Exists:**
```bash
test -f "{path}" && echo "PASS:file_exists" || echo "FAIL:file_missing"
```

**File Contains:**
```bash
grep -q "{pattern}" "{path}" && echo "PASS:content_found" || echo "FAIL:content_missing"
```

**Directory Exists:**
```bash
test -d "{path}" && echo "PASS:dir_exists" || echo "FAIL:dir_missing"
```

### Environment Patterns (bash-based)

**Environment Variable Set:**
```bash
test -n "${VAR_NAME}" && echo "PASS:env_set" || echo "FAIL:env_missing"
```

### Browser Patterns

For browser-required criteria, delegate to browser-verification skill:
- Use the browser tool fallback chain (ExecuteAutomation → BrowserMCP → Playwright → ChromeDevTools)
- Pass the route, selector, and expected state from the criterion

## Step 4: Execution Protocol

### Timeouts

| Tool | Timeout | Rationale |
|------|---------|-----------|
| curl | 5 seconds | Network requests should be fast |
| bash (file ops) | 2 seconds | Local operations are quick |
| browser | 30 seconds | Page loading and interaction take time |

### Execution Steps

1. **Generate command** from detected pattern
2. **Set timeout** based on tool
3. **Execute via Bash tool** (for curl/bash patterns)
4. **Parse output** for PASS/FAIL prefix
5. **Capture full output** for evidence
6. **Handle errors** (timeout, connection refused, etc.)

### Error Handling

| Error Type | Action |
|------------|--------|
| Timeout | Mark as FAIL with "timeout after {N}s" |
| Connection refused | Mark as FAIL with "connection refused - is server running?" |
| Command not found | Mark as FAIL with "tool not available" |
| Unexpected output | Mark as FAIL with captured output |

## Step 5: Result Interpretation

Return a structured result for each criterion:

```
AUTO-VERIFY RESULT
------------------
Criterion: "{original criterion text}"
Pattern Detected: {pattern name or "none"}
Tool Used: {curl | bash | browser | none}
Command: {executed command or "N/A"}
Status: PASS | FAIL | MANUAL
Duration: {execution time in ms}
Output: {captured output, truncated if >500 chars}
Suggested Fix: {if FAIL, provide actionable suggestion}
Reason: {if MANUAL, explain why automation not possible}
```

### Status Definitions

| Status | Meaning | Next Action |
|--------|---------|-------------|
| **PASS** | Criterion verified automatically | No human review needed |
| **FAIL** | Automation attempted but failed | Show error, suggest fix, allow human override |
| **MANUAL** | No automation possible | List for human review with reason |

## Step 6: Integration Examples

### Example 1: API Endpoint Verification

**Input:**
```
Criterion: "POST /api/users returns 201 with user object"
Verify: POST /api/users with body {"name": "test"}
```

**Processing:**
1. Pattern detected: `API`, `returns`, `201` → curl pattern
2. Tool: curl (always available)
3. Command: `curl -s -o /dev/null -w "%{http_code}" -X POST -H "Content-Type: application/json" -d '{"name":"test"}' http://localhost:3000/api/users`
4. Execute: Returns "201"
5. Result: PASS

**Output:**
```
AUTO-VERIFY RESULT
------------------
Criterion: "POST /api/users returns 201 with user object"
Pattern Detected: API status code
Tool Used: curl
Command: curl -s -o /dev/null -w "%{http_code}" -X POST ...
Status: PASS
Duration: 234ms
Output: 201
```

### Example 2: Page Load Check

**Input:**
```
Criterion: "Dashboard page loads at /dashboard"
```

**Processing:**
1. Pattern detected: `page loads` → curl first (HTTP status sufficient)
2. Tool: curl
3. Command: `curl -sf http://localhost:3000/dashboard -o /dev/null`
4. Execute: Success (exit code 0)
5. Result: PASS (no browser needed)

**Output:**
```
AUTO-VERIFY RESULT
------------------
Criterion: "Dashboard page loads at /dashboard"
Pattern Detected: page accessibility
Tool Used: curl (HTTP-first)
Command: curl -sf http://localhost:3000/dashboard -o /dev/null
Status: PASS
Duration: 156ms
Output: HTTP 200 OK
```

### Example 3: Truly Manual Criterion

**Input:**
```
Criterion: "Copy matches brand tone and feels professional"
```

**Processing:**
1. Pattern detected: `feels`, `brand`, `tone` → truly manual
2. Tool: none
3. Result: MANUAL

**Output:**
```
AUTO-VERIFY RESULT
------------------
Criterion: "Copy matches brand tone and feels professional"
Pattern Detected: subjective judgment
Tool Used: none
Command: N/A
Status: MANUAL
Duration: 0ms
Output: N/A
Reason: Subjective criteria requiring human judgment (brand tone, professional feel)
```

### Example 4: Failed Automation

**Input:**
```
Criterion: "API returns list of users"
```

**Processing:**
1. Pattern detected: `API`, `returns` → curl pattern
2. Tool: curl
3. Command: `curl -sf http://localhost:3000/api/users`
4. Execute: Connection refused
5. Result: FAIL

**Output:**
```
AUTO-VERIFY RESULT
------------------
Criterion: "API returns list of users"
Pattern Detected: API response
Tool Used: curl
Command: curl -sf http://localhost:3000/api/users
Status: FAIL
Duration: 5012ms
Output: curl: (7) Failed to connect to localhost port 3000: Connection refused
Suggested Fix: Start the dev server with `npm run dev` or check if port 3000 is correct
```

## URL and Path Extraction

When generating commands, extract URLs and paths from criterion text:

### Base URL Resolution

Before constructing URLs, resolve the base URL from deployment configuration:

```
1. Read verification-config.json
2. IF deployment.enabled AND deployment.useForBrowserVerification:
     - Invoke vercel-preview skill (or use cached result)
     - IF preview URL found: BASE_URL = preview URL
     - ELSE IF fallbackToLocal: BASE_URL = devServer.url (warn)
     - ELSE: Return BLOCKED (no URL available)
3. ELSE:
     - BASE_URL = devServer.url
```

**IMPORTANT:** All auto-verify HTTP checks must use BASE_URL, not hardcoded localhost.

### URL Extraction

```
# Look for explicit URLs (use as-is, don't prepend BASE_URL)
/https?:\/\/[^\s]+/

# Look for route patterns (prepend BASE_URL)
/(?:at|to|from)\s+\/[a-zA-Z0-9\/_-]+/

# Look for localhost patterns (replace with BASE_URL if deployment enabled)
/localhost:\d+[^\s]*/

# Default to BASE_URL from resolution above
```

### Path Extraction

```
# Look for file paths
/(?:file|path|in)\s+[a-zA-Z0-9\/_.-]+/

# Look for common patterns
/src\/[^\s]+/
/\.\/[^\s]+/
```

## Truly Manual Patterns

These patterns indicate criteria that genuinely require human judgment:

| Pattern | Reason |
|---------|--------|
| `looks`, `appears`, `visual` | Subjective visual assessment |
| `feels`, `intuitive`, `UX` | User experience judgment |
| `brand`, `tone`, `voice` | Brand consistency |
| `professional`, `polished` | Quality perception |
| `easy to use`, `user-friendly` | Usability judgment |
| `appropriate`, `suitable` | Context-dependent evaluation |
| `creative`, `engaging` | Subjective content quality |

When these patterns are detected, return MANUAL status with a clear reason explaining
why automation is not feasible.

## Configuration

This skill respects settings from `.claude/verification-config.json`:

```json
{
  "devServer": {
    "url": "http://localhost:3000"
  },
  "deployment": {
    "enabled": true,
    "service": "vercel",
    "useForBrowserVerification": true,
    "fallbackToLocal": true
  },
  "autoVerify": {
    "enabled": true,
    "httpTimeout": 5000,
    "fileTimeout": 2000,
    "browserTimeout": 30000,
    "httpFirst": true
  }
}
```

**URL resolution priority:**
1. If `deployment.enabled` and preview URL available → use preview URL
2. If `deployment.enabled` but no preview and `fallbackToLocal` → use devServer.url (warn)
3. If `deployment.enabled` but no preview and NOT `fallbackToLocal` → BLOCKED
4. If deployment not enabled → use devServer.url

If `autoVerify.enabled` is false, skip automation attempts and return MANUAL for all criteria.