Afrexai Qa Engine

# QA & Test Engineering Command Center

Complete quality assurance system — from test strategy to automation frameworks, coverage analysis, and release readiness. Works for any stack, any team size.

## When to Use

- Planning test strategy for a new feature or project
- Writing unit, integration, or E2E tests
- Reviewing test quality and coverage gaps
- Setting up test automation and CI/CD quality gates
- Performance testing and load analysis
- Security testing checklist
- Bug triage and defect management
- Release readiness assessment

---

## Phase 1: Test Strategy

### Strategy Brief

Before writing any tests, define the strategy:

```yaml
# test-strategy.yaml
project: "[name]"
scope: "[feature/module/full product]"
risk_level: high | medium | low
stack:
  language: "[TypeScript/Python/Java/Go]"
  framework: "[React/Express/Django/Spring]"
  test_runner: "[Jest/Vitest/pytest/JUnit/Go test]"
  e2e_tool: "[Playwright/Cypress/Selenium]"

# What are we testing?
test_scope:
  - area: "[e.g., Auth module]"
    risk: high
    test_types: [unit, integration, e2e]
    priority: 1
  - area: "[e.g., Settings page]"
    risk: low
    test_types: [unit]
    priority: 3

# What's NOT in scope (and why)
exclusions:
  - "[e.g., Third-party widget — covered by vendor]"

# Quality targets
targets:
  line_coverage: 80
  branch_coverage: 70
  critical_path_coverage: 100
  max_flaky_rate: 2%
  max_test_duration_unit: 10ms
  max_test_duration_integration: 500ms
  max_test_duration_e2e: 30s
```

### Risk-Based Test Allocation

Not everything needs the same testing depth. Use the risk matrix:

| Risk Level | Unit Tests | Integration | E2E | Manual/Exploratory |
|-----------|-----------|-------------|-----|-------------------|
| **Critical** (payments, auth, data loss) | 95%+ coverage | Full API coverage | Happy + error paths | Exploratory session |
| **High** (core features, user-facing) | 85%+ coverage | Key integrations | Happy path | Spot check |
| **Medium** (secondary features) | 70%+ coverage | Critical paths only | Smoke only | On release |
| **Low** (admin, internal tools) | 50%+ coverage | None | None | None |

### Test Pyramid

Follow the pyramid — not the ice cream cone:

```
         /  E2E  \          ← Few (5-10%) — slow, expensive, brittle
        / Integr. \         ← Some (15-25%) — API contracts, DB queries
       /   Unit    \        ← Many (65-80%) — fast, isolated, cheap
```

**Anti-pattern: Ice cream cone** (mostly E2E, few unit tests) = slow CI, flaky builds, expensive maintenance.

**Decision rule:** Can this be tested at a lower level? → Test it there.

---

## Phase 2: Unit Testing

### Anatomy of a Good Unit Test

Every unit test follows AAA (Arrange-Act-Assert):

```
1. ARRANGE — Set up test data, mocks, state
2. ACT     — Call the function/method under test
3. ASSERT  — Verify the output matches expectations
```

### Unit Test Checklist (per function)

For each function/method, verify:

- [ ] **Happy path** — expected input → expected output
- [ ] **Edge cases** — empty input, null/undefined, zero, max values
- [ ] **Boundary values** — off-by-one, min-1, max+1
- [ ] **Error handling** — invalid input → correct error thrown
- [ ] **Return types** — correct type, shape, structure
- [ ] **Side effects** — does it modify state it shouldn't?
- [ ] **Idempotency** — calling twice gives same result?

### What to Mock (and What NOT to Mock)

**Mock these:**
- External APIs (HTTP calls, third-party services)
- Database queries (in unit tests only)
- File system operations
- Date/time (use fake timers)
- Random number generators
- Environment variables

**DO NOT mock these:**
- The function under test itself
- Pure utility functions (test them directly)
- Data transformations
- Simple value objects

**Mock rule of thumb:** If removing the mock would make the test hit the network, file system, or database → mock it. Otherwise → don't.

### Test Naming Convention

Use the pattern: `[unit] [scenario] [expected result]`

Examples:
- `calculateTotal returns 0 for empty cart`
- `validateEmail throws for missing @ symbol`
- `parseDate handles ISO 8601 with timezone offset`

### Coverage Analysis

**Metrics that matter:**
| Metric | Target | Why |
|--------|--------|-----|
| Line coverage | 80%+ | Basic completeness |
| Branch coverage | 70%+ | Catches missed if/else paths |
| Function coverage | 90%+ | Ensures all functions are tested |
| Critical path coverage | 100% | Business-critical code fully verified |

**Coverage traps to avoid:**
- 100% line coverage ≠ good tests (assertions matter more than lines hit)
- Coverage on generated code inflates numbers
- Trivial getters/setters pad coverage without value
- Coverage should INCREASE over time, never decrease

---

## Phase 3: Integration Testing

### What Integration Tests Cover

Integration tests verify that components work TOGETHER:

- API endpoint → middleware → handler → database → response
- Service A calls Service B and handles the response
- Message queue producer → consumer → side effect
- Auth flow: login → token → authenticated request

### Integration Test Patterns

**Pattern 1: API Contract Testing**
```
1. Start test server (or use supertest/httptest)
2. Send HTTP request with specific payload
3. Assert: status code, response body shape, headers
4. Assert: database state changed correctly
5. Assert: side effects triggered (emails, events)
```

**Pattern 2: Database Integration**
```
1. Start test database (SQLite in-memory or test container)
2. Run migrations
3. Seed test data
4. Execute query/operation
5. Assert: data matches expectations
6. Teardown (truncate or rollback transaction)
```

**Pattern 3: External Service**
```
1. Record real API response (VCR/nock/wiremock)
2. Replay recorded response in tests
3. Assert: your code handles the response correctly
4. Also test: timeout, 500 error, malformed response
```

### Integration Test Checklist

- [ ] **Happy path** — full flow works end-to-end
- [ ] **Auth** — unauthenticated returns 401, wrong role returns 403
- [ ] **Validation** — bad payload returns 400 with error details
- [ ] **Not found** — missing resource returns 404
- [ ] **Conflict** — duplicate create returns 409
- [ ] **Rate limiting** — excessive requests return 429
- [ ] **Database constraints** — unique violations, foreign keys
- [ ] **Concurrency** — two simultaneous writes don't corrupt data
- [ ] **Timeout handling** — external service timeout → graceful fallback

---

## Phase 4: End-to-End (E2E) Testing

### E2E Strategy

E2E tests verify complete user journeys. They're expensive — be strategic:

**Test these E2E:**
- User registration → email verification → first login
- Purchase flow → payment → confirmation
- Critical business workflows (the ones that make money)
- Cross-browser/device smoke tests

**DON'T test these E2E:**
- Individual form validations (unit test)
- API error handling (integration test)
- Edge cases (lower-level tests)
- Visual styling (visual regression tools)

### E2E Test Template

```yaml
test_name: "[User journey name]"
preconditions:
  - "[User is logged in]"
  - "[Product exists in catalog]"
steps:
  - action: "Navigate to /products"
    verify: "Product list is visible"
  - action: "Click 'Add to Cart' on Product A"
    verify: "Cart badge shows 1"
  - action: "Click 'Checkout'"
    verify: "Checkout form displayed"
  - action: "Fill payment details and submit"
    verify: "Order confirmation page with order ID"
postconditions:
  - "Order exists in database with status 'paid'"
  - "Confirmation email sent"
max_duration: 30s
```

### Flaky Test Management

Flaky tests are the #1 CI killer. Handle them:

**Flaky Test Triage:**
1. **Identify** — Track test pass rates over 10+ runs
2. **Classify** — Why is it flaky?
   - Timing/race condition → Add explicit waits, not sleep()
   - Test data dependency → Isolate test data per run
   - External service → Mock it or use test container
   - Browser rendering → Use visibility checks, not delays
3. **Quarantine** — Move to @flaky suite, run separately
4. **Fix or delete** — Flaky test unfixed for 2 weeks → delete it

**Flaky rate target:** < 2% of total test runs

---

## Phase 5: Performance Testing

### Performance Test Types

| Type | Purpose | When |
|------|---------|------|
| **Load test** | Normal traffic handling | Before every release |
| **Stress test** | Find breaking point | Quarterly or before scaling |
| **Spike test** | Sudden traffic burst | Before marketing campaigns |
| **Soak test** | Memory leaks over time | Monthly or after major changes |
| **Capacity test** | Max users/throughput | Planning infrastructure |

### Performance Test Plan

```yaml
test_name: "[API/Page] Load Test"
target: "[URL or endpoint]"
baseline:
  p50_response: "[current p50 ms]"
  p95_response: "[current p95 ms]"
  p99_response: "[current p99 ms]"
  error_rate: "[current %]"

scenarios:
  - name: "Normal load"
    vus: 50          # virtual users
    duration: 5m
    ramp_up: 30s
    thresholds:
      p95_response: "< 500ms"
      error_rate: "< 1%"

  - name: "Peak load"
    vus: 200
    duration: 10m
    ramp_up: 1m
    thresholds:
      p95_response: "< 2000ms"
      error_rate: "< 5%"

  - name: "Stress test"
    vus: 500
    duration: 5m
    ramp_up: 2m
    # Find the breaking point — no thresholds, observe
```

### Performance Metrics Dashboard

Track these per endpoint:

| Metric | Green | Yellow | Red |
|--------|-------|--------|-----|
| p50 response | < 200ms | 200-500ms | > 500ms |
| p95 response | < 500ms | 500ms-2s | > 2s |
| p99 response | < 1s | 1-5s | > 5s |
| Error rate | < 0.1% | 0.1-1% | > 1% |
| Throughput | > baseline | 80-100% baseline | < 80% |
| CPU usage | < 60% | 60-80% | > 80% |
| Memory usage | < 70% | 70-85% | > 85% |
| DB query time | < 50ms avg | 50-200ms | > 200ms |

### Common Performance Fixes

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| Slow API response | N+1 queries | Batch/join queries |
| Memory climbing | Object retention | Profile heap, fix leaks |
| Timeout spikes | Connection pool exhaustion | Increase pool, add queuing |
| Slow page load | Large bundle | Code split, lazy load |
| DB bottleneck | Missing index | Add index on WHERE/JOIN columns |
| High CPU | Synchronous compute | Move to worker/queue |

---

## Phase 6: Security Testing

### Security Test Checklist

Run through these for every feature/release:

**Authentication & Authorization:**
- [ ] Passwords hashed with bcrypt/argon2 (not MD5/SHA1)
- [ ] Session tokens are random, sufficient length (128+ bits)
- [ ] JWT tokens have short expiry (15 min access, 7 day refresh)
- [ ] Failed login rate limiting (5 attempts → lockout)
- [ ] Password reset tokens expire (1 hour max)
- [ ] Role-based access enforced server-side (not just UI)
- [ ] Can't access other users' data by changing IDs in URL

**Input Validation:**
- [ ] SQL injection — parameterized queries everywhere
- [ ] XSS — output encoding, CSP headers
- [ ] CSRF — tokens on state-changing requests
- [ ] Path traversal — validate file paths, no `../`
- [ ] Command injection — never pass user input to shell
- [ ] File upload — validate type, size, scan for malware
- [ ] JSON/XML parsing — depth limits, entity expansion disabled

**Data Protection:**
- [ ] HTTPS everywhere (HSTS header)
- [ ] Sensitive data encrypted at rest
- [ ] PII not logged (mask in log output)
- [ ] API keys not in client-side code
- [ ] CORS configured correctly (not `*`)
- [ ] Security headers set (X-Frame-Options, X-Content-Type-Options)

**Infrastructure:**
- [ ] Dependencies scanned for CVEs (npm audit / pip audit)
- [ ] Docker images scanned (Trivy/Snyk)
- [ ] Secrets not in code/env files (use vault)
- [ ] Error messages don't leak internals
- [ ] Admin endpoints behind VPN/IP allowlist

### OWASP Top 10 Quick Reference

| # | Vulnerability | Test For |
|---|--------------|----------|
| A01 | Broken Access Control | Access other users' resources, bypass role checks |
| A02 | Cryptographic Failures | Weak hashing, plaintext secrets, expired certs |
| A03 | Injection | SQL, XSS, command, LDAP injection |
| A04 | Insecure Design | Business logic flaws, missing rate limits |
| A05 | Security Misconfiguration | Default creds, verbose errors, open ports |
| A06 | Vulnerable Components | Outdated deps with known CVEs |
| A07 | Authentication Failures | Brute force, weak passwords, session fixation |
| A08 | Data Integrity Failures | Unsigned updates, CI/CD pipeline injection |
| A09 | Logging Failures | Missing audit logs, no alerting on breaches |
| A10 | SSRF | Internal network access via user-controlled URLs |

---

## Phase 7: Bug Triage & Defect Management

### Bug Report Template

```yaml
bug_id: "[auto or manual]"
title: "[Short description of the bug]"
severity: P0-critical | P1-high | P2-medium | P3-low
reporter: "[name]"
date: "[YYYY-MM-DD]"

environment:
  os: "[OS + version]"
  browser: "[Browser + version]"
  app_version: "[version/commit]"
  
steps_to_reproduce:
  1. "[Step 1]"
  2. "[Step 2]"
  3. "[Step 3]"

expected_result: "[What should happen]"
actual_result: "[What actually happens]"
frequency: "always | intermittent | once"
screenshots: "[links]"
logs: "[relevant log output]"
```

### Severity Classification

| Level | Definition | SLA | Example |
|-------|-----------|-----|---------|
| **P0 Critical** | System down, data loss, security breach | Fix in 4 hours | Payment processing broken |
| **P1 High** | Major feature broken, no workaround | Fix in 24 hours | Users can't login |
| **P2 Medium** | Feature broken with workaround | Fix this sprint | Search returns wrong results sometimes |
| **P3 Low** | Minor issue, cosmetic | Fix when convenient | Button alignment off by 2px |

### Bug Triage Process (Weekly)

```
1. Review all new bugs (unassigned)
2. For each bug:
   a. Reproduce — can you trigger it?
   b. Classify severity (P0-P3)
   c. Estimate fix effort (S/M/L)
   d. Assign to owner + sprint
   e. Link to related bugs/stories
3. Review P0/P1 bugs from last week — are they fixed?
4. Close bugs that can't be reproduced (after 2 attempts)
5. Update metrics dashboard
```

### Bug Metrics Dashboard

Track weekly:

| Metric | Formula | Target |
|--------|---------|--------|
| Bug escape rate | Bugs found in prod / total bugs | < 10% |
| Mean time to fix (P0) | Avg hours from report to deploy | < 8 hours |
| Mean time to fix (P1) | Avg hours from report to deploy | < 48 hours |
| Bug reopen rate | Reopened bugs / closed bugs | < 5% |
| Test escape analysis | Bugs that SHOULD have been caught | Track & reduce |
| Open bug count | Total open by severity | Trending down |

---

## Phase 8: Release Readiness

### Release Checklist

Before shipping to production:

**Code Quality:**
- [ ] All unit tests passing
- [ ] All integration tests passing
- [ ] E2E smoke suite passing
- [ ] No new lint warnings/errors
- [ ] Code reviewed and approved
- [ ] No known P0/P1 bugs open for this release

**Coverage & Quality Gates:**
- [ ] Line coverage ≥ target (80%)
- [ ] Branch coverage ≥ target (70%)
- [ ] No coverage decrease from last release
- [ ] Mutation testing score ≥ 60% (if applicable)

**Performance:**
- [ ] Load test passed (within thresholds)
- [ ] No performance regressions vs baseline
- [ ] Bundle size within budget

**Security:**
- [ ] Dependency audit clean (no critical/high CVEs)
- [ ] Security checklist completed
- [ ] Secrets rotated if needed

**Operational Readiness:**
- [ ] Monitoring/alerts configured for new features
- [ ] Rollback plan documented
- [ ] Feature flags in place for risky changes
- [ ] Database migration tested and reversible
- [ ] Runbook updated

### Release Readiness Score

Score 0-100 across 5 dimensions:

| Dimension | Weight | Scoring |
|-----------|--------|---------|
| **Test coverage** | 25% | 100 if targets met, -10 per gap area |
| **Bug status** | 25% | 100 if 0 P0/P1, -20 per open P0, -10 per P1 |
| **Performance** | 20% | 100 if all green, -15 per yellow, -30 per red |
| **Security** | 20% | 100 if clean, -25 per critical, -15 per high |
| **Operational** | 10% | 100 if checklist complete, -20 per missing item |

**Ship threshold: ≥ 80 overall, no dimension below 60**

---

## Phase 9: CI/CD Quality Gates

### Pipeline Quality Gates

Configure these gates in your CI pipeline:

```yaml
# Quality gate configuration
gates:
  - name: "Lint"
    stage: pre-commit
    command: "npm run lint"
    blocking: true
    
  - name: "Unit Tests"
    stage: commit
    command: "npm test -- --coverage"
    blocking: true
    thresholds:
      pass_rate: 100%
      coverage_line: 80%
      coverage_branch: 70%
      
  - name: "Integration Tests"
    stage: merge
    command: "npm run test:integration"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Security Scan"
    stage: merge
    command: "npm audit --audit-level=high"
    blocking: true
    
  - name: "E2E Smoke"
    stage: staging
    command: "npm run test:e2e:smoke"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Performance"
    stage: staging
    command: "npm run test:perf"
    blocking: false  # Alert only
    thresholds:
      p95_regression: 20%
```

### Test Automation Maturity Model

Rate your team 1-5:

| Level | Description | Characteristics |
|-------|------------|-----------------|
| **1 — Manual** | All testing is manual | No automation, long release cycles |
| **2 — Reactive** | Some unit tests, no CI | Tests written after bugs, not before |
| **3 — Structured** | Test pyramid, CI pipeline | Unit + integration, automated on push |
| **4 — Proactive** | Full automation, quality gates | E2E + perf + security in pipeline, TDD |
| **5 — Optimized** | Self-healing, predictive | Flaky auto-quarantine, AI-assisted testing, continuous deployment |

---

## Phase 10: Test Maintenance

### Weekly Test Health Review

```yaml
review_date: "[YYYY-MM-DD]"

metrics:
  total_tests: 0
  pass_rate_7d: "0%"
  flaky_tests: 0
  flaky_rate: "0%"
  avg_suite_duration: "0s"
  coverage_line: "0%"
  coverage_branch: "0%"
  
actions:
  quarantined: []     # Tests moved to flaky suite
  deleted: []         # Tests removed (obsolete/unfixable)
  fixed: []           # Flaky tests fixed this week
  added: []           # New tests added
  
trends:
  coverage_delta: "+0%"     # vs last week
  flaky_delta: "+0"         # vs last week
  duration_delta: "+0s"     # vs last week
  
notes: ""
```

### Test Maintenance Rules

1. **No commented-out tests** — delete or fix, never comment
2. **No skipped tests > 2 weeks** — fix or remove
3. **No test duplication** — each behavior tested once at the right level
4. **Test names must be readable** — someone new should understand what broke
5. **Shared test utilities** — common setup in fixtures/factories, not copy-pasted
6. **Test data isolation** — each test creates its own data, cleans up after
7. **No magic numbers** — use named constants in assertions
8. **Assertion messages** — custom messages on complex assertions

### Common Test Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **Sleeping tests** | `sleep(2000)` instead of waiting | Use explicit waits/polling |
| **Test interdependence** | Test B relies on Test A's state | Isolate — each test sets up its own state |
| **Assertionless tests** | Test runs code but doesn't assert | Add meaningful assertions |
| **Brittle selectors** | CSS selectors that break on redesign | Use data-testid or aria roles |
| **God test** | One test verifying 20 things | Split into focused tests |
| **Mock overload** | Everything mocked, nothing real tested | Only mock external boundaries |
| **Hardcoded data** | Tests break when seed data changes | Use factories/builders |
| **Ignoring test output** | "It passed, ship it" | Review WHY it passed — is the assertion meaningful? |

---

## Quick Reference: Natural Language Commands

Tell the agent:
- **"Create test strategy for [feature]"** → Generates strategy brief
- **"Write unit tests for [function/file]"** → AAA-structured tests with edge cases
- **"Review test coverage for [module]"** → Gap analysis + recommendations
- **"Write integration tests for [API endpoint]"** → Full HTTP test suite
- **"Plan E2E tests for [user journey]"** → E2E test template
- **"Run security checklist for [feature]"** → OWASP-based security review
- **"Triage these bugs: [list]"** → Severity classification + assignment
- **"Release readiness check"** → Full readiness score + blockers
- **"Performance test plan for [endpoint]"** → Load/stress test configuration
- **"Fix flaky test [name]"** → Root cause analysis + fix strategy