Skip to content
Back to skills

Testing Methodology

ASecurity

Testing methodology — test harness design, mock strategies, flakiness prevention, and structured test planning

  • 10 stars
  • 0 votes
  • 0 copies
  • 1 view
  • Added November 13, 2026
developmentgojavabashsqltestingapidatabasebackend

Works with

  • cli
  • api

Security analysis

A100/100

Scanned November 13, 2026

npx -y skills add reliant-labs/forge --skill testing-methodology --agent claude-code

Installs into .claude/skills of the current project.

Are you the author of Testing Methodology?

Add the live security badge to your README. It updates with every re-scan.

Security grade badge for Testing Methodology
[![Security: A — Skills Directory](https://www.skillsdirectory.com/api/skills/reliant-labs-testing-methodology/badge)](https://www.skillsdirectory.com/skills/reliant-labs-testing-methodology)

More formats (shields.io, HTML) on the badges page. Keep it an A: scan every change in CI with Pro.

Download with Pro
SKILL.md
---
name: testing-methodology
description: Testing methodology — test harness design, mock strategies, flakiness prevention, and structured test planning
---

# Testing Methodology

## Phase 1: Test Infrastructure Discovery

Before writing any tests, understand what exists:

### Test Framework Detection
- What test frameworks are installed? (jest, pytest, go test, etc.)
- How are tests run? (`npm test`, `make test`, `go test ./...`)
- Where do tests live? (`__tests__/`, `*_test.go`, `tests/`)
- What's the naming convention? (`*.test.ts`, `*_test.py`, `Test*.java`)

### Existing Patterns
- What test utilities/helpers exist?
- Are there existing fixtures or factories?
- What mocking libraries are used?
- How is test data managed?
- Are there existing harnesses to build on?

### Test Execution Environment
- How is CI configured for tests?
- What's the coverage tool and current coverage?
- Are there test databases or containers?
- What environment variables do tests need?

**Document findings** - this information helps you and future agents write consistent tests.

## Phase 2: Test Harness Design

The key to testing complex systems is good harnesses. A harness handles setup, teardown, and provides utilities so individual tests stay focused.

### What to Mock vs What to Keep Real

**Keep REAL when:**
- It's the thing you're actually testing
- It's fast and deterministic (pure functions, in-memory DBs)
- Mocking it would miss real bugs (ORM behavior, query logic)
- The real thing is simpler than a mock

**MOCK when:**
- External services (APIs, third-party services)
- Non-deterministic sources (time, random, network)
- Slow resources that aren't under test
- Components with complex setup that aren't relevant to the test

**The boundary rule:** Mock at system boundaries, keep internals real.

### Harness Patterns

#### Integration Server Harness
For API/backend testing:
```
TestHarness
├── Server (real, started once per suite)
├── Database (isolated per test, migrations run)
├── External Service Mocks (controllable)
└── Test Client (authenticated, with helpers)
```

#### Database Harness
For data layer testing:
```
TestHarness
├── Database Connection (transaction per test, rolled back)
├── Migration Runner (schema always current)
├── Fixture Loader (predictable test data)
└── Query Assertion Helpers
```

#### Async/Queue Harness
For message processing:
```
TestHarness
├── Queue (in-memory or real, isolated)
├── Worker Pool (controllable concurrency)
├── Event Collector (capture for assertions)
└── Completion Waiter (with timeout)
```

#### External Service Harness
For integration points:
```
TestHarness
├── Mock Server (httptest, wiremock, etc.)
├── Request Recorder (verify calls made)
├── Response Programmer (control responses)
└── Failure Injector (test error handling)
```

## Phase 3: Writing Good Tests

### Test Structure
Every test should have:
1. **Arrange**: Set up preconditions (use harness)
2. **Act**: Execute the thing being tested
3. **Assert**: Verify the outcome
4. **Cleanup**: Handled by harness (teardown)

### Test Naming
Names should describe the scenario and expected outcome:
- `test_create_user_with_valid_email_succeeds`
- `TestPaymentProcessor_RefundExceedingBalance_ReturnsError`
- `it('should reject duplicate usernames', ...)`

### Avoiding Flakiness

**Timing issues:**
- Never use fixed sleeps (`time.Sleep(1s)`)
- Poll with timeout for async operations
- Use event-driven waiting when possible

```
// Bad
time.Sleep(2 * time.Second)
assert(result.IsComplete())

// Good
waitFor(func() bool { return result.IsComplete() }, timeout: 5s, poll: 100ms)
```

**State isolation:**
- Each test gets fresh state (new DB transaction, clean mocks)
- Don't rely on test execution order
- Clean up resources in teardown, not in test body

**Live / heavy tests are opt-in:**
- A test that hits a live external dependency (real sandbox, deployed environment, real credential) must be gated behind an explicit opt-in env var (e.g. `RUN_LIVE_TESTS=1`) and skip-with-a-reason when it's unset.
- The default suite stays hermetic and green in CI; the live test runs only when a human or a dedicated job asks for it. A live test that runs by default is a flake and a credential-leak risk.

**Non-determinism:**
- Mock time.Now() for time-dependent logic
- Seed random generators deterministically
- Control UUIDs/IDs in tests

### Test Categories

#### Unit Tests
- Test single functions/methods in isolation
- Mock all dependencies
- Fast (<10ms each)
- High coverage of edge cases

#### Integration Tests
- Test components working together
- Real database, real internal services
- Mock external boundaries
- Medium speed (<1s each)

#### E2E Tests
- Test complete user flows
- Minimal mocking (only external services)
- Slower but high confidence
- Focus on critical paths

#### Contract Tests
- Verify API contracts between services
- Both provider and consumer tests
- Catch breaking changes early

#### Concurrency Tests
- Test race conditions explicitly
- Use race detectors (go test -race)
- Stress test with goroutines/threads
- Test deadlock scenarios

## Phase 4: Test Tooling

### Run the cheapest tier that answers the question

Most of an edit loop's test time goes to compiling, linking, and re-running
tests whose result is already known — not to test bodies. Keep two tiers and
use the right one:

- **Inner loop, after each edit:** fast tests only, scoped to the packages you
  touched, with the runner's cache on. In Go that is
  `go test -short ./internal/<pkg>/...` with no `-count=1` and no `-race`.
  Go's test cache keys on the test binary, its flags, and the files and env
  vars the test reads, so a cached pass is a real pass for a hermetic test;
  `-count=1` throws that away, and `-race` rebuilds everything under a
  separate cache key and runs 2-10x slower. Never point the build cache at a
  private directory — a cold cache recompiles the world.
- **Full gate, once before you finish (and in CI):** the whole suite with
  `-race -count=1`, plus the tagged integration and e2e lanes.
- **Keep the fast tier fast.** A test that takes more than ~2s gets
  `if testing.Short() { t.Skip("<why>") }`, or has its slow side-effect
  bypassed under `-short`, and its slow path still runs in the full gate. Gate
  it, don't gut it: never weaken an assertion to get under the budget.
- A run expected to take more than ~2 minutes goes in the background; never
  block on a long foreground call.

<!-- @forge-only:start -->
In a forge project both tiers are Taskfile targets:
`task test:short -- ./internal/<pkg>/...` is the inner loop, and `task test`
is the full lane CI runs.
<!-- @forge-only:end -->

### Useful Test Scripts

Create tools that make testing easier. Write to `.reliant/tools/` or the project's scripts directory.

**Run the inner-loop tier (pass the packages you touched):**
```bash
#!/bin/bash
# .reliant/tools/test-unit.sh
go test -short "${@:-./...}"
```

**Run with coverage:**
```bash
#!/bin/bash
# .reliant/tools/test-coverage.sh
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
echo "Coverage report: coverage.html"
```

**Seed test database:**
```bash
#!/bin/bash
# .reliant/tools/seed-test-db.sh
DB_PATH="${TEST_DB:-./test.db}"
sqlite3 "$DB_PATH" < ./testdata/seed.sql
echo "Test database seeded at $DB_PATH"
```

**Run flaky test detector:**
```bash
#!/bin/bash
# .reliant/tools/flaky-detector.sh
TEST_NAME="$1"
RUNS="${2:-10}"
for i in $(seq 1 $RUNS); do
  if ! go test -run "$TEST_NAME" ./...; then
    echo "FLAKY: Failed on run $i"
    exit 1
  fi
done
echo "STABLE: Passed $RUNS consecutive runs"
```

## Phase 5: Test Plan Output

When reviewing code or planning tests, provide a structured test plan:

### Coverage Assessment
- What's currently tested
- What's missing coverage
- Risk areas without tests

### Unit Tests to Add
| Function/Method | Test Cases | Priority |
|-----------------|------------|----------|
| `validateEmail` | valid format, invalid format, empty, unicode | High |
| `calculateTotal` | normal, with discount, zero items, overflow | High |

### Integration Tests to Add
| Scenario | Components | Mocking Strategy |
|----------|------------|------------------|
| User registration flow | API + DB + Email | Mock email service |
| Payment processing | API + DB + Stripe | Mock Stripe API |

### E2E Tests to Add
| User Flow | Steps | Assertions |
|-----------|-------|------------|
| Complete checkout | Login -> Add item -> Pay -> Confirm | Order created, email sent |

### Concurrency Tests
- Race conditions to test
- Deadlock scenarios to verify
- Load/stress test considerations

### What to Mock vs Keep Real
| Component | Decision | Rationale |
|-----------|----------|-----------|
| Database | Real (postgres) | Need to test actual queries |
| Redis | Mock | Not testing caching logic |
| Stripe API | Mock | External, non-deterministic |

# Guidelines

## DO:
- Follow existing test patterns in the codebase
- Create reusable harnesses and helpers
- Test edge cases and error paths
- Make test failures self-explanatory
- Run tests before and after changes

## DON'T:
- Write tests that pass when code is broken
- Create flaky tests (fix or delete them)
- Over-mock (test realistic scenarios)
- Ignore existing test utilities
- Write tests without understanding the code
- Run the full suite with `-count=1` / `-race` after every edit — that is the full gate's job, not the inner loop's

## Test Quality Checklist
- [ ] Tests actually test the right thing (not just coverage theater)
- [ ] Tests fail when code is broken
- [ ] Tests are deterministic (no flakiness)
- [ ] Tests are fast enough for the feedback loop
- [ ] Tests have clear failure messages
- [ ] Test code is maintainable (DRY with helpers/fixtures)

Attribution

Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.

View all in development →

Comments

Loading comments…

Newsletter

Get the weekly list of safe new skills, and the ones we pulled.