Harness Feature Flags

# Harness Feature Flags

> Flag lifecycle management, A/B testing infrastructure, and gradual rollout design. Ship features safely with controlled exposure and clean retirement.

## When to Use

- When designing feature flag architecture for a new project or feature
- When auditing existing flags for staleness, test coverage, or hygiene
- When planning gradual rollout or A/B testing strategies
- NOT for deployment pipeline configuration (use harness-deployment)
- NOT for environment variable management (use harness-secrets)
- NOT for application performance testing under flag variants (use harness-perf)

## Process

### Phase 1: DETECT -- Discover Flag Definitions and Usage

1. **Identify flag provider.** Scan the project for feature flag infrastructure:
   - **Third-party SDKs:** LaunchDarkly (`launchdarkly-node-server-sdk`), Unleash (`unleash-client`), Flagsmith (`flagsmith-nodejs`), Split (`@splitsoftware/splitio`)
   - **Custom implementations:** Files matching `*feature-flag*`, `*toggle*`, `*flags*`
   - **Configuration-based:** `flags.json`, `features.json`, `feature-flags.yaml`
   - **Environment-based:** Feature flags controlled via environment variables (`FEATURE_*`, `FF_*`)

2. **Catalog all flag definitions.** For each flag found, record:
   - Flag name/key
   - Flag type (boolean, multivariate, percentage rollout)
   - Default value (what happens when the flag service is unreachable)
   - Where it is defined (provider dashboard, config file, code)
   - Creation date or first appearance in git history

3. **Map flag usage in code.** For each flag, find all evaluation points:
   - Source files that check the flag value
   - Conditional branches gated by the flag
   - Components that render differently based on flag state
   - API endpoints with flag-dependent behavior
   - Test files that exercise flag variants

4. **Detect flag categories.** Classify each flag:
   - **Release flag:** Gates a new feature for gradual rollout (temporary)
   - **Experiment flag:** Controls A/B test variants (temporary)
   - **Ops flag:** Circuit breaker or kill switch (permanent)
   - **Permission flag:** Controls access to premium features (permanent)

5. **Present detection summary:**

   ```
   Feature Flag Detection:
     Provider: LaunchDarkly (SDK v7.x)
     Flags defined: 23
     Flags evaluated in code: 19 (4 defined but unused)
     Categories: 12 release, 5 experiment, 4 ops, 2 permission
     Stale candidates: 6 (release flags older than 90 days)
   ```

---

### Phase 2: DESIGN -- Recommend Flag Architecture and Rollout Strategy

1. **Evaluate flag naming convention.** Check for consistency:
   - Recommended format: `{team}.{feature}.{variant}` (e.g., `payments.stripe-v2.enabled`)
   - Flags should have descriptive names (not `flag_123` or `test_flag`)
   - Naming should indicate category (prefix with `ops.` for operational flags)
   - Temporary flags should include a target removal date in metadata

2. **Design flag evaluation architecture.** Recommend patterns for:
   - **Server-side evaluation:** Flags evaluated on the backend, consistent behavior
   - **Client-side evaluation:** Flags evaluated in the browser/app, real-time updates
   - **Edge evaluation:** Flags evaluated at CDN layer for zero-latency decisions
   - **Default values:** Every flag must have a safe default (feature off) for provider outages

3. **Design rollout strategy.** For release flags:
   - **Percentage rollout:** Start at 1%, monitor metrics, increase to 5%, 25%, 50%, 100%
   - **User segment targeting:** Internal users first, then beta users, then general availability
   - **Geographic rollout:** Single region first, expand after validation
   - **Time-based gates:** Enable during low-traffic periods for initial validation
   - Define success criteria for each rollout stage (error rate, latency, conversion)

4. **Design A/B testing infrastructure.** For experiment flags:
   - Consistent user assignment (same user always sees same variant)
   - Statistical significance calculation before declaring a winner
   - Event tracking integration for measuring experiment outcomes
   - Guardrail metrics that auto-disable experiments if health degrades
   - Experiment duration limits (no indefinite experiments)

5. **Design fallback behavior.** Ensure resilience:
   - Provider SDK timeout configuration (fail fast, not block)
   - Local cache for flag values (stale-while-revalidate)
   - Default values that are safe (feature disabled, not enabled)
   - Circuit breaker on flag evaluation errors
   - Monitoring on flag evaluation latency and error rate

---

### Phase 3: VALIDATE -- Verify Flag Hygiene and Test Coverage

1. **Check test coverage per flag.** For each flag, verify:
   - Tests exist for both the flag-on and flag-off code paths
   - Integration tests cover the default value behavior (provider down scenario)
   - No tests depend on a specific flag value being set in production
   - Test utilities exist for overriding flag values in test context

2. **Validate flag consistency.** Check for common issues:
   - Flags checked in code but not defined in the provider (will use defaults)
   - Flags defined in the provider but never checked in code (dead flags)
   - Same flag checked with different keys in different files (typos)
   - Nested flag checks that create combinatorial complexity

3. **Check for flag coupling.** Identify problematic patterns:
   - Flags that depend on other flags (if A and B, then C)
   - Flags that must be enabled in a specific order
   - Flags that share state or side effects
   - Recommend decoupling or consolidating dependent flags

4. **Validate rollout configuration.** For active rollouts:
   - Percentage values are within expected ranges
   - Targeting rules are correctly configured
   - Kill switch is functional (can disable the feature instantly)
   - Rollback plan is documented

5. **Generate validation report:**

   ```
   Flag Validation: [PASS/WARN/FAIL]

   Test coverage: WARN (3 flags missing flag-off tests)
   Consistency: PASS (all code references match definitions)
   Coupling: WARN (2 flags have interdependencies)
   Rollout config: PASS (active rollouts correctly configured)

   Issues:
     1. payments.stripe-v2 -- no test for flag-off fallback path
     2. search.new-algorithm + search.reranking -- coupled flags
     3. checkout.express -- missing kill switch test
   ```

---

### Phase 4: LIFECYCLE -- Audit Stale Flags and Plan Cleanup

1. **Identify stale flags.** Flag candidates for removal:
   - Release flags that have been at 100% for more than 30 days
   - Experiment flags past their declared end date
   - Flags with no evaluation in the last 90 days (from provider analytics)
   - Flags whose feature has been fully adopted (no rollback expected)

2. **Generate cleanup plan.** For each stale flag:
   - List all code locations that reference the flag
   - Identify which branch to keep (the flag-on path for graduated features)
   - Estimate lines of code to remove (dead branch cleanup)
   - Determine if any tests need updating after flag removal
   - Assign a cleanup owner and deadline

3. **Assess technical debt.** Calculate flag burden:
   - Total active flags per service (recommend under 20)
   - Ratio of temporary to permanent flags
   - Average flag age for temporary flags
   - Code complexity added by flag branching (conditional paths)
   - Estimate of test matrix growth due to flags

4. **Recommend lifecycle policies.** Design governance:
   - Maximum lifetime for release flags (e.g., 90 days)
   - Maximum lifetime for experiment flags (e.g., 30 days)
   - Required metadata: owner, creation date, expiry date, cleanup ticket
   - Automated alerting when flags exceed their lifetime
   - Monthly flag review cadence with the team

5. **Generate lifecycle report:**

   ```
   Flag Lifecycle Report:

   Active flags: 23
   Stale flags: 6 (candidates for removal)
   Technical debt: ~340 lines of dead code across 6 stale flags

   Cleanup plan:
     1. payments.old-checkout -- at 100% for 45 days, 4 files, ~80 LOC to remove
     2. search.v1-algorithm -- experiment ended 60 days ago, 2 files, ~40 LOC
     3. onboarding.legacy-flow -- at 100% for 90 days, 6 files, ~120 LOC
     4. notifications.email-v1 -- unused for 120 days, 3 files, ~50 LOC
     5. dashboard.old-charts -- at 100% for 35 days, 2 files, ~30 LOC
     6. auth.password-reset-v1 -- at 100% for 60 days, 1 file, ~20 LOC

   Recommendations:
     - Adopt 90-day maximum lifetime policy for release flags
     - Add flag expiry dates to LaunchDarkly metadata
     - Schedule monthly flag cleanup sprint
     - Target: reduce active flag count from 23 to 17
   ```

---

## Harness Integration

- **`harness skill run harness-feature-flags`** -- Primary invocation for flag analysis and lifecycle management.
- **`harness validate`** -- Run after flag cleanup to verify project health.
- **`harness check-deps`** -- Verify flag provider SDK dependencies are installed.
- **`emit_interaction`** -- Present lifecycle report and gather decisions on cleanup priorities.

## Success Criteria

- All feature flags in the project are discovered and cataloged
- Flags are categorized by type (release, experiment, ops, permission)
- Stale flags are identified with specific cleanup plans
- Test coverage for both flag-on and flag-off paths is verified
- Rollout configuration is validated for active flags
- Lifecycle policies are recommended with enforcement mechanisms

## Rationalizations to Reject

| Rationalization                                                            | Why It Is Wrong                                                                                                                        |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| "This release flag has been at 100% for a while, but removing it is risky" | Release flags at 100% for more than 30 days are stale candidates. Every stale flag adds dead code branches and test matrix complexity. |
| "We only need to test the flag-on path since that is the path we ship"     | No flags without test coverage for both paths. The flag-off path IS the fallback when the flag provider is unreachable.                |
| "These two flags depend on each other, but they work fine together"        | No coupled flag dependencies is a blocking finding. Flags that require other flags creates combinatorial complexity.                   |
| "Setting the flag default to true makes the rollout easier"                | Every flag must default to safe (feature disabled). A default of true means a provider outage enables the feature for everyone.        |
| "We do not need a naming convention -- our flag count is small"            | Inconsistent naming becomes unmanageable as flag count grows. The skill flags inconsistency as a warning even at small scale.          |

## Examples

### Example: React SPA with LaunchDarkly

```
Phase 1: DETECT
  Provider: LaunchDarkly (React SDK v3.x)
  Flags defined: 15 (in LaunchDarkly dashboard)
  Flags in code: 13 (2 unused in dashboard)
  Categories: 8 release, 3 experiment, 2 ops

Phase 2: DESIGN
  Naming: WARN -- inconsistent (mix of camelCase and kebab-case)
  Recommended convention: team.feature.variant (kebab-case)
  Evaluation: Client-side (React SDK), 200ms init timeout
  Default values: WARN -- 2 flags default to true (should default to false)
  Rollout: checkout.express-pay at 25% (targeting premium users)

Phase 3: VALIDATE
  Test coverage: WARN -- 4 flags missing useFlags mock in component tests
  Consistency: PASS
  Coupling: PASS (no interdependent flags)
  Kill switch: PASS (all release flags can be instantly disabled)

Phase 4: LIFECYCLE
  Stale: 3 flags at 100% for 30+ days
  Cleanup estimate: 180 LOC across 8 files
  Recommendation: Remove search.v2-results (at 100% for 65 days, 3 components)
  Result: WARN -- 3 stale flags, 4 missing tests, 2 unsafe defaults
```

### Example: Spring Boot API with Custom Flag System

```
Phase 1: DETECT
  Provider: Custom implementation (FeatureFlagService.java)
  Flag store: PostgreSQL table (feature_flags)
  Flags defined: 28
  Flags in code: 24 (4 in database but unreferenced)
  Categories: 15 release, 6 experiment, 5 ops, 2 permission

Phase 2: DESIGN
  Architecture: Server-side evaluation with 60s cache refresh
  Concern: No SDK resilience -- database outage disables all flags
  Recommend: Add in-memory cache with stale-while-revalidate pattern
  Recommend: Add health check endpoint for flag service status
  Rollout: Using percentage field in database -- no segment targeting
  Recommend: Add user segment support for targeted rollouts

Phase 3: VALIDATE
  Test coverage: WARN -- @FeatureFlag annotation used but no test helper
    to toggle flags in test context (tests rely on database state)
  Consistency: WARN -- 4 database entries have no code references
  Coupling: FAIL -- 3 flags form a dependency chain
    (billing.new-pricing requires billing.tax-calc-v2 requires billing.currency-v3)
  Recommend: Consolidate into single billing.pricing-v3 flag

Phase 4: LIFECYCLE
  Stale: 8 flags enabled for all users for 60+ days
  Technical debt: ~520 LOC of dead code
  Flag count: 28 (above recommended 20 per service)
  Recommendation: Immediate cleanup sprint targeting 8 stale flags
  Recommendation: Enforce 60-day lifetime policy going forward
  Result: FAIL -- flag coupling detected, high stale flag count
```

## Gates

- **No flags without default values.** Every flag evaluation must specify a default value for provider outage scenarios. Missing defaults are blocking findings.
- **No stale release flags beyond policy limit.** Release flags that exceed their maximum lifetime (default 90 days at 100%) are blocking warnings. They must be cleaned up or have an explicit extension with documented justification.
- **No coupled flag dependencies.** Flags that require other flags to be in a specific state create combinatorial complexity and are fragile. Flag coupling is a blocking finding that requires consolidation.
- **No flags without test coverage for both paths.** Feature flags that lack tests for the off-path leave the fallback behavior unverified. Both paths must be tested.

## Escalation

- **When flag count exceeds 30 per service:** The flag system is becoming a maintenance burden. Recommend a flag audit sprint with a target of reducing to under 20. Prioritize removing the oldest release flags first.
- **When a flag provider outage affects production:** If the application does not handle provider unavailability gracefully, this is an architectural issue. Recommend implementing local caching, safe defaults, and a circuit breaker pattern before adding more flags.
- **When experiment results are inconclusive:** Do not extend the experiment indefinitely. Recommend increasing sample size (broader targeting), simplifying the variants, or declaring the experiment inconclusive and removing it.
- **When flag cleanup requires database migration:** Removing flags that are stored in a database may require migration scripts. Coordinate cleanup with a release cycle and ensure rollback is possible if the migration fails.