Severe Test Protocol

---
name: severe-test-protocol
description: Subject hypotheses, strategies, or beliefs to their strongest possible objections. A theory proves its worth not by accumulating confirmations but by surviving attempts to disprove it.
license: MIT
metadata:
  author: sethmblack
  version: 1.0.4953
repository: https://github.com/sethmblack/paks-skills
keywords:
- severe-test-protocol
- writing
---

# Severe Test Protocol

Subject hypotheses, strategies, or beliefs to their strongest possible objections. A theory proves its worth not by accumulating confirmations but by surviving attempts to disprove it.

---

## When to Use

- Validating a strategy before committing resources
- Testing the robustness of a belief or conclusion
- User asks "What would disprove this?" or "Play devil's advocate"
- Before making high-stakes decisions
- When consensus feels too easy or comfortable
- Countering confirmation bias in planning or analysis

---

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| hypothesis | Yes | The claim, strategy, or belief to test |
| domain | No | Context that informs what tests are relevant |
| constraints | No | Limits on what tests are feasible |

---

## Darwin's Standard

"If it could be demonstrated that any complex organ existed, which could not possibly have been formed by numerous, successive, slight modifications, my theory would absolutely break down. But I can find no such case."

Darwin didn't merely seek evidence supporting natural selection—he actively identified what would disprove it and then searched for those disproof conditions. This is the severe test: specify what would break your theory, then look hard for exactly that.

A hypothesis that cannot specify what would disprove it is not a scientific hypothesis. A strategy that cannot identify what would indicate failure is not a rigorous strategy. The severe test protocol forces this clarity.

---

## The Severe Test Framework

### Step 1: State the Hypothesis Clearly

Before testing, crystallize exactly what's being claimed:

**The claim:** [Precise statement of what's believed to be true]
**Key assumptions:** [What must be true for this to work]
**Predicted outcomes:** [What should we observe if this is correct]

**Clarity check:**
- Is this falsifiable? (Could evidence show it wrong?)
- Is it specific enough to test?
- What's the confidence level going in?

### Step 2: Identify Potential Falsifiers

What evidence or outcomes would show the hypothesis is wrong?

**Direct falsifiers:**
- What observation would directly contradict the claim?
- What experiment could produce disproof?
- What data pattern would be impossible if the hypothesis is true?

**Indirect falsifiers:**
- What necessary condition, if absent, would undermine the hypothesis?
- What prediction, if wrong, would challenge the foundation?
- What assumption, if false, would invalidate the reasoning?

**The "breaks down" list:**
- If [X] is true, my hypothesis breaks down
- If [Y] is observed, my hypothesis breaks down
- If [Z] occurs, my hypothesis breaks down

### Step 3: Seek the Falsifiers Actively

Don't wait for disconfirming evidence to appear—hunt for it:

**Evidence search:**
- Where would falsifying evidence exist if it existed?
- Who would have observed it?
- What data would reveal it?

**Steelman the opposition:**
- What's the strongest argument against this hypothesis?
- Who disagrees, and what's their best case?
- What do critics point to?

**Hardest cases:**
- What examples seem most likely to disprove this?
- What edge cases would stress the hypothesis?
- What situations does it handle least well?

### Step 4: Conduct the Tests

Run the actual tests against the hardest objections:

**For each potential falsifier:**
- State the test
- State what outcome would falsify vs. support
- Conduct the test or evaluate available evidence
- Record the result honestly

**Integrity requirements:**
- Pre-commit to what counts as falsification
- Don't move goalposts after seeing results
- Report negative results, not just positives

### Step 5: Evaluate and Update

Based on test results, update the hypothesis:

**If hypothesis survives:**
- It's more robust (but not proven—only unfalsified)
- Document the tests it passed
- Identify remaining untested vulnerabilities

**If hypothesis fails:**
- Identify which specific element failed
- Determine if modification can salvage it
- Consider alternative hypotheses

**If results are mixed:**
- Specify conditions under which it holds/fails
- Narrow the scope of the hypothesis
- Note boundary conditions

---

## Workflow

### Step 1: Gather and Review Inputs

Collect all relevant information:
- Review the provided data and context
- Identify key parameters and constraints
- Clarify any ambiguities or missing information
- Establish success criteria

### Step 2: Analyze the Situation

Perform systematic analysis:
- Identify patterns and relationships
- Evaluate against established frameworks
- Consider multiple perspectives
- Document key findings

### Step 3: Generate Recommendations

Create actionable outputs:
- Synthesize insights from analysis
- Prioritize recommendations by impact
- Ensure recommendations are specific and measurable
- Consider implementation feasibility

## Output Format

```markdown
## Severe Test Protocol: [Hypothesis]

### Hypothesis Statement
**Claim:** [Precise statement]
**Key assumptions:**
1. [Assumption 1]
2. [Assumption 2]
**Predictions if true:**
- [Prediction 1]
- [Prediction 2]

### Potential Falsifiers Identified

**Direct falsifiers:**
1. [What would directly disprove this]
2. [What observation would be incompatible]

**Indirect falsifiers:**
1. [What assumption failure would undermine this]
2. [What prediction failure would challenge this]

**"Breaks down if" list:**
- If [condition 1], hypothesis breaks down
- If [condition 2], hypothesis breaks down
- If [condition 3], hypothesis breaks down

### Severe Tests Conducted

**Test 1: [Name]**
- What's being tested: [Falsifier being examined]
- Test method: [How the test was conducted]
- Falsification criterion: [What would count as failure]
- Result: [Pass/Fail/Partial]
- Evidence: [What was observed]
- Assessment: [What this means for the hypothesis]

**Test 2: [Name]**
- ...

### Steelman Opposition
**Strongest argument against this hypothesis:**
[The best case against, presented fairly]

**How hypothesis responds:**
[Whether and how it survives this objection]

### Hardest Case Analysis
**Most challenging case for this hypothesis:**
[The example or scenario that most strains it]

**How hypothesis handles it:**
[Whether it explains this case or fails]

### Test Results Summary

| Test | Target | Result | Impact on Hypothesis |
|------|--------|--------|---------------------|
| [Test 1] | [What falsifier] | [Pass/Fail] | [How it affects confidence] |
| [Test 2] | [What falsifier] | [Pass/Fail] | [How it affects confidence] |

### Updated Assessment

**Hypothesis status:** [Supported/Weakened/Falsified/Needs modification]

**Confidence change:** [Higher/Lower/Same] because [reason]

**Surviving version:** [If modified, what's the revised hypothesis]

**Remaining vulnerabilities:** [What tests haven't been conducted]

### Conclusion
[Summary judgment on hypothesis robustness]
```

---

## Types of Severe Tests

### Crucial Experiments
Tests where hypothesis makes a clear prediction that alternatives don't.
"If my theory is right, X should happen; if the alternative is right, Y should happen."

### Hardest Case Analysis
Test against the examples most likely to break the hypothesis.
"If this works even for [extreme case], it probably works generally."

### Necessary Condition Tests
Verify that conditions required by the hypothesis actually hold.
"My hypothesis requires X to be true. Is X true?"

### Prediction Tests
Check whether the hypothesis accurately predicts outcomes.
"If this is correct, we should observe Y. Do we observe Y?"

### Boundary Condition Tests
Find where the hypothesis stops working.
"Under what conditions does this break down?"

---

## Constraints

- Some hypotheses can't be fully tested with available evidence—acknowledge limits
- Passing tests doesn't prove truth, only failure to disprove
- Tests must be genuine attempts to falsify, not theater
- Moving goalposts after tests corrupts the process
- Some valuable hypotheses are unfalsifiable in principle (be clear when this is the case)

---

## Outputs

**Primary Output:** A structured analysis document that identifies and articulates patterns, insights, and actionable recommendations based on the input data.

**Format:**
```markdown
## Analysis: [Topic]

### Key Findings
- [Finding 1]
- [Finding 2]
- [Finding 3]

### Recommendations
1. [Action 1]
2. [Action 2]
3. [Action 3]
```

**Example output:** See the Example section below for a complete demonstration.

## Example

**Input:** "We believe our new product will succeed because it's 40% cheaper than competitors."

**Output:**



**Why this works:**

This example demonstrates the key principles of the skill in action. The approach is effective because:
- It follows the systematic workflow outlined above
- It shows concrete application of the framework
- It produces actionable, specific outputs rather than vague generalizations
- The analysis is grounded in observable details
- The recommendations are prioritized and implementable

**Alternative applications:**

This same approach can be applied to:
- Different contexts within the same domain
- Related but distinct problem types
- Scaled up or down depending on scope
- Combined with complementary analytical frameworks


## Severe Test Protocol: Price-Based Competitive Advantage

### Hypothesis Statement
**Claim:** Our new product will succeed in the market because it's priced 40% below competitors.

**Key assumptions:**
1. Price is a primary purchase driver in this market
2. Our product is functionally equivalent enough that price comparison is valid
3. We can maintain the price advantage sustainably
4. Competitors won't match our pricing
5. Customers can accurately compare prices

**Predictions if true:**
- Customers who evaluate our product should convert at higher rates than competitors
- Price-sensitive segments should adopt rapidly
- Competitor customers should switch when made aware of pricing

### Potential Falsifiers Identified

**Direct falsifiers:**
1. If price-sensitive buyers don't choose us, price isn't driving decisions
2. If competitors match pricing without losing margin, our advantage isn't real
3. If customers prefer competitors despite knowing our pricing, other factors dominate

**Indirect falsifiers:**
1. If our product is perceived as inferior, price won't compensate
2. If switching costs exceed savings, customers stay despite price
3. If the market isn't price-sensitive, the discount is irrelevant

**"Breaks down if" list:**
- If customers don't comparison shop, hypothesis breaks down
- If quality perception prevents consideration, hypothesis breaks down
- If competitors have cost structures allowing them to match, hypothesis breaks down
- If sales cycles ignore price until late stages, hypothesis breaks down
- If 40% discount signals "low quality" to buyers, hypothesis breaks down

### Severe Tests Conducted

**Test 1: Price Sensitivity Analysis**
- What's being tested: Is this market actually price-sensitive?
- Test method: Review industry surveys, analyze competitor pricing history, interview recent buyers
- Falsification criterion: If <30% of buyers rank price in top 3 factors, price isn't the driver
- Result: **Partial fail**
- Evidence: Industry survey shows price is #4 factor (after reliability, features, support). Only 23% rank price in top 3. However, in the SMB segment specifically, 51% rank price in top 3.
- Assessment: Hypothesis fails for general market but may hold for SMB segment

**Test 2: Perceived Equivalence Test**
- What's being tested: Do customers see our product as functionally equivalent?
- Test method: Blind feature comparison with target customers (n=15)
- Falsification criterion: If <70% see us as equivalent or better, equivalence assumption fails
- Result: **Fail**
- Evidence: 47% rated us equivalent, 20% better, 33% worse. Key gap: enterprise integrations
- Assessment: Price comparison isn't valid when product isn't seen as equivalent. Customers aren't comparing apples to apples.

**Test 3: Competitive Response Prediction**
- What's being tested: Will competitors maintain higher prices?
- Test method: Analyze competitor cost structures, gross margins, historical pricing responses
- Falsification criterion: If competitor gross margins allow 40% cut profitably, they can match
- Result: **Pass (marginal)**
- Evidence: Primary competitor has ~65% gross margin but high fixed costs. Matching our pricing would require 30% volume increase to maintain contribution. Not impossible but painful.
- Assessment: Short-term advantage likely holds; medium-term uncertain

**Test 4: Quality Signal Test**
- What's being tested: Does 40% discount signal low quality?
- Test method: A/B positioning test with target buyers
- Falsification criterion: If "much cheaper" positioning reduces consideration vs. "comparable" positioning
- Result: **Partial fail**
- Evidence: "40% less" headline reduced enterprise consideration by 24% but increased SMB consideration by 31%
- Assessment: Price positioning helps in SMB, hurts in enterprise

### Steelman Opposition

**Strongest argument against this hypothesis:**
"Price-based competition is the weakest form of advantage. It's immediately visible to competitors, easy to match (if painful), signals low value to sophisticated buyers, and attracts the least loyal customers. Your 40% advantage is temporary at best—if you succeed, competitors match; if you fail, the discount wasn't enough to compensate for other weaknesses. You're competing on the one dimension where you have no sustainable edge."

**How hypothesis responds:**
The hypothesis doesn't fully survive this objection. The 40% price advantage is a temporary wedge for market entry, not a sustainable competitive position. The response should be: "Price gets us in the door in price-sensitive segments; we must build other advantages quickly to retain customers when competitors respond."

### Hardest Case Analysis

**Most challenging case for this hypothesis:**
Enterprise buyers evaluating us against entrenched competitor with multi-year deployment.

**How hypothesis handles it:**
It doesn't. For enterprise, switching costs far exceed 40% price savings. A 3-year contract at current prices may still be cheaper than switching costs plus our lower prices. The hypothesis has essentially zero applicability to enterprise accounts with existing vendor relationships.

### Test Results Summary

| Test | Target | Result | Impact on Hypothesis |
|------|--------|--------|---------------------|
| Price Sensitivity | Is market price-driven? | Partial fail | Hypothesis valid only for SMB segment |
| Perceived Equivalence | Are we comparable? | Fail | Can't win on price if seen as inferior |
| Competitive Response | Will advantage persist? | Marginal pass | Short-term yes, medium-term uncertain |
| Quality Signal | Does cheap = bad? | Partial fail | Enterprise yes, SMB no |

### Updated Assessment

**Hypothesis status:** Weakened—requires significant narrowing

**Confidence change:** Lower. Initial confidence that price would drive broad success is not supported.

**Surviving version:** "Our 40% price advantage will help us acquire SMB customers who are not locked into existing vendors, provided we close perceived feature gaps and position as 'smart value' rather than 'cheap option.' This advantage is temporary (12-18 months) and must be converted to stickier value before competitors respond."

**Remaining vulnerabilities:**
- SMB segment size and profitability (is it big enough?)
- Speed of competitor response (unknown)
- Whether SMB success creates path to enterprise (unproven)

### Conclusion

The original hypothesis—that 40% lower pricing will drive product success—fails the severe test as stated. Price sensitivity is segment-dependent, perceived equivalence doesn't hold, and pricing signals quality concerns to enterprise buyers.

A narrower hypothesis survives: price advantage is valid for SMB new customer acquisition as a wedge strategy with 12-18 month window. This requires honest acknowledgment that it's a beachhead tactic, not a sustainable competitive position.

*"But I can find no such case."*—Unlike Darwin, we did find the cases that break this hypothesis. The honest response is to narrow the claim to what survives the tests.

---

## Integration

This skill is part of the **Charles Darwin** expert persona. Use it before committing to any significant strategy or belief. It pairs with:
- **patient-accumulation-method** to gather evidence needed for testing
- **variation-mapping** to identify which variants of the hypothesis might survive
- **selection-pressure-analysis** to understand what forces will test the hypothesis in practice