Ab Test Analysis

<!-- AUTO-GENERATED by export-skills.py — DO NOT EDIT -->
---
name: ab-test-analysis
description: Analyze A/B test results with statistical rigor — calculate significance, check guardrails, and make ship/extend/stop decisions. Use when evaluating experiment results or interpreting test data.
tags: [experimentation, ab-testing, analytics, data-driven]
---

# A/B Test Analysis

Analyze experiment results with statistical rigor and produce a clear **Ship / Investigate / Extend / Stop** recommendation.

This skill complements `ab-test-setup` (which handles experiment design). Use this skill when you have **results to analyze**.

---

## Purpose

Most A/B test interpretations are wrong — teams either call tests too early, ignore guardrail metrics, or ship on directional trends without statistical significance. This skill enforces disciplined analysis.

---

## When to Use

- An A/B test has completed its planned duration
- You have conversion data for control and variant groups
- Stakeholders are asking "did the test win?"
- You need to decide: ship, extend, or kill

## When NOT to Use

- Designing or setting up an experiment (use `ab-test-setup`)
- The test hasn't reached minimum sample size yet
- You're analyzing observational data (not a controlled experiment)

---

## Required Data (Ask If Missing)

| Field | Description |
| --- | --- |
| **Primary metric** | What the test is trying to improve (e.g., conversion rate) |
| **Control group** | Sample size (N) and conversions (C) for the control |
| **Variant group** | Sample size (N) and conversions (C) for the variant |
| **Test duration** | How long the test ran |
| **Planned duration** | How long it was designed to run |
| **Guardrail metrics** | Metrics that must not degrade (e.g., revenue, page load time) |
| **MDE** | Minimum Detectable Effect used in power calculation |

---

## Analysis Process

### Step 1: Validate the Setup

Before analyzing results, check:

- [ ] **Sample size adequate?** Compare actual N to planned N from power analysis
- [ ] **Duration sufficient?** Must cover at least 1–2 full business cycles (e.g., weekday + weekend)
- [ ] **SRM check?** Sample Ratio Mismatch — control and variant should have ~equal N (within 1%). If skewed, the test is invalid.
- [ ] **No novelty effects?** If you can, check early vs. late behavior. New UI elements get more clicks initially.

**If any check fails, the test results may be unreliable. Flag this before proceeding.**

### Step 2: Calculate Core Statistics

For conversion rate tests:

```text
Control conversion rate: p_c = C_control / N_control
Variant conversion rate: p_v = C_variant / N_variant
Relative lift: (p_v - p_c) / p_c × 100%

Pooled proportion: p = (C_control + C_variant) / (N_control + N_variant)
Standard error: SE = sqrt(p × (1-p) × (1/N_control + 1/N_variant))
Z-score: Z = (p_v - p_c) / SE
P-value: two-tailed from Z

95% Confidence Interval: (p_v - p_c) ± 1.96 × SE
```

### Step 3: Assess Significance

| Criterion | Threshold | Status |
| --- | --- | --- |
| Statistical significance | p-value < 0.05 | Pass / Fail |
| Practical significance | Lift > MDE | Pass / Fail |
| Confidence interval | Does CI exclude 0? | Pass / Fail |

**Both statistical AND practical significance are required to ship.**

### Step 4: Check Guardrail Metrics

For each guardrail metric:

| Guardrail | Control | Variant | Change | Status |
| --- | --- | --- | --- | --- |
| [metric name] | [value] | [value] | [+/- %] | OK / Warning / Degraded |

A guardrail is **degraded** if it shows a statistically significant negative change.

### Step 5: Make the Decision

Use this decision matrix:

| Primary Metric | Guardrails | Recommendation |
| --- | --- | --- |
| Significant positive | All OK | **Ship** — roll out to 100% |
| Significant positive | Some degraded | **Investigate** — understand trade-off before deciding |
| Not significant, positive trend | All OK | **Extend** — run longer if sample size was insufficient |
| Not significant, flat | All OK | **Stop** — no effect detected, free up the experiment slot |
| Significant negative | Any | **Don't Ship** — revert and learn from the result |

---

## Output Format

```markdown
# A/B Test Results: [Test Name]

## Summary

| Field | Value |
| --- | --- |
| Test name | [name] |
| Hypothesis | [We believed X would cause Y] |
| Primary metric | [metric name] |
| Duration | [start] — [end] ([N] days) |
| Decision | **Ship / Investigate / Extend / Stop / Don't Ship** |

---

## Results

| Group | Sample Size | Conversions | Rate |
| --- | --- | --- | --- |
| Control | [N] | [C] | [rate]% |
| Variant | [N] | [C] | [rate]% |

**Relative lift:** [+/- X.X%]
**P-value:** [value]
**95% CI:** [[lower]%, [upper]%]
**Statistically significant:** Yes / No
**Practically significant:** Yes / No (MDE was [X]%)

---

## Guardrail Metrics

| Metric | Control | Variant | Change | Status |
| --- | --- | --- | --- | --- |
| [metric] | [val] | [val] | [change] | OK / Warning |

---

## Recommendation

**Decision: [Ship / Investigate / Extend / Stop / Don't Ship]**

**Rationale:** [2–3 sentences explaining the decision]

**Next steps:**
1. [action]
2. [action]

---

## Learnings

- [What we learned from this test, regardless of outcome]
- [How this informs future experiments]
```

---

## Common Pitfalls

| Pitfall | Why It's Wrong | Correct Approach |
| --- | --- | --- |
| Peeking at results daily | Inflates false positive rate | Wait for planned duration and sample size |
| Calling it at p=0.06 | "Almost significant" isn't significant | Set the threshold before the test, stick to it |
| Ignoring guardrails | Winning on one metric while losing on another | Always check guardrails before shipping |
| Post-hoc segmentation | Finding "it worked for mobile users!" after the fact is data mining | Pre-register segments or treat as hypothesis for next test |
| Running too many variants | Each variant needs full sample size | Limit to 1–2 variants per test |
| Not learning from losses | "It didn't work" is not a learning | Document WHY it didn't work and what to try next |

---

## Anti-Patterns

| Avoid | Why | Instead |
| --- | --- | --- |
| "Directional win" | Not a statistical standard | Require p < 0.05 and lift > MDE |
| Shipping without guardrail check | May degrade critical metrics | Always check before shipping |
| Ending early because it "looks good" | Sequential testing bias | Run to planned duration |
| Not documenting learnings | Same failed experiments get repeated | Maintain an experiment log |

---

## References

- Kohavi, R., Tang, D., & Xu, Y. *Trustworthy Online Controlled Experiments* (2020)
- [Evan Miller's A/B Test Calculator](https://www.evanmiller.org/ab-testing/)
- [Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html)

<!-- Source: .faos/custom/skills/business/ab-test-analysis/SKILL.md -->