Diagnosing Experiment Results

---
name: diagnosing-experiment-results
description: "Diagnoses bias, anomalies, and strange results on a PostHog experiment. Covers 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, surprises after mid-run edits, and qualitative follow-up via a variant-split survey.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping or A/A significance, finds PostHog numbers disagreeing with their SQL, reports surprises after mid-run edits, or wants qualitative feedback or a survey for an experiment.\nDO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle)."
---

# Diagnosing experiment results

This skill answers: **My PostHog experiment results look wrong, biased, or empty — what's going on?**

Match the user's complaint in the dispatch table, then read the matching reference file for the
diagnostic.

Each diagnostic in the reference files is tagged `[HIGH]`, `[MEDIUM]`, or `[LOW]` based on how
strongly it's verified — `[HIGH]` is verified directly in PostHog code, `[MEDIUM]` is partially or
team-source verified, `[LOW]` describes SDK/external behavior that wasn't verified here. Treat `[LOW]`
items as hypotheses to test, not facts to assert.

## Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the `finding-experiments` skill first to
resolve it to a concrete ID.

Call `experiment-get` and pull these fields. They are inputs for almost every diagnostic:

- `feature_flag.filters.multivariate.variants[].rollout_percentage` — the variant split
- `exposure_criteria.multiple_variant_handling` — defaults to `"exclude"` if absent
- `exposure_criteria.exposure_config.event` — unset means the default exposure event; read which one
  from `resolved_exposure_event` (`$feature_flag_called` or `$experiment_exposure` — resolved
  server-side, same properties either way)
- `exposure_criteria.filterTestAccounts` — defaults to `true`
- `feature_flag.active`, status (`draft` / `running` / `paused` / `exposure_frozen` / `stopped`), `start_date`, `end_date`
- `feature_flag.filters.groups[]` — for each group read `variant`, `properties`, and
  `rollout_percentage` (that group's rollout, % of the matched bucketing units — persons, or groups
  when `feature_flag.filters.aggregation_group_type_index` is set — that enter the experiment; each
  group has its own, so there is no single overall rollout unless the flag has one unconditional
  group). Any non-null `variant` is a forced-variant override on the matched cohort
  (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a
  variant-pinned group with broad/empty `properties` at high rollout, or no group left randomized
  (`variant: null`) / no release path to one arm — that starves the other variant (one arm gets ~0
  analyzable exposures). See `references/bias-and-skew.md`.
- `stats_config` — Bayesian (default) or Frequentist

## Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in
[references/diagnostic-snapshot.md](references/diagnostic-snapshot.md). Most diagnostics in this skill
can be confirmed or ruled out from that data without an interview.

## Step 2 — Match symptom to diagnostic

| User says...                                                                               | Diagnostic group                             |
| ------------------------------------------------------------------------------------------ | -------------------------------------------- |
| "Smaller variant looks biased" / banner says bias                                          | A — bias & skew                              |
| "Variant ratio doesn't match my split" / SRM warning                                       | A — bias & skew                              |
| "Why isn't it 50/50?" / "users in both groups"                                             | A — bias & skew                              |
| "Users in both control and test" / high `$multiple` %                                      | A — bias & skew                              |
| Multi-variant exposure on a server-rendered app                                            | A — bias & skew                              |
| Banner about feature-flag/experiment state mismatch                                        | A — bias & skew                              |
| "Migrating distinct_id" / "switching from anonymous to user_id" mid-run                    | A — bias & skew                              |
| Metric count is much smaller than exposures (e.g. 10× or 100× gap)                         | A — bias & skew (route here before D)        |
| "Experiment shows 0 / not enough data" / empty                                             | B — empty experiment                         |
| "Variant always undefined / false"                                                         | B — empty experiment                         |
| "$feature_flag_called fires but no exposures show up"                                      | B — empty experiment                         |
| "Experiment says running but exposures haven't moved in weeks/months"                      | B — empty experiment                         |
| "Significance keeps flipping as we run longer"                                             | C — interpretation traps                     |
| "Significance was declared, then it wasn't significant anymore"                            | C — interpretation traps                     |
| "30/16 split at 46 exposures, is this broken?"                                             | C — interpretation traps                     |
| "A/A test is showing significant results"                                                  | C — interpretation traps                     |
| "Many metrics — some significant, some not"                                                | C — interpretation traps                     |
| "Bayesian says 96% chance to win — should we ship?"                                        | C — interpretation traps                     |
| "Confidence intervals overlap — does that mean not significant?"                           | C — interpretation traps                     |
| "An external tool (significance calculator or AI agent) disagrees with PostHog"            | C — interpretation traps                     |
| "Should I ship? Primary is up but a secondary is down"                                     | C — interpretation traps                     |
| "PostHog numbers ≠ my SQL count"                                                           | D — numbers vs SQL                           |
| "Funnel says X% but my raw event count says Y"                                             | D — numbers vs SQL                           |
| "Sum of revenue looks wrong" / "breakdown shows 'none'"                                    | D — numbers vs SQL                           |
| "Recordings panel doesn't match the stats"                                                 | D — numbers vs SQL                           |
| "I applied a filter but the user count didn't change"                                      | D — numbers vs SQL                           |
| "I want to slice results by current person properties (as of now, not as of exposure)"     | D — numbers vs SQL                           |
| "Changed split / rollout / metric / criteria mid-run, now odd"                             | E — mid-run changes                          |
| "Ended/shipped — flag now flipped to 0/100 unexpectedly"                                   | E — mid-run changes                          |
| "Long-term metric moves opposite from primary"                                             | E — mid-run changes                          |
| "Retention metric counts users I didn't expect"                                            | E — mid-run changes                          |
| "Can't convert the feature flag back to a simple (boolean) flag after the experiment ends" | E — mid-run changes                          |
| "How do I restart an experiment with new variants?"                                        | E — mid-run changes                          |
| Metric line is rendered but the result block is empty / no chance-to-win or significance   | E — mid-run changes (E13 legacy methodology) |
| "Results won't load" / many metric rows show `data: null` (not a legacy experiment)        | Step 1.5 — diagnostic snapshot (null rows)   |
| "What do users think of the new flow?" / wants qualitative feedback on an experiment       | F — qualitative feedback                     |
| "Why did users prefer control?" / "what did they dislike about the test variant?"          | F — qualitative feedback                     |

If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes
— do not guess.

## Step 3 — Surface every diagnostic the evidence supports

After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic
that applies before recommending an action.

Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them
into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a
_systematic_ bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a _statistical_ pattern
(e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the
bundled advice.

Only list mechanisms that have a path to verification in the project state — config (from
`experiment-get`), snapshot data, activity log, or repo source. Config-derived mechanisms count: an
80/20 split with default `multiple_variant_handling="exclude"` is visible in `experiment-get` and is
therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean
variant ratio) is not.

## Diagnostic groups

### A — Bias & skew

Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are
showing up under multiple variants. Covers the uneven-split + Exclude interaction, SRM, identity
fragmentation, bootstrap × `/decide` mismatch, and flag/experiment state inconsistency.

→ See [references/bias-and-skew.md](references/bias-and-skew.md)

### B — Empty experiment / 0 exposures / "not enough data"

A frequent pain point. Covers SDK call (wrong evaluation method, `identify()` timing, dedup),
exposure capture (custom event missing variant property, required properties, ad-blockers), and
exposure-criteria match (test-account filter, eligibility ordering, events firing before exposure).

→ See [references/empty-experiment.md](references/empty-experiment.md)

### C — Significance / interpretation traps

Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple
comparisons, low-volume variance, peeking / early stopping. Includes the legacy stats issue (A/A tests
historically over-fired before the new Bayesian module) and how the win-probability methodology changed in
Jan 2025 (single test vs control, not control vs all variants).

→ See [references/interpretation.md](references/interpretation.md)

### D — Numbers don't match (PostHog vs the user's SQL / raw count)

The experiment page applies an exposure scope, `$multiple` exclusion, test-account filter, and date range
that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats),
breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user
confusion, and the recordings-panel-vs-stats divergence.

→ See [references/numbers-vs-sql.md](references/numbers-vs-sql.md)

### E — Surprises after mid-run changes (incl. lifecycle and retention quirks)

Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding
metrics mid-run is p-hacking; ship-variant can rewrite the flag in surprising ways; reset clears
results not the flag. Also covers retention-metric quirks (first-event-must-be-after-exposure design),
"matured users" filtering, and long-term vs short-term metric divergence.

→ See [references/mid-run-changes.md](references/mid-run-changes.md)

### F — Qualitative feedback: how the change landed, not how far the number moved

Groups A–E find mechanisms. F is for the question they can't reach: what the people in the experiment made
of the change. A short survey, shown when users finish the experimented flow, adds that qualitative half — a rating and
an optional comment, readable per variant — and works over MCP today.

It suits some experiments and not others. The gate is whether a user could describe the change without
being shown both versions: a reworked flow, layout, or process work; a threshold or ranking tweak
don't, however large its measured effect. Don't offer it while a mechanical diagnostic is still open — a
survey on top of a broken flag gate collects opinions about a feature half the audience never received —
and check whether an existing or recent survey already covers the window before proposing a new one.

Two things to get right before creating one: a survey shown to a single variant is itself a difference
between the variants, and a _running_ survey linked to the flag can generate exposure events.

→ See [references/qualitative-feedback.md](references/qualitative-feedback.md)

## Step 4 — Calibrate recommendations to experiment state

Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the
experiment's current state permits.

- **Draft** — config changes are free; recommend and apply.
- **Running** — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe?
  user-visible?) before recommending. See `configuring-experiment-rollout` and its reference file
  `references/changing-distribution-after-launch.md` for the mid-run rules.
- **Stopped / archived** — the experiment AND its feature flag represent the documented outcome of
  the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for
  the _next_ experiment, or (c) explaining what happened.

On a stopped or archived experiment, don't preemptively offer reversal of a state mutation
(ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?",
explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't
signal. Conditional offers like _"if this wasn't intended, you could…"_ or _"want me to revert it?"_
count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?",
"can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface
reversal mechanics.

Use consistent terminology: variant _split_ (between variants) is distinct from _rollout_ (overall %
entering); the _default exposure event_ (`resolved_exposure_event`) is distinct from a _custom exposure event_; the
_Exclude_ / _First seen_ options control multivariate handling, not exposure.

## Related skills

- **`configuring-experiment-analytics`** — fix the exposure or metric configuration the diagnosis points at
- **`configuring-experiment-rollout`** — split-change anti-patterns and safe rollout adjustments
- **`managing-experiment-lifecycle`** — reset, end, or restart when the experiment can't be salvaged in place
- **`analyzing-experiment-session-replays`** — when the numbers are fine but you need to see the behavior behind them