Experiment

---
name: experiment
description: "Design, run, and evaluate a controlled A/B test (split test) on your ad accounts with evidence discipline — one variable, a falsifiable hypothesis, a fixed window, and a per-variant outcome verdict. Use when the user asks to run an A/B test, split test, or experiment, to 'test whether X beats Y', to validate a creative-refresh or learning hunch properly, to validate a breakdown exclusion or reallocation hypothesis before acting on it, or requests A/Bテスト / スプリットテスト設計 / 実験を回したい / どちらが勝ったか評価して / 仮説を検証したい. Forces a designed experiment instead of an ad-hoc change, records the baseline in action_log, and forbids peeking-based decisions before the window closes."
metadata:
  version: 0.21.3
---

# Experiment

> PREREQUISITE: Read `../_mureo-shared/SKILL.md` for auth, security rules, output format, and **Tool Selection** (Read/Write on Code, `mureo_strategy_*` / `mureo_state_*` MCP on Desktop / Cowork).

Turn an ad-hoc change into a **designed experiment**. Most "we tried X and CPA moved" claims are noise — one variable was never isolated, the window was never fixed, and the decision was made by peeking. This skill enforces the discipline: a falsifiable hypothesis, exactly **one** variable, a success metric tied to a Goal, a pre-committed duration and sample floor, no peeking, and a per-variant verdict where **inconclusive is a valid outcome**.

Read `../_mureo-learning/SKILL.md` first — its **Observation Windows** and **Minimum Sample Sizes** tables set the duration and sample floor this skill commits to, and its evidence lifecycle is the rulebook for the no-peeking rule below.

## Prerequisites
- STRATEGY.md and STATE.json must exist (run the `onboard` skill first)

## Steps

**Before you start**: Run the **Diagnostic preamble** from ../_mureo-shared/SKILL.md — load learning insights (mureo_learning_insights_get) and consult advisors (mureo_consult_advisor) before drawing conclusions.

0. **Establish today**: call `mureo_state_get` **first, on every host** (including Claude Code, where you would otherwise `Read` the file) and take `server_now` from its response — ISO 8601 with UTC offset, e.g. `2026-07-28T10:12:33+09:00`. Its date is the **only source of the current date** for this run: the designed window's `start_time` / `end_time`, the `observation_due` you commit to in step 7, and the "has the window closed?" gate in step 9 are all measured from it. A stale date is not cosmetic here — it is how a test gets evaluated early, which is precisely the peeking failure step 8 forbids. Do not shell out (this skill must run in Bash-less headless hosts) and do not read the date off STATE.json — `last_synced_at`, `reports.*.period` and `action_log` timestamps are **history**, never evidence of what day it is now. **Never write `server_now` into STATE.json**: it is a response field, and a persisted copy becomes tomorrow's stale "today".

1. **Hypothesis intake — force a falsifiable statement**. Take the hypothesis from the operator, from a `/creative-refresh` finding, or from a VALIDATED learning insight, and rewrite it into a single testable sentence with three explicit parts: **variable** (the ONE thing that differs), **expected effect** (direction + rough magnitude), and **success metric** (the number that decides it). Reject vague inputs ("make it better") — push back until it is falsifiable, e.g. *"Video creative (variable) will lower CPA by ≥15% (effect) vs the current static image, measured on cost_per_conversion (metric)."* If two things would differ, split into two experiments or drop one.

2. **Design — one variable, tied to a Goal**. Confirm **exactly one variable** changes between variants (creative, headline, audience, placement, bid — not several at once; confounded designs are unattributable). Set the **variant count** (2 is the default; more variants need proportionally more sample and duration). Bind the **success metric** to a specific STRATEGY.md Goal so the winner means something (a CTR win that doesn't move the Goal's CPA/CV is not a win). Derive the **minimum duration and sample floor** from `../_mureo-learning/SKILL.md`: creative/targeting changes need a 14-day window; the metric's sample floor (CPA: ≥30 conversions/variant; CTR: ≥1,000 impressions/variant; CVR: ≥200 clicks/variant) must be *reachable within the window at current traffic* — if it is not, say so and either extend the window or advise against running (an underpowered test wastes spend).

3. **Discover platforms**. Identify configured platforms from STATE.json `platforms`. Include any **hosted official-MCP connector** present in the session (e.g. TikTok, key `tiktok_ads`) and `mcp__mureo__<plugin>_*` plugin platforms — but note honestly that a true randomized split-test tool only exists for **Meta** below; on other platforms the experiment is a manual/approximate construction (see `../_mureo-shared/SKILL.md` → *Hosted-connector platforms* and *Plugin platforms*). mureo BYOD data lives in the workspace `byod/` directory and is reachable only through mureo MCP tools — do not look for raw CSVs in the project directory.

4. **Setup — Meta (native split test)**. Meta has a real experiment engine. The variants are **ad sets** — the split-test tool references pre-existing ad sets and does **not** create them, so first make sure each variant exists as its own ad set (identical except for the one variable) via `meta_ads_ad_sets_list` / `meta_ads_ad_sets_create`. In BYOD mode the list response is wrapped as `{"source": "byod_import", "as_of": <import time>, "data": [...]}` — take the rows from `data` and treat them as of `as_of`, not as live. Then, after the approval gate (step 6), create the test with `meta_ads_split_tests_create`:
   - `cells`: one `{name, adsets:[ad_set_id]}` per variant; Meta splits traffic evenly.
   - `objectives`: `[{type: COST_PER_RESULT | CONVERSIONS | REACH | CPC | CPM}]` — pick the type matching the Goal metric from step 2.
   - `start_time` (must be in the future — i.e. after `server_now`) and `end_time` — honor the designed window; Meta requires **≥4 days** between them.
   - `confidence_level` (80–99, default 95). Do not lower it just to force a faster verdict.
   Inspect a running test with `meta_ads_split_tests_list` / `meta_ads_split_tests_get`. If mureo's Meta tools are unavailable (`MUREO_DISABLE_META_ADS=1` after `mureo providers add meta-ads-official`), fall back to the official `meta-ads-official` hosted MCP for whatever experiment surface it exposes, and self-apply the guardrail/confirm rules yourself (mureo's PolicyGate cannot see that call).

5. **Setup — Google Ads (no split-test tool — honest manual paths)**. mureo ships **no Google Ads campaign-experiment / drafts-and-experiments tool** — there is no `google_ads_experiment*` in the tool surface, so do **not** invent one. Offer the operator the two honest alternatives and require **explicit acknowledgment of their caveats** before proceeding:
   - **(a) RSA asset A/B (in-ad, lowest-risk).** Add the challenger headline/description alongside the control in the same Responsive Search Ad and let Google's asset rotation gather per-asset data; read the split with `google_ads_rsa_assets_analyze` (per-asset `performance_label` + CTR) and `google_ads_rsa_assets_audit`. **Caveat**: this is asset-level rotation, **not** a randomized user split — Google serves assets non-uniformly (it favors likely winners), so it is directional, not a clean controlled test.
   - **(b) Duplicated-campaign split (campaign-level).** Duplicate the campaign (`google_ads_campaigns_create` + `google_ads_ads_create`), change only the one variable, split budget, and compare with `google_ads_ad_performance_report` / `google_ads_ad_performance_compare`. **Caveats**: the two campaigns compete in the same auction (they can cannibalize each other), Smart Bidding re-enters its learning phase on each, and there is no traffic-randomization guarantee — confounds are larger than a Meta split test. State these plainly and get the operator to accept them.
   If neither caveat is acceptable, recommend **not** running the Google Ads experiment rather than shipping a test the operator will over-trust.

6. **Approval gate before ANY write**. Setup steps 4–5 create/mutate platform state. Apply the *Confirm Before Write Operations* rule from `../_mureo-shared/SKILL.md`: show the full design (variants, variable, split, budget, window, success metric, sample floor) and get explicit **approval** before creating any ad set, split test, or duplicated campaign. Respect Operation Mode and `## Guardrails` (budget caps, `blocked_operations`); refuse a design that breaches them.

7. **Record the design in `action_log`**. After the test is created, append one `mureo_state_action_log_append` entry per variant (or one entry capturing all variants) with `metrics_at_action` = the **baseline per variant** (the control's current cost_per_conversion / conversions / ctr / impressions, and the challenger's starting zero/baseline), `command="/experiment"`, a `summary` naming the hypothesis, and `observation_due` = the **designed end date** (not a default 7/14). This is the commitment device that makes step 8's no-peeking rule enforceable and lets daily-check pick the evaluation up automatically. **Give every entry a display line**: `display_title` (at most 40 characters) and `display_summary` (at most 120) — what this action WAS, in the operator's words, as plain text with no markdown (`**bold**` reaches a person as asterisks). The dashboard row shows those two and stops there; the full `summary` is drill-down only, so keep writing it as fully as the next agent needs and let the display line be the short one. Over either bound the append is refused, never truncated.

8. **Interim rule — NO peeking-based decisions before the window closes**. While the test runs, you may *report* interim numbers if asked, but you must **refuse to call a winner or stop a variant early** on them. State why: repeatedly checking and acting on a running test inflates the false-positive rate (the sequential-testing / "peeking" problem) — a lead that looks significant mid-flight reverses far more often than the confidence level implies. If a variant is causing genuine harm (runaway spend, a policy issue), that is an *incident* to escalate, not an experiment result — handle it via `/rescue`, and mark the experiment confounded.

9. **Evaluation — only when the window has closed**. Once `observation_due` has passed — compare it against `server_now`'s date from step 0, never against another `action_log` timestamp:
   - **Check sample-size adequacy FIRST.** Pull per-variant results (`meta_ads_split_tests_get` for Meta; `google_ads_ad_performance_report` / `google_ads_rsa_assets_analyze` for the Google Ads paths). If any variant is below its sample floor from step 2, the result is **inconclusive-underpowered** — do not read the metric delta as signal. Say so and stop here (optionally extend the window if traffic allows).
   - **Per-variant verdict.** For each challenger, call `mureo_outcome_evaluate` with `before` = the variant baseline from step 7 and `after` = its final numbers — the deterministic verdict avoids eyeballing. Cross-read Meta's own `winner_cell_id` / `confidence_interval` from `meta_ads_split_tests_get` where present.
   - **Declare one of three outcomes**, honestly: **winner** (challenger beats control on the success metric, above noise, sample-adequate), **no-difference** (within the noise band — a real and useful result: it means the variable doesn't matter here, stop spending attention on it), or **inconclusive** (underpowered or confounded). Inconclusive is a valid outcome — never manufacture a winner to close the ticket.

10. **Close out**. Apply the outcome (**approval gate**): promote the winner (shift budget to it / adopt the winning creative or copy), and **stop the losing variant** (`meta_ads_split_tests_end` if ending a Meta study early is warranted, or pause the losing ad set / ad). Show current-vs-proposed and confirm before each mutation.

11. **Offer to save the finding via `/learn`**. A closed experiment — winner, no-difference, or inconclusive-because-underpowered — is exactly the reusable know-how the knowledge base is for. Propose the generalized lesson (e.g. *"Video beat static image by 22% CPA for this Persona — prefer video-first creative briefs"*) and, on approval, save it through the **`/learn`** flow (`mureo learn add`, per that skill's conventions — do not duplicate its steps here). Never record account IDs or personal data.

12. **Persist the report summary** (best-effort): Call `mureo_state_report_set` with `report="experiment"` and a concise `summary` object so the read-only dashboard can render the experiment without re-running you. Follow this convention:
    - `generated_at`: ISO 8601 timestamp of this run — use `server_now`
    - `period`: the experiment window (start_time..end_time) and its status (running / evaluated)
    - `totals`: the account's headline figures, using the canonical metric vocabulary — `spend`, `conversions`, `cpa`, `ctr`, `clicks`, `impressions`. This is the block the dashboard renders **as figures**. **Raw numbers only**: `773957`, not `"¥773,957"`; `0.0466`, not `"4.66%"` — one of those keys carrying a string is refused, because it sits where the view reads a figure and renders as nothing. A key outside the vocabulary may ride along: it is stored, just not shown as a headline number. Omit `totals` entirely if this run gathered no account-level figures.
    - `kpis`: per-variant `{metric_at_baseline, metric_final, verdict}` and the success metric name — the breakdown, not the headline row
    - `flags`: notable items (e.g. `["underpowered_variant_B", "google_ads_manual_split_caveats_accepted"]`)
    - `narrative`: the outcome (winner / no-difference / inconclusive, with confidence) and what you propose next, **at most 400 characters** — the tool refuses a longer one rather than truncating it (a sentence cut in half is worse than a long one). Do not restate the figures and do not list the findings here: numbers belong in `totals`, findings in `flags`.

    **Reflect the FINAL state, and persist this LAST** — after every `action_log` entry and any winner-promotion / loser-stop you applied this run. This is best-effort: if `mureo_state_report_set` is unavailable (e.g. a pure file-mode host without the context MCP), skip it silently — the rest of this skill still works.

13. **Persist the display contract** (best-effort): Call `mureo_state_display_set` with what the DASHBOARD shows. This is a different audience from the summary above — STATE.json is written for the next agent, and this small bounded section is the only thing the operator's screen reads. Write it in the SAME pass as the report, from the SAME figures, and **reach no new verdict here**: every value below is one you already decided, and this step only renders it.
    - `source`: your own skill name — REQUIRED whenever you write any section, at most 24 characters. The contract is replaced wholesale by whoever writes it last, so this is what lets the card say whose answer it is showing. `generated_at` is stamped by the server — do not compute it.
    - `nav_message`: the ONE line the operator should act on today — where the test stands, e.g. `"Variant B wins on CPA — promote it and stop the control"`. One line, not a summary of the report.
    - `highlights`: at most 3 chips, and the `tone` must match the verdict you already gave — `good` for a target met or a clear win, `watch` for something to keep an eye on, `bad` for something deteriorating. Choose the three that matter; a fourth is refused and nothing is dropped for you. Map a finding's severity to a chip tone: action → bad / watch → watch / positive → good. info does NOT become a highlight — a neutral note would spend one of the 3 chips an action or a win needed, and it is still in the report for whoever wants it.
    - `breakdown.campaigns` / `.adgroups`: one row per entity you judged, carrying figures you ALREADY hold — `spend`, `mcpa` (the measured CPA) and `target_cpa` — plus `state` from the closed set (`target_met` / `improving` / `watch` / `worsening` / `no_data`) and a `note` of at most 40 characters. Use `campaigns` for the campaigns under test and `adgroups` for per-variant ad sets / ad groups; a design-only run with no results yet omits `breakdown` rather than inventing rows. **Omit a figure you do not have**: `0` states a perfect CPA rather than the absence of one, and `no_data` is the state for an entity with too little delivery to judge.
    - `proposals`: the recommendations you did NOT carry out, one entry each — `title` (what to do) plus `body` (why, in one line) plus `date` = `server_now`'s date as `YYYY-MM-DD`. Something you DID apply this run is the same entry with `status: "done"`. **Record each of these as a decision too** — `mureo_decision_record` with `status: "proposed"`, your reasoning in `rationale` and the figures you judged it on in `metrics`. This section is the screen and is replaced whole by the next skill that writes it; the decision record is the one that survives, and it is what stops the next session re-proposing something the operator already turned down.
    - `stated_values`: label + figure chips ONLY (`{"label": "…", "value": …}`) — the design and the result: `{"label": "variants", "value": 2}`, `{"label": "days run", "value": 14}`, `{"label": "CPA lift", "value": "-18%"}`. **No prose notes here.** A sentence lands in a numeric column and the tool refuses it; anything that needs a sentence stays in the report's `narrative` above, exactly where it goes today.

    The bounds, verbatim from the tool: The dashboard reads THIS section and nothing else — keep your reasoning where it already goes. Every bound below refuses the write rather than truncating it, because a sentence cut in half is worse than a long one. nav_message: one line, at most 80 characters. highlights: at most 3 items of {tone, text}, tone one of good/watch/bad, text at most 60 characters. proposals: {title, body, status, date}, title at most 30 and body at most 80 characters, status one of proposed/done. breakdown.campaigns / breakdown.adgroups: rows of {name, spend, mcpa, target_cpa, state, note} — the three figures are raw numbers, state is one of target_met/improving/watch/worsening/no_data, note at most 40 characters. stated_values: {label, value}, label at most 24 characters and value a raw number or a string of at most 12 characters — a sentence there is refused, because it lands in a numeric column. Do NOT write the KPI funnel or the daily chart: mureo computes both from the stored totals.

    **Over a bound the write is REFUSED — shorten and rewrite, do not re-send the same sentence trimmed.** Lead with the point and drop the connectives; a noun phrase is fine (`CPA 12% over target`, not `The CPA is currently running about 12% above the target we agreed`). Trimming one character at a time and calling again spends the run's context on a bound one rewrite would have met.

    **You may not be the first writer today.** ``display`` is REPLACED WHOLE and the last writer wins — there is no merge. Before you write it, read the current one (``mureo_state_get``). Of what another skill wrote TODAY, carry exactly one thing into your own write: its ``proposals`` that are still live — not yet done, and not contradicted by what you just found. Everything else you write from your own run alone, because a screen assembled from two runs shows a moment that never happened. And carry over NOTHING ELSE: never copy another skill's ``nav_message``, ``highlights``, ``breakdown`` or ``stated_values``, which would put its judgement under your name when you cannot vouch for it. Name yourself in ``source`` so the screen says whose answer it is.

    This is best-effort: if `mureo_state_display_set` is unavailable (e.g. a pure file-mode host without the context MCP), skip it silently — the rest of this skill still works.