← All craftbooks

A/B Test Readout

Data and analysisevalv1.0.0released 2026-06-05workflow: build-loop

Steps

Entry step: hypothesis-scope. Each step names the specialist role it wants; the full working prompt is expandable.

1. Scope the hypothesisdata-analystentry

   lock the hypothesis, primary metric, and decision rule before looking

   Show working prompt

   Define the experiment's contract BEFORE computing anything, to prevent metric-shopping. Step 1: state the hypothesis ('variant B increases checkout conversion vs control'). Step 2: name the ONE primary metric the decision rests on and how it's defined (conversions / exposures). Step 3: list guardrail metrics that must not regress. Step 4: state the decision rule up front (e.g. 'ship if lift is positive and significant at 95% and no guardrail drops > 2%'). Step 5: write an acceptance-criteria checklist ('reports lift with a confidence interval', 'reports a significance test for the primary metric', 'checks sample size / power', 'gives a clear ship/no-ship tied to the decision rule', 'flags any guardrail regression'). Call write_task_note and write the hypothesis + metric + decision rule + checklist to notes/hypothesis-scope.md.

2. Analyze the experimentdata-analyst

   compute lift, significance, and confidence intervals

   Show working prompt

   Compute the statistics for the primary metric and guardrails. Step 1: for control and variant, report exposures, conversions, and the rate. Step 2: compute absolute and relative lift. Step 3: run the appropriate significance test (two-proportion z-test for a rate; report the p-value) and a confidence interval on the lift. Step 4: assess whether the sample size is adequate (was the experiment powered, or is it underpowered/peeking?). Step 5: compute the guardrail metrics and flag any regression. Step 6: apply the decision rule to reach a tentative verdict. Write the computed numbers to data/stats.json (rates, lift, p-value, CI, guardrails) and call write_task_note with the figures and the tentative call.

3. Write the readoutcopywriter

   produce a plain-language ship/no-ship recommendation

   Show working prompt

   Write a decision-ready readout from the computed stats — no new numbers. Step 1: lead with the recommendation (Ship / Don't ship / Inconclusive) in the first sentence, tied to the decision rule. Step 2: state the lift with its confidence interval and whether it cleared significance, in plain language ('+8% conversion, 95% CI [3%, 13%], significant at p=0.01'). Step 3: note the guardrail check result. Step 4: caveat honestly — underpowered, short runtime, novelty effect, or a result too good to be true. Step 5: end with the concrete next action. Write the readout to readout.md and call write_task_note with the recommendation. On a loop-back, fix only the named gaps.

4. Evaluatereviewer

   Grade the deliverable against every acceptance criterion. All pass → finish; any fail → loop back and fix the gap.

   Show working prompt

   Read readout.md and check it against the scope. Verify: it leads with a clear ship/no-ship recommendation tied to the stated decision rule; it reports lift WITH a confidence interval; it reports a significance test for the primary metric; it addresses sample size / power; and it flags any guardrail regression. Re-check the headline lift and significance call against data/stats.json. Confirm the verdict actually follows from the decision rule (not metric-shopped). Write PASS/FAIL per criterion; any FAIL loops back to write.
   
   Then route — this is the whole point of the loop:
   
   - **Every criterion PASSES →** call `advance_task_step({ ref, stepId: "evaluate", next: "finish" })`.
   - **Any criterion FAILS →** write the specific gaps to notes, then call `advance_task_step({ ref, stepId: "evaluate", next: "write" })` to loop back. The builder fixes exactly those gaps.
   
   Never route to `finish` while any criterion is unmet. The build phase's completion gate already blocked a grossly-incomplete deliverable; your job is the judgment an automated check cannot make (does it actually work, read well, look right). After ~3 unproductive loops, stop and report DONE_WITH_CONCERNS so the user can step in.

5. Finishdeveloper

   All acceptance criteria met. Stamp a short summary and report DONE.

   Show working prompt

   Every acceptance criterion passed. Write a one-paragraph DONE summary to task notes via `write_task_note`: what was built, the deliverable path(s), and a one-line confirmation that each criterion is met. Then report DONE.

Triggers

Phrases that suggest this craftbook to a crew.

• analyze an ab test
• experiment readout
• is this result significant
• ab test results
• ship or no ship from a test

Source

View this craftbook on GitHub · MIT license