Fix a Flaky Test
Hunt down an intermittently failing test and make it deterministic: reproduce the flake with real recorded runs, isolate the nondeterminism mechanism, remove it at the real site — never by deleting, skipping, or retrying the test — and prove stability with three consecutive green runs.
Steps
Entry step: reproduce. Each step names the specialist role it wants; the full working prompt is expandable.
- Reproduce the flake with recorded runsdebugging engineerentry
Run the suite repeatedly until the intermittent failure is a recorded fact, and characterize when it appears.
Show working prompt
The flaky test to fix: {{scope}} Establish the flake as a recorded fact before theorizing. Find how this project runs its tests (`list_package_scripts`, then read the test layout). Run the suite with `run_package_script` (`test`) repeatedly — several runs if needed — until the intermittent failure is recorded, and keep the passing runs on record too: the pass/fail mix IS the evidence. Note the exact failure signature (test name, assertion, message) and the conditions it appears under — full suite versus alone, ordering, concurrency, timing, environment. If the command needs first-use approval, say so and wait rather than guessing. If the failure refuses to appear after many runs, record every run honestly and what you varied between them; do not fabricate a failing run. Write `{{workPath}}/flake-evidence.md` with exactly these sections: `## Symptom` (which test fails, and how often out of how many runs), `## Runs observed` (each run and its outcome, with the failing output quoted from a real run), `## Failure signature` (the exact assertion and message), `## Suspected class` (ordering, shared state, timing, or environment — with the observation that points there). Cite the real test and source files with backtick paths. Observable handoff: write the completed result to `{{workPath}}/flake-evidence.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections. - Isolate the mechanismroot-cause analyst
Pin the nondeterminism with a targeted experiment and name the defect site. No fixes yet.
Show working prompt
Read `{{workPath}}/flake-evidence.md`, then pin the MECHANISM with a targeted experiment, not a guess. Good experiments: run the failing test alone versus in the full suite; temporarily reorder or comment out neighboring tests (and restore them afterwards); repeat runs to confirm the pattern is what you think it is. Read the test and the code under test until you can name the defect site precisely — the shared module state, missing await, unseeded randomness, time or filesystem dependence, or leaked resource that makes the outcome depend on anything besides the code under test. Check siblings: which neighboring tests share the same hazard? Undo every temporary experiment edit before finishing, and make no fix in this phase. If the experiments stay inconclusive, record what each one showed and ruled out — an honest unknown beats a confident guess. Write `{{workPath}}/diagnosis.md` with exactly these sections: `## Mechanism` (the nondeterminism, named concretely), `## Experiment` (what you ran and what each variation showed), `## Defect site` (file and symbol, with backtick paths), `## Siblings checked` (neighboring tests that share or escape the hazard). Every path you cite must be real. Before working, open `{{workPath}}/flake-evidence.md` with `read_artifact`. Observable handoff: write the completed result to `{{workPath}}/diagnosis.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections. - Remove the nondeterminismsoftware engineer
Make the outcome deterministic at the real site — isolation, cleanup, awaits, seeding — never masking.
Show working prompt
Read `{{workPath}}/diagnosis.md` and remove the nondeterminism at the defect site it names. There is no prescribed output path for source changes — edit the actual files from the diagnosis with `read_file` plus `write_file`/`replace_in_file`. The right shapes: cleanup or isolation that runs for EVERY test (a beforeEach/afterEach reset, a fresh instance per test), an explicit await on work that was racing, seeded or injected randomness and time, releasing leaked resources. NEVER delete or skip the flaky test, never weaken its assertion, and never mask the flake with retries or widened timeouts — the test must still assert exactly what it asserted. A one-off reset inside the failing test alone is not cleanup; future tests inherit the same hazard, so fix it at the level every test shares. Cover the sibling hazards the diagnosis flagged. Re-run the suite with `run_package_script` (`test`) while you work. Write `{{workPath}}/fix-notes.md` with exactly these sections: `## Problem` (one paragraph naming the mechanism), `## Change` (what you altered and why the outcome can no longer depend on order, timing, or luck), `## Files touched` (backtick path per line), `## Regression coverage` (why the flake cannot silently return — what fails if the cleanup is removed), `## Risk` (anything unverified, honestly), `## How to verify` (the exact commands a person runs). Before working, open `{{workPath}}/diagnosis.md` with `read_artifact`. Observable handoff: write the completed result to `{{workPath}}/fix-notes.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections. - Prove stabilitysoftware engineer
Three consecutive green suite runs, read from real receipts — one green run means nothing for a flake.
Show working prompt
Prove the fix holds under repetition — one green run means nothing for a flake. Run the full suite with `run_package_script` (`test`) at least THREE times in a row; every run must be green, and the runs must be real receipts, not claims. If any run fails, the mechanism is not gone: record the failure honestly and route back through the evidence rather than re-rolling until you get lucky. Compare against `{{workPath}}/flake-evidence.md`: the once-flaky test must be present and passing in every run. Write `{{workPath}}/validation.md` with exactly these sections: `## Runs` (each run and its result, in order), `## Result` (stability confirmed with the receipts, or what remains unstable and why). Before working, open `{{workPath}}/flake-evidence.md` with `read_artifact`. Observable handoff: write the completed result to `{{workPath}}/validation.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections. - Evaluate the deliverabletest reliability reviewer
Independently grade the observable deliverable and route it to finish, repair, or user escalation.
Show working prompt
Review `{{workPath}}/validation.md`, `{{workPath}}/flake-evidence.md`, `{{workPath}}/diagnosis.md`, `{{workPath}}/fix-notes.md` against every criterion below. Inspect the underlying evidence files named by the workflow; do not grade from the author's summary alone. 1. flake-evidence.md records real runs with at least one genuine failure quoted from run output and names the conditions the failure appeared under — not a summary of a hunch. 2. diagnosis.md names a concrete nondeterminism mechanism and a defect site that exists in this codebase, supported by a targeted experiment whose causal story actually explains the observed pass/fail pattern. 3. The change removes the mechanism at that site — open the touched files and check it is isolation, cleanup, awaited work, or seeded inputs at the level every test shares, not a one-off patch inside the failing test. 4. No masking: the once-flaky test still exists and still asserts what it asserted — open the test file and compare its assertion against flake-evidence.md's failure signature; it gained no skip, retry loop, or widened timeout. 5. validation.md's three consecutive green runs are backed by real run receipts with the once-flaky test present and passing — judged from the recorded runs, not the author's claims. 6. fix-notes.md names every touched file with paths that exist, explains why the flake cannot silently return, and states residual risk honestly. Open `{{workPath}}/validation.md`, `{{workPath}}/flake-evidence.md`, `{{workPath}}/diagnosis.md`, `{{workPath}}/fix-notes.md` with `read_artifact`. Write the evidence-backed review to `{{workPath}}/review.md` in the artifacts drawer with `write_artifact`. List the findings as a markdown table with columns `| Severity | File | Line | Problem | Fix |` (severities: critical/major/minor/nit; empty table only on PASS). Give each criterion a PASS or FAIL with a concrete path, excerpt, measurement, or observed behavior. End with exactly `Verdict: PASS` or `Verdict: REVISE`. The gate ENFORCES the verdict: a well-formed REVISE is rejected and routed back to `repair` automatically, carrying your findings — so list every finding in the table with a concrete fix. On PASS, `advance_task_step` to `finish`. Never write PASS while a criterion is unmet. - Finishproject lead
All deterministic and reviewer criteria passed.
Show working prompt
The independent review passed. Read `{{workPath}}/review.md` with `read_artifact`, then use `write_task_note` to record a concise DONE summary with the final deliverable paths (`{{workPath}}/validation.md`, `{{workPath}}/flake-evidence.md`, `{{workPath}}/diagnosis.md`, `{{workPath}}/fix-notes.md`) and the evidence that each acceptance criterion passed. Report DONE without starting new work. - Repair the deliverablesoftware engineer
Fix only the concrete gaps from the latest independent review.
Show working prompt
Read `{{workPath}}/review.md` with `read_artifact` and repair every failed criterion in `{{workPath}}/validation.md`, `{{workPath}}/flake-evidence.md`, `{{workPath}}/diagnosis.md`, `{{workPath}}/fix-notes.md`. Make changes on each file's declared surface (`write_artifact` for artifact inputs, `write_file` for workspace inputs), not in task notes or a reply. Preserve evidence that already passed. Re-run or re-check anything the reviewer found unproven. Ensure `{{workPath}}/validation.md` is genuinely updated this turn so the repair is observable, then hand it back for independent evaluation. - Escalate unresolved concernsproject lead
The bounded repair loop ended without a defensible pass.
Show working prompt
The deliverable did not pass after 3 review rounds. Do not claim success. Read `{{workPath}}/review.md` with `read_artifact`, then use `write_task_note` to record DONE_WITH_CONCERNS: the unmet criteria, what was attempted, the affected paths, and the smallest user decision or missing input needed to continue.
Triggers
Phrases that suggest this craftbook to a crew.
- fix this flaky test
- deflake
- test fails intermittently
- fails in CI but passes locally
- intermittent test failure
Source
View this craftbook on GitHub · MIT license