← All craftbooks

Test Suite Backfill

Build: software and codeevalv1.0.0released 2026-06-05workflow: build-loop

Steps

Entry step: coverage-scope. Each step names the specialist role it wants; the full working prompt is expandable.

1. Coverage scopeplannerentry

   lock the surface, critical paths, edge cases, target

   Show working prompt

   Decide WHAT to test before writing tests — coverage that matters beats line-count chasing. Step 1: Map the public surface of the code under test and what each function is supposed to do. Step 2: Rank by risk: which paths are critical/complex/most likely to break, and which are trivial. Step 3: For the high-risk paths, enumerate the cases that must be covered: happy path, boundary values, empty/null inputs, error conditions, and any tricky branches. Step 4: Set the target (the specific behaviors covered, or a coverage threshold if one was requested). Step 5: Write a numbered acceptance-criteria checklist of 5-9 items (e.g. 'every public function has at least a happy-path test', 'each error condition is asserted', 'boundary/empty inputs are covered', 'tests would fail if the behavior regressed'). `write_task_note` the test plan + checklist and write the same to the produces path.

2. Write the testsdeveloper

   characterization + edge-case tests that catch regressions

   Show working prompt

   Write the test suite to the plan. Step 1: For each public function, write a happy-path test using clear arrange-act-assert structure with meaningful assertions (assert the actual value/shape, not just 'did not throw'). Step 2: Add the edge and error cases from the plan — boundaries, empty/null, and each error condition (assert it throws/returns the right error). Step 3: Where behavior is undocumented, write characterization tests that pin the CURRENT behavior so future changes are caught. Step 4: Run the suite and confirm it passes against the real code; a test that cannot fail is worthless, so sanity-check that each would fail if behavior changed. Write the tests to the produces path. On a loop-back, add only the missing cases the reviewer named. `write_task_note` the test file path and the case count.

3. Evaluatereviewer

   Grade the deliverable against every acceptance criterion. All pass → finish; any fail → loop back and fix the gap.

   Show working prompt

   Run the test suite. For EACH acceptance criterion: confirm every public function has a happy-path test with a real assertion, confirm each error condition and boundary/empty case is covered, confirm the tests pass against the current code, and judge whether the tests would actually catch a regression (not just smoke tests). Write PASS/FAIL per criterion; trivial assertions that cannot fail count as a FAIL of the coverage-that-matters criterion.
   
   Then route — this is the whole point of the loop:
   
   - **Every criterion PASSES →** call `advance_task_step({ ref, stepId: "evaluate", next: "finish" })`.
   - **Any criterion FAILS →** write the specific gaps to notes, then call `advance_task_step({ ref, stepId: "evaluate", next: "write-tests" })` to loop back. The builder fixes exactly those gaps.
   
   Never route to `finish` while any criterion is unmet. The build phase's completion gate already blocked a grossly-incomplete deliverable; your job is the judgment an automated check cannot make (does it actually work, read well, look right). After ~3 unproductive loops, stop and report DONE_WITH_CONCERNS so the user can step in.

4. Finishdeveloper

   All acceptance criteria met. Stamp a short summary and report DONE.

   Show working prompt

   Every acceptance criterion passed. Write a one-paragraph DONE summary to task notes via `write_task_note`: what was built, the deliverable path(s), and a one-line confirmation that each criterion is met. Then report DONE.

Triggers

Phrases that suggest this craftbook to a crew.

• add tests to this code
• backfill tests
• increase test coverage
• write tests for an untested module
• test this function

Source

View this craftbook on GitHub · MIT license