gilde

← All craftbooks

Test Suite Backfill

Build: software and codeevalv2.0.2released 2026-09-05workflow: build-loop

Add tests that actually bite to untested code: rank the untested behavior by breakage cost, write tests in the project's real layout that a wrong implementation would fail, and prove the suite green with real run receipts — under an enforced independent review.

Steps

Entry step: risk-map. Each step names the specialist role it wants; the full working prompt is expandable.

  1. Map the riskquality engineerentry

    Read the target modules and rank untested behavior by breakage cost; choose concrete targets. No test writing yet.

    Show working prompt
    Before writing any test, learn what is worth testing. Read the target module(s) named by the task with `read_file` — the whole file, not the signatures — and read the existing test layout (`list_dir` on the test folder, then the test files) to learn where tests live and how this project writes them. Rank the UNTESTED behavior by breakage cost: what would hurt users or downstream code most if it went silently wrong — boundary semantics, error contracts, arithmetic, state transitions — not what is easiest to cover. Then choose concrete targets: the specific functions and the specific behaviors of each you will pin, including the edge and error cases. Be honest about what you are deliberately leaving untested and why.

Write `{{workPath}}/risk-map.md` with exactly these sections: `## Modules` (each target module and what it does, with backtick paths), `## Risks ranked` (the untested behaviors ordered by breakage cost, with the cost stated), `## Chosen targets` (the tests you will write, each naming the behavior it pins). Every path you cite must be real.

Observable handoff: write the completed result to `{{workPath}}/risk-map.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections.
  2. Write tests that can failsoftware engineer

    Add the tests in the project's real layout and style, asserting actual behavior — never tautologies.

    Show working prompt
    Read `{{workPath}}/risk-map.md` and write the chosen tests in the project's REAL test layout and style — the same folder, naming convention, and assertion idioms the existing tests use, added with `write_file`. There is no prescribed output path for the tests themselves; they live where this project's tests live. Every test must be able to FAIL: assert actual values, actual thrown errors, actual edge-case results. Never write tautologies — expecting true, asserting a mock returns what it was told, or asserting whatever the implementation currently returns without knowing why it is right. Backfill means tests, not source changes: do not modify the module under test. If a test exposes what you believe is a genuine bug, pin the current behavior with a characterization test or flag it honestly in your notes for the user — never silently edit the source to match your expectation. Run the suite with `run_package_script` (`test`) as you go and make each test pass for the right reason; if the command needs first-use approval, say so and wait rather than guessing.

Then write `{{workPath}}/test-notes.md` with exactly these sections: `## Tests added` (the real path of every new test file, backtick path per line), `## Behaviors pinned` (each test and the exact behavior it asserts), `## How each test would catch a realistic break` (for each test, the plausible wrong implementation it would fail on). Before working, open `{{workPath}}/risk-map.md` with `read_artifact`.

Observable handoff: write the completed result to `{{workPath}}/test-notes.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections.
  3. Verify the suite greensoftware engineer

    Run the whole suite and record the real result, with a per-test statement of the regression it guards.

    Show working prompt
    Prove the backfill with a full, fresh run. Execute the whole suite with `run_package_script` (`test`) — every test, old and new, must pass. If anything fails, fix the TEST (or honestly report the module bug it exposed); do not touch the module under test to force green. Then account for the coverage: for each new test, one line stating the regression it now guards — what break it would catch that yesterday's suite would have missed. Compare against `{{workPath}}/risk-map.md`: any chosen target you did not cover gets an honest deferral with the reason, never silence.

Write `{{workPath}}/verification.md` with exactly these sections: `## Suite` (the exact command and the real full-suite result), `## Coverage summary` (per-test: the regression it guards; plus honest deferrals), `## Result` (green confirmed, or what remains open and why). Before working, open `{{workPath}}/risk-map.md` with `read_artifact`.

Observable handoff: write the completed result to `{{workPath}}/verification.md` in the project's artifacts drawer with `write_artifact`. Do not merely describe what the file would contain. Re-read it with `read_artifact` before finishing this phase and repair any incomplete sections.
  4. Evaluate the deliverabletest reviewer

    Independently grade the observable deliverable and route it to finish, repair, or user escalation.

    Show working prompt
    Review `{{workPath}}/verification.md`, `{{workPath}}/risk-map.md`, `{{workPath}}/test-notes.md` against every criterion below. Inspect the underlying evidence files named by the workflow; do not grade from the author's summary alone.

1. The new tests live in the project's real test layout and follow its existing naming and assertion style — open the test files and check.
2. Each test pins observable behavior a wrong implementation would fail: open the assertions — expecting true, asserting a mock returns what it was told, or restating the implementation back at itself does not pass.
3. The edge and error cases the risk map ranked are covered, or honestly deferred with a stated reason — reconcile risk-map.md's chosen targets against test-notes.md and verification.md.
4. The whole suite is green by real run receipt, judged from the recorded runs rather than the author's summary.
5. The module under test was not modified to make tests pass — the only changes are new or updated test files and the working notes.

Open `{{workPath}}/verification.md`, `{{workPath}}/risk-map.md`, `{{workPath}}/test-notes.md` with `read_artifact`. Write the evidence-backed review to `{{workPath}}/review.md` in the artifacts drawer with `write_artifact`.

List the findings as a markdown table with columns `| Severity | File | Line | Problem | Fix |` (severities: critical/major/minor/nit; empty table only on PASS). Give each criterion a PASS or FAIL with a concrete path, excerpt, measurement, or observed behavior. End with exactly `Verdict: PASS` or `Verdict: REVISE`. The gate ENFORCES the verdict: a well-formed REVISE is rejected and routed back to `repair` automatically, carrying your findings — so list every finding in the table with a concrete fix. On PASS, `advance_task_step` to `finish`. Never write PASS while a criterion is unmet.
  5. Finishproject lead

    All deterministic and reviewer criteria passed.

    Show working prompt
    The independent review passed. Read `{{workPath}}/review.md` with `read_artifact`, then use `write_task_note` to record a concise DONE summary with the final deliverable paths (`{{workPath}}/verification.md`, `{{workPath}}/risk-map.md`, `{{workPath}}/test-notes.md`) and the evidence that each acceptance criterion passed. Report DONE without starting new work.
  6. Repair the deliverablesoftware engineer

    Fix only the concrete gaps from the latest independent review.

    Show working prompt
    Read `{{workPath}}/review.md` with `read_artifact` and repair every failed criterion in `{{workPath}}/verification.md`, `{{workPath}}/risk-map.md`, `{{workPath}}/test-notes.md`. Make changes on each file's declared surface (`write_artifact` for artifact inputs, `write_file` for workspace inputs), not in task notes or a reply. Preserve evidence that already passed. Re-run or re-check anything the reviewer found unproven. Ensure `{{workPath}}/verification.md` is genuinely updated this turn so the repair is observable, then hand it back for independent evaluation.
  7. Escalate unresolved concernsproject lead

    The bounded repair loop ended without a defensible pass.

    Show working prompt
    The deliverable did not pass after 3 review rounds. Do not claim success. Read `{{workPath}}/review.md` with `read_artifact`, then use `write_task_note` to record DONE_WITH_CONCERNS: the unmet criteria, what was attempted, the affected paths, and the smallest user decision or missing input needed to continue.

Triggers

Phrases that suggest this craftbook to a crew.

Source

View this craftbook on GitHub · MIT license