← All craftbooks

Clean a Dataset

Data and analysisevalv1.2.1released 2026-07-16workflow: build-loop

Steps

Entry step: profile. Each step names the specialist role it wants; the full working prompt is expandable.

1. Profile the raw datadata-analystentry

   measure the data's shape, types, and defects before touching it

   Show working prompt

   Load the raw dataset and PROFILE it before deciding any fix. Step 1: report row count, column count, and the delimiter/encoding. Step 2: for EACH column report inferred type, % null/blank, # distinct values, and 3 sample values. Step 3: count exact-duplicate rows and flag candidate key columns. Step 4: list concrete defects you observe — mixed types in a column, inconsistent casing ('USA' vs 'usa'), stray whitespace, mojibake/encoding issues, impossible values (negative ages, dates in the future), and outliers (values > 3 stdev). Step 5: turn each defect into a planned, deterministic fix and write an acceptance-criteria checklist (e.g. 'no exact-duplicate rows', 'date column parses as ISO-8601', 'no nulls in key columns', 'cleaned row count >= 95% of raw unless dupes explain the drop'). Call write_task_note with the profile table + defects + fixes + checklist, and also write the same content to notes/profile.md. No cleaning code yet.

2. Clean the datadeveloper

   apply the planned deterministic transforms and emit the cleaned file

   Show working prompt

   Apply EXACTLY the fixes the profile planned — no new undocumented transforms. Step 1: read the raw file with the correct encoding/delimiter. Step 2: in order, drop exact duplicates, coerce each column to its target type, normalize whitespace and casing, standardize known value variants to a canonical form, and handle nulls per the plan (impute, drop, or leave with a reason). Step 3: keep a running tally of rows-in vs rows-out and WHY any row was dropped. Step 4: write the cleaned dataset to data/clean.csv with a header row and consistent quoting. Step 5: call write_task_note with the path, the rows-in/rows-out delta, and which checklist items the output now satisfies. Do not lose columns the profile expected to keep.
   
   Produce the output by EXECUTING your transform (prefer `derive_file({ script, outputPath })`, or write a script file and run it with `run_nodejs_script`); do not hand-type rows into the output file — hand-typed derived data loses records.

3. Evaluatereviewer

   Grade the deliverable against every acceptance criterion. All pass → finish; any fail → loop back and fix the gap.

   Show working prompt

   Validate that data/clean.csv is genuinely analysis-ready by re-profiling it. Step 1: confirm it parses as CSV, has a header row, and zero exact-duplicate rows. Step 2: confirm the key column(s) have no nulls and every column holds its declared type. Step 3: reconcile the row count — raw rows minus documented duplicates must equal the cleaned row count, and no column was dropped. Step 4: re-measure at least two of these numbers yourself rather than trusting the build note. Step 5: spot-check 5 rows against the raw source to confirm values were transformed, not corrupted. Write PASS/FAIL per acceptance criterion from the profile note; any FAIL loops back to clean.
   
   Then route — this is the whole point of the loop:
   
   - **Every criterion PASSES →** call `advance_task_step({ ref, stepId: "evaluate", next: "finish" })`.
   - **Any criterion FAILS →** write the specific gaps to notes, then call `advance_task_step({ ref, stepId: "evaluate", next: "clean" })` to loop back. The builder fixes exactly those gaps.
   
   Never route to `finish` while any criterion is unmet. The build phase's completion gate already blocked a grossly-incomplete deliverable; your job is the judgment an automated check cannot make (does it actually work, read well, look right). After ~3 unproductive loops, stop and report DONE_WITH_CONCERNS so the user can step in.

4. Finishdeveloper

   All acceptance criteria met. Stamp a short summary and report DONE.

   Show working prompt

   Every acceptance criterion passed. Write a one-paragraph DONE summary to task notes via `write_task_note`: what was built, the deliverable path(s), and a one-line confirmation that each criterion is met. Then report DONE.

Triggers

Phrases that suggest this craftbook to a crew.

• clean this dataset
• tidy a csv
• deduplicate data
• fix dirty data
• prep data for analysis

Source

View this craftbook on GitHub · MIT license