Migrate Data Between Tools
Migrate records from a source tool/format to a destination tool/format — CRM to CRM, spreadsheet to app, JSON export to a new API — with a field mapping, value transforms, and a reconciliation that proves nothing was lost or corrupted. Locks the source→destination field map, the transform/normalization rules, and the reconciliation criteria FIRST, then builds the extract-transform-load with a dry-run preview, then verifies record counts match, key fields round-trip, and a diff report flags any mismatch. Mapping-and-reconciliation-before-load is the discipline that makes a migration trustworthy rather than a one-shot leap of faith.
Steps
Entry step: scope. Each step names the specialist role it wants; the full working prompt is expandable.
- Scope the migrationplannerentry
field map, transforms, reconciliation criteria
Show working prompt
Define the migration contract before moving data. Step 1: profile the source (format, record count, fields, key) and the destination (target schema, required fields, the key/upsert semantics). Step 2: build the source→destination field map, including fields that split/merge/rename, and list destination required-fields with no source — each needs a default or makes the record invalid. Step 3: define per-field transforms (date/format normalization, enum remapping, unit conversion) and the validation rules a destination record must satisfy. Step 4: define the reconciliation criteria — counts match (in vs out vs rejected), a chosen key set round-trips, and key totals/sums match. Step 5: write an acceptance-criteria checklist ('every source record is loaded or explicitly rejected with a reason', 'field map applied, no unmapped data dropped silently', 'transforms produce valid destination values', 'counts reconcile', 'a dry-run preview exists before any write', 'a diff report lists every mismatch'). Call write_task_note with the map + transforms + reconciliation + checklist and write notes/scope.md. No load yet. - Build the migrationdeveloper
extract, transform, dry-run, load, reconcile
Show working prompt
Implement the migration to the locked map. Step 1: extract all source records. Step 2: transform each record through the field map and transforms into a destination-shaped object; validate it, and on failure route it to a rejected list with the reason rather than dropping it. Step 3: produce a DRY-RUN preview first — the transformed records and a summary (to-load / rejected counts) — and write it before any destination write. Step 4: load into the destination using the upsert key so a re-run is idempotent (no duplicates). Step 5: run reconciliation — produce a diff report comparing source vs destination on counts and the chosen key fields, listing every mismatch — and write the migration module plus the diff report. Call write_task_note with the module path, the diff-report path, and the loaded/rejected counts.
- Verify the migrationreviewer
counts reconcile, fields round-trip, diff clean
Show working prompt
Verify the migration against scope. Step 1: confirm the module runs and produces both a dry-run preview and a diff report. Step 2: confirm coverage — every source record is either loaded or in the rejected list with a reason; loaded + rejected equals the source count. Step 3: confirm the field map applied with no silent data loss, and that destination records pass the validation rules. Step 4: confirm reconciliation — counts match and the chosen key fields round-trip (spot-check 5 records source↔destination), and the diff report has no unexplained mismatches. Step 5: confirm idempotency — a re-run via the upsert key creates no duplicates. Write PASS/FAIL per criterion to notes/verify.md and task notes; on failure, name the records and loop back to build.
- Evaluatereviewer
Grade the deliverable against every acceptance criterion. All pass → finish; any fail → loop back and fix the gap.
Show working prompt
Run migrate.py in dry-run and load modes (or trace it) and verify every criterion from notes/scope.md: a dry-run preview precedes any write, every source record is loaded or rejected-with-reason (loaded+rejected = source count), the field map applied with no silent loss, destination records validate, counts reconcile and key fields round-trip (spot-check 5), the diff report has no unexplained mismatches, and a re-run via the upsert key creates no duplicates. Write PASS/FAIL per criterion; on any failure, name the records and loop back to build. Then route — this is the whole point of the loop: - **Every criterion PASSES →** call `advance_task_step({ ref, stepId: "evaluate", next: "finish" })`. - **Any criterion FAILS →** write the specific gaps to notes, then call `advance_task_step({ ref, stepId: "evaluate", next: "build" })` to loop back. The builder fixes exactly those gaps. Never route to `finish` while any criterion is unmet. The build phase's completion gate already blocked a grossly-incomplete deliverable; your job is the judgment an automated check cannot make (does it actually work, read well, look right). After ~3 unproductive loops, stop and report DONE_WITH_CONCERNS so the user can step in. - Finishdeveloper
All acceptance criteria met. Stamp a short summary and report DONE.
Show working prompt
Every acceptance criterion passed. Write a one-paragraph DONE summary to task notes via `write_task_note`: what was built, the deliverable path(s), and a one-line confirmation that each criterion is met. Then report DONE.
Triggers
Phrases that suggest this craftbook to a crew.
- migrate data between tools
- move my CRM data
- import an export into another app
- convert and load records
- migrate a spreadsheet to an app
Source
View this craftbook on GitHub · MIT license