Source Quality Audit
Once a month, make the reading list earn its place: score each source on what it actually delivered — accuracy, originality, noise — and keep, demote, or drop with evidence, so the brief stays sharp as the feeds drift. Locks the audit criteria FIRST, then builds the per-source scorecard from the month's briefs, then writes the verdict table with the changes to make. Use for source audits, pruning the feed list, 'the brief feels noisy', monthly reading-list reviews, and deciding which feeds to drop.
Steps
Entry step: scope. Each step names the specialist role it wants; the full working prompt is expandable.
- Scope the auditreviewerentry
roster read, evidence base gathered, verdict rules locked
Show working prompt
Define the audit before scoring anything. Step 1: read sources.json from the workspace — every source on the roster — and gather the month's briefs from briefs/ as the evidence base. Step 2: lock the audit criteria — every source on the roster gets a verdict; verdicts come from a fixed set (keep, demote, drop); and every verdict must cite concrete evidence from the briefs: items it delivered, corrections it caused, or the silence of never being used. Step 3: agree what each verdict means in practice — keep means earning its place, demote means probation with a named reason, drop means off the list now. Step 4: write an acceptance-criteria checklist ('every source in sources.json appears in the verdict table', 'every verdict is keep, demote, or drop', 'every verdict cites at least one concrete example or explicitly notes the source went unused', 'the report ends with the changes to make'). Call write_task_note with the source count + criteria + checklist and write notes/scope.md. - Build the scorecardreviewer
per-source evidence: delivered, corrected, or silent
Show working prompt
Build the per-source scorecard from the month's evidence. Step 1: for each source in the roster, sweep the month's briefs and collect what it actually delivered — items attributed to it, stories it broke versus merely echoed, corrections or walk-backs it caused, and whether it appeared at all. Step 2: score each source in plain terms — accuracy (was it right), originality (was it first or just loud), and noise (how much of it was worth the owner's morning). Step 3: draft a verdict per source from the fixed set with the evidence beside it. Step 4: write the per-source scorecard with cited examples to notes/audit.md and call write_task_note with the draft verdict counts.
- Write the audit reportcopywriter
verdict table plus the changes to make
Show working prompt
Write the audit report the owner acts on. Step 1: open with the month in one honest paragraph — what the reading list did well and where the noise came from. Step 2: the verdict table — a Markdown table with columns Source, Verdict, and Why, one row per source, every verdict backed by its best evidence in a phrase. Step 3: the changes to make — which sources to drop or demote now, and what that leaves covered or exposed. Step 4: note anything worth adding to the roster to fill a gap the audit exposed. Keep it fair and specific — sources earn their place with evidence, not vibes. Write report.md and call write_task_note with the keep/demote/drop counts.
- Evaluatereviewer
Grade the deliverable against every acceptance criterion. All pass → finish; any fail → loop back and fix the gap.
Show working prompt
Open report.md, notes/audit.md, and the month's briefs and verify every criterion from notes/scope.md: every source in sources.json has a row in the verdict table, every verdict is keep, demote, or drop, every verdict cites at least one concrete example from the briefs (spot-check two against the brief files) or explicitly notes the source went unused, and the closing changes list matches the verdicts. Write PASS/FAIL per criterion; on any failure, name the gap and loop back to report. Then route — this is the whole point of the loop: - **Every criterion PASSES →** call `advance_task_step({ ref, stepId: "evaluate", next: "finish" })`. - **Any criterion FAILS →** write the specific gaps to notes, then call `advance_task_step({ ref, stepId: "evaluate", next: "report" })` to loop back. The builder fixes exactly those gaps. Never route to `finish` while any criterion is unmet. The build phase's completion gate already blocked a grossly-incomplete deliverable; your job is the judgment an automated check cannot make (does it actually work, read well, look right). After ~3 unproductive loops, stop and report DONE_WITH_CONCERNS so the user can step in. - Finishdeveloper
All acceptance criteria met. Stamp a short summary and report DONE.
Show working prompt
Every acceptance criterion passed. Write a one-paragraph DONE summary to task notes via `write_task_note`: what was built, the deliverable path(s), and a one-line confirmation that each criterion is met. Then report DONE.
Triggers
Phrases that suggest this craftbook to a crew.
- audit the sources
- the brief feels noisy
- review the reading list
- which feeds should we drop
- source quality audit
Source
View this craftbook on GitHub · MIT license