Audit one-shot migration tooling and remove it from the Lambda artifact #174

Status: done

Decision

Keep simple one-off scripts that read retained raw source data (Trello JSON,
CSV, workbooks, archives, Calendar, Newsletter, Sponsor exports, etc.) and write
it into DataOps after the final destination model is ready. They live outside
backend/src, are excluded from the Lambda artifact, and are not ongoing CI
test surfaces.

Delete permanent migration infrastructure: production-only migration seams,
planners, approvals, checkpoints/resume, reconciliation, rollback,
backfill/scrub compatibility guards, orphan management, and their tests.

Do not run any import as part of this issue.

Itemized deletion ledger

Each checkbox is handled independently: inspect dependencies, implement one
coherent deletion, run proportionate normal-product checks (never the deleted
migration suite), commit it, record the SHA/evidence here, then tick it.

• Remove the Trello/CSV one-off import's ongoing tests only:
  backend/tests/migrate-data.test.ts, its
  backend/tests/fixtures/trello-active-cards.json fixture, and
  backend/e2e/migration.spec.js. Keep backend/scripts/migrate-data.ts.
  Evidence: commit 7e88f24; retained script exists; no remaining test/config
  references; npm --prefix backend run typecheck passed.
• Remove the completed Task-title storage scrub:
  backend/scripts/scrub-task-titles.ts and
  backend/tests/scrub-task-titles.test.ts.
  Evidence: commit abe2181; no remaining scrub symbol/path references;
  npm --prefix backend run typecheck passed.
• Remove the completed Task-title display compatibility guard:
  stripTitleSuffix in frontend/src/core/workspace.js; make
  workTaskTitle read the current title shape directly and remove only tests
  dedicated to the leaked short-link suffix.
  Evidence: commit 4b4bc66; no suffix compatibility symbols/fixtures remain;
  npm --prefix frontend test passed (181 tests).
• Remove Sponsor CRM migrationSnapshot=true list behavior from
  backend/src/routes/sponsorCrm.ts and its migration-only assertions/callers;
  ordinary paginated CRM list behavior remains.
  Evidence: commit 0d65037; the route has no snapshot-mode reference;
  backend typecheck and the ordinary Sponsor CRM suite passed (14 tests).
• Remove Newsletter migrationSnapshot=true behavior from
  backend/src/routes/newsletterSlots.ts and its migration-only assertions;
  ordinary newsletter-slot list behavior and alert maintenance remain.
  Evidence: commit c5c98d7; no snapshot-mode reference remains in the route
  or its caller; backend typecheck and ordinary Newsletter suite passed (5
  tests before the embedded one-off importer case is removed separately).
• Remove the Calendar one-off import assertions/imports embedded in
  backend/tests/calendar.test.ts (the workbook parsing/formula/credential and
  write-through cases beginning with "imports only the two allowlisted
  sheets"); keep backend/scripts/import-calendar.ts and ordinary Calendar
  runtime/API/UI tests.
  Evidence: commit 5425f59; the retained script is no longer imported by
  tests; backend typecheck and the six remaining ordinary Calendar tests passed.
• Remove the Newsletter one-off import assertion/imports embedded in
  backend/tests/newsletter-slots.test.ts ("dry-runs offline and writes only
  synthetic allowlisted Newsletter fields through HTTP"); keep
  backend/scripts/import-newsletter-slots.ts and ordinary Newsletter runtime
  tests.
  Evidence: commit 74cc4b6; the script remains and no test imports it;
  backend typecheck and the remaining ordinary Newsletter suite passed (4
  tests).
• Replace the generalized Sponsor CRM framework
  (backend/src/sponsorCrmMigration/{api,artifacts,canonical,execution,manifest,planner,runner,types}.ts)
  and backend/tests/sponsor-crm-migration.test.ts with a straightforward
  script-local source-to-ordinary-CRM-API importer under backend/scripts;
  remove planner/approval/checkpoint/resume/reconcile/rollback/cleanup and
  destination-snapshot modes, while retaining the raw Sponsor source inputs.
  Evidence: commit 7abd906; all eight backend/src/sponsorCrmMigration
  modules and the 2,020-line framework suite are gone. The retained script
  validates the raw Sponsor JSON shape and posts only through ordinary CRM APIs
  behind --write, exact-origin confirmation, and an operator session. Backend
  typecheck and the ordinary Sponsor CRM suite passed (14 tests); no import ran.
• Simplify the bookkeeping document one-off importer by removing
  approval-plan/checkpoint/resume/reconcile/rollback/backfill orchestration from
  backend/scripts/import-bookkeeping-documents.ts; move only necessary raw
  archive/manifest parsing helpers out of
  backend/src/bookkeeping-document-import/ into a script-local directory;
  delete backend/tests/bookkeeping-document-import.test.ts and retain the
  source archive manifest/schema needed for a future manual import.
  Evidence: commit 93b9292; 1,558 generalized importer/test lines were
  removed, archive/manifest/types moved from runtime source to
  backend/scripts/bookkeeping-document-import, and the retained script now
  validates offline or performs one straight-line ordinary API import. No
  importer helper remains under backend/src; backend typecheck and build
  passed, including the runtime boundary check. No import or migration suite ran.
• Remove the finished bookkeeping document hash-claim backfill seam:
  /api/bookkeeping/documents/hash-claims/backfill,
  backfillDocumentHashClaims, documentHashClaimsReady, the
  DOCUMENT_HASH_MIGRATION#READY guard, and only its migration-specific test
  assertions. Ordinary document hash claims and atomic document upload remain.
  Evidence: commit dfea591; marker, readiness guard, endpoint, DB backfill,
  local-server setup, importer calls/flag, and backfill assertions are gone. No
  matching symbols remain; backend typecheck and 5 ordinary bookkeeping API
  tests passed. Migration suites were intentionally not run.
• Remove bookkeeping migration-run reconciliation/rollback:
  /api/bookkeeping/migration-runs/{runId}/{reconcile|rollback},
  listRunOwnedItems/listRunOwnedPage, rollback-only delete state/helpers,
  and the route's migration-only test cases. Ordinary bookkeeping document,
  link, report, and transaction behavior remains.
  Evidence: commit 1976e54; the two routes, run ownership fields, owned-item
  scans, rollback state/helpers, and 222 lines of migration-only API tests are
  gone. Idempotency-key ownership still protects ordinary atomic uploads.
  Backend typecheck, 16 ordinary bookkeeping/Sponsor Finance tests, and 6
  ordinary atomic bookkeeping tests passed. No migration suite ran.
• Remove the stale frontend bookkeeping runId upload ownership field;
  the backend no longer stores or checks migration-run ownership, and ordinary
  atomic upload ownership is the idempotency key.
  Evidence: commit 9d3d19f; the frontend now sends only the idempotency key
  for prepare/complete ownership. All 7 Finance surface tests passed.
• Remove importer-only POST /api/bookkeeping/transactions/resolve and its
  test probes; the retained one-off document script resolves transactions via
  the ordinary transaction list API.
  Evidence: commit 657ad32; the route and its probe are gone. Backend
  typecheck and all 6 ordinary atomic bookkeeping tests passed.
• Remove importer-only POST /api/bookkeeping/documents/hash-lookup and its
  test probes; ordinary duplicate detection remains inside document prepare.
  Evidence: commit 68b3e99; the production route/probes are gone and ordinary
  tests now exercise hash duplicate/retry/cleanup behavior through prepare,
  cancel, and document list. Backend typecheck, 5 bookkeeping API tests, and 6
  atomic document tests passed.
• Remove importer-only POST /api/bookkeeping/links/lookup and its test
  probes; ordinary deterministic/idempotent link creation remains.
  Evidence: commit e509e60; the route/probes are gone and the ordinary
  concurrency case verifies the single created link through the list API.
  Backend typecheck and all 6 atomic document tests passed.
• Remove the retired compatibility response at
  POST /api/bookkeeping/documents/upload and its assertion; the ordinary
  frontend and retained script already use prepare/upload/complete.
  Evidence: commit 9efcdad; the dead route and assertion are gone. Backend
  typecheck and all 5 ordinary bookkeeping API tests passed.
• Remove Sponsor Communications suppression-key migration/orphan machinery:
  /suppressions/migrate, /suppressions/orphans,
  /suppressions/orphans/{id}/reconcile, repository migration/orphan
  functions/record creation, private-archive orphan handling, dispatch entries,
  and the dedicated migration test case. Normal suppression enforcement and
  unknown-send-attempt reconciliation remain.
  Evidence: commit 16de943; all three routes, repository migration/orphan
  functions and record schema, private-archive handling, dispatch entries, UI
  controls, and the dedicated transaction case are gone. Backend typecheck,
  the 3 ordinary route-dispatch tests, and all 181 frontend tests passed.
  Normal suppression and live send-attempt reconciliation code remains; no
  migration test ran.
• Remove ongoing tests for the retained workbook one-off importer:
  backend/tests/import-bookkeeping.test.ts; keep
  backend/scripts/import-bookkeeping.ts.
  Evidence: commit aac51cf; retained script exists; backend typecheck passed.
• Remove the workbook import-only /api/bookkeeping/ingest production
  route, ingestion-secret/rate-limit support, and its cases/import-script
  coupling in backend/tests/api-bookkeeping.test.ts; simplify the retained
  backend/scripts/import-bookkeeping.ts to use the ordinary authenticated
  transaction-create API. Remove the now-dead BookkeepingIngestionSecret
  CloudFormation resource, its Backend IAM reference, and
  BOOKKEEPING_INGESTION_SECRET_NAME environment wiring from
  infra/template.full.yaml.
  Evidence: commit a8b337d; the route, auth bypass, secret lookup/rate limit,
  CloudFormation secret/IAM/env wiring, and route-specific tests are gone. The
  retained workbook script now posts to ordinary authenticated transaction
  creation. No ingestion symbols remain; backend typecheck, 5 ordinary
  bookkeeping API tests, SAM lint, and 35 focused infra tests passed.
• Rename the misleading infra test
  tests/infra/test_bookkeeping_document_migration.py to an ordinary
  bookkeeping-documents infrastructure test, because its assertions protect
  the live product bucket/IAM boundary rather than migration behavior.
  Evidence: commit aaf6ead; renamed to
  tests/infra/test_bookkeeping_documents.py; both infra assertions passed.
• Strengthen backend/scripts/verify-runtime-boundary.mjs so backend builds
  fail if importer/framework output (including sponsorCrmMigration or
  bookkeeping-document-import) reappears in the shipped runtime artifact.
  Evidence: commit 23cc914; the verifier rejects removed framework/importer
  directories, any shipped scripts directory, and importer inputs in the SAM
  esbuild manifest. Two focused boundary tests prove both directory and
  manifest failures; backend build passed with the strengthened verifier.
• Remove the now-empty frontend oversized-module migration baseline and
  simplify the architecture test to require zero modules over 1,000 lines.
  Evidence: commit e6a2ad6; the empty exception mechanism is gone and the
  test directly requires no oversized modules. Both architecture tests passed.
• Delete the stale tracked _docs/2026-08-13-session-handoff.md; it is an
  internal operational handoff that incorrectly says deleted migration
  machinery/scripts still exist and belongs outside this public repository.
  Evidence: commit cb10b04; the 166-line stale internal handoff is no longer
  tracked in the public repository.
• Remove the finished Google Doc-to-SOP migration map tooling:
  backend/scripts/build-google-doc-map.ts,
  backend/scripts/google-doc-sop-map.ts, and the
  build:google-doc-map/check:google-doc-map package commands. It generates a
  historical mapping of private source links rather than exporting raw source
  data, and does not belong in the product or this public repository.
  Evidence: commit a47851e; both migration-map scripts and both package
  commands are gone (873 lines). No references remain and backend typecheck
  passed.
• Remove stale migration-only package commands/copy and re-inventory tracked
  runtime files for migrationSnapshot, migration-runs, backfill, scrub,
  suppression-orphan, generalized planner/checkpoint/rollback, and migration
  test leftovers. Retained one-off commands use import naming.
  Evidence: commit 4918fdc; retained Trello/CSV and Sponsor scripts are now
  import-trello-csv.ts and import-sponsor-crm.ts, the package command is
  sponsor-crm:import, and Calendar docs use import terminology. Backend
  typecheck/build passed. Re-inventory found none of the listed runtime/test
  leftovers. The only matching tracked source terms are intentionally retained
  raw-script-local transformation/provenance, portable artifact provenance,
  live conversational summary checkpoints, the runtime exclusion guard, and
  normative/future migration documentation; none is migration orchestration.\n Follow-up: commit 7940763 removed the sole stale bookkeeping response that\n referenced the deleted rollback endpoint; its ordinary API assertion and\n isolated backend typecheck passed.

Explicit keep classifications

• Keep backend/scripts/export-execution-data.ts,
  backend/scripts/validate-execution-export.ts,
  backend/scripts/dry-run-import.ts, backend/scripts/restore-drill.ts, and
  their tests: these are portable DataOps export validation and disaster
  recovery, not source-system migration frameworks.
• Keep backend/scripts/import-calendar.ts,
  backend/scripts/import-newsletter-slots.ts, and raw podcast/source
  extraction/export tools as untested one-off source import/export scripts.
• Keep Sponsor Finance reconcile, conversational execution reconcile, and
  Sponsor Communications unknown-send-attempt reconcile: these recover live
  product workflows from uncertain external effects and are not data migration.
• Keep ordinary bookkeeping document upload/link/report APIs and their product
  tests. The bookkeeping bucket/IAM infra test remains, under a truthful name.
• Keep portable artifact source provenance values (import/migration) in
  exported runtime records unless a separate data-model decision removes them;
  they are data provenance, not an orchestration framework.

Acceptance criteria

• Simple raw-source-to-database import scripts remain available for the
  eventual post-model cutover.
• All importer-only helpers are outside backend/src and excluded from the
  TypeScript runtime output and Lambda artifact.
• No migration-only production route/query mode remains.
• No generalized migration planner/checkpoint/reconciliation/rollback/orphan
  framework remains.
• One-off import suites are absent from the normal backend test command and
  CI; no replacement ongoing migration test command is added.
• Runtime product behavior and ordinary tests remain intact.
• Portable export/validation and disaster-recovery restore tooling remain.
• Build-time artifact checks fail if importer/framework modules reappear in
  shipped runtime output.
• Backend/frontend ongoing tests, typecheck, build, SAM artifact checks,
  planning/infra tests, deployment, and production smoke pass.
  Local evidence complete: backend 916 tests (911 pass, 5 expected skips),
  frontend 181 tests, infra 35 tests, backend typecheck/build, SAM validation,
  six-function SAM build, real frontend artifact verification, and real runtime
  boundary verification all passed. Deployment and production smoke remain for
  the lifecycle owner; no import ran.

Out of scope

• Running any source-data import now.
• Finalizing the destination data model.
• Deleting retained one-off import scripts after cutover.
• Replacing non-empty production tables without a separately reviewed exact
  CloudFormation change.