Skip to content

Remove mailing-list export from DataOps; Dapier now owns it #246

Closed
Closed
Remove mailing-list export from DataOps; Dapier now owns it#246
Labels
P1ImportantbackendBackend/APIdataData model, migration, storageenhancementNew or improved functionalityfrontendFrontend UIinfraDeployment and infrastructureportalShared portal shell and UXtestingTests and QA

Description

@alexeygrigorev
alexeygrigorev
opened on Oct 5, 2026

Remove mailing-list export from DataOps; Dapier now owns it

Status: pending
Tags: enhancement, portal, frontend, backend, infra, testing, data, P1
Depends on: None
Blocks: None

Reporter context

Alexey: "remove mailing export frmo the code and from ui - it's done in dapier now"

The live Operations nav still has Mailing exports. The page heading is Mailing-list exports and currently shows a load error. Dapier now runs the Mailchimp export, so DataOps must stop requesting, storing, scheduling, and presenting it.

This is a complete capability deletion, not a hide-the-nav or disable-config change. Do not leave shims, dual paths, empty routes, or a dormant Mailchimp reader.

Current surface (researched)

Frontend

  • Operations nav item #mailing-exports-nav-button in frontend/index.html
  • Hash route /#/mailing-exports in frontend/src/core/workspace.js
  • Surface frontend/src/surfaces/finance/mailing.js, composed through frontend/src/surfaces/finance/index.js
  • Runtime wiring in frontend/src/runtime/application.js and frontend/src/runtime/surface-composition.js
  • Title map in frontend/src/surfaces/operations-overview.js
  • Mailing CSS in frontend/src/styles.css
  • Asset inventory backend/src/docs/frontend-assets.json (src/surfaces/finance/mailing.js)

Unknown hashes already recover to Home (/#/). After the route is deleted, /#/mailing-exports must use that existing recovery. Do not add a mailing-specific redirect.

Backend API and service

  • GET /api/mailing-exports and POST /api/mailing-exports/run in backend/src/routes/mailingExports.ts, dispatched from backend/src/router.ts
  • Service, Mailchimp adapter, provider registry, and Dapier credential reader under backend/src/mailingExports/
  • Job persistence backend/src/db/mailingExports.ts (PK/SK MAILING_EXPORT#… in TABLE_ARTIFACTS)
  • Scheduled handler branch detail.dataopsAction === 'mailing-export' in backend/src/handler.ts
  • Private artifact download special-case for the mailing bucket in backend/src/routes/artifacts.ts
  • Telemetry family mailing_exports in backend/src/apiRequestTelemetry.ts
  • Local/e2e provider simulator in backend/scripts/test-server.ts (/__e2e__/mailing-provider)
  • Local env: DATAOPS_MAILING_EXPORTS_CONFIG, DATAOPS_MAILING_EXPORTS_BUCKET, DATAOPS_DAPIER_CREDENTIALS_TABLE

This is an operator workflow being deleted. Remove the authenticated API path too. After removal, /api/mailing-exports and /api/mailing-exports/run must behave like any other unknown /api/* route (unauthenticated 401 before existence is revealed; authenticated 404).

Infra, schedule, and deploy

All of this lives in this repo’s infra/template.full.yaml (not ../aws-infra):

  • Parameters MailingExportsConfig, DapierCredentialsTableName, DapierCredentialsTableArn
  • MailingExportsBucket (DeletionPolicy: Retain, UpdateReplacePolicy: Retain, versioned, public-blocked, AES256) and MailingExportsBucketPolicy
  • Daily EventBridge BackendFunctionDailyMailingExport at cron(0 9 * * ? *) with input dataopsAction: mailing-export, plus BackendFunctionDailyMailingExportPermission
  • Backend IAM: dynamodb:GetItem on the Dapier credentials table ARN with dynamodb:LeadingKeys: [mailchimp]; s3:GetObject/s3:PutObject on the mailing bucket
  • Env: DATAOPS_MAILING_EXPORTS_CONFIG, DATAOPS_MAILING_EXPORTS_BUCKET, DATAOPS_DAPIER_CREDENTIALS_TABLE

GitHub Actions .github/workflows/deploy-dataops-v1.yml currently requires Dapier table name/ARN and passes MAILING_EXPORTS_CONFIG. Those parameters and the required-var check go away with the template parameters.

DATAOPS_DAPIER_CREDENTIALS_TABLE is used only by the mailing-export credential reader. After this issue, DataOps must not read the Dapier credentials table.

Tests and docs that still assume the product exists

  • backend/tests/mailing-exports.test.ts (delete with the module)
  • Pagination/auth/telemetry coverage that names /api/mailing-exports
  • Frontend finance-surface, routing, and runtime-composition mailing cases
  • E2E: canonical-capability-behavior mailing journey, canonical-route-parity, frontend-module-characterization, issue-208-pagination-journeys mailing continuation, frontend-capabilities.json / frontend-capabilities.test.mjs, isolated-capability mailing config, parity-target mailing config
  • Infra: tests/infra/test_template.py mailing privacy/IAM test, BackendFunctionDailyMailingExport schedule inventory, and the processed-template identity inventory that still lists that rule
  • docs/mailing-list-exports.md (delete)
  • JTBD bullets for weekly Mailchimp backup in docs/operations-manager-platform-jtbd.md
  • Conversational plugin architecture bullet “mailing-list exports” in docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md
  • Baseline recurring seed Backup MailChimp mailing list to Google Drive in backend/src/deploymentSeeds.ts
  • Local env allow-list in scripts/dev-portal-lib.mjs

Scope

Delete the DataOps mailing-list export product end to end:

  1. UI. Remove the Operations nav item, hash route, finance mailing surface, runtime wiring, CSS, and asset-list entry. Remaining Operations items (Newsletter, Calendar, Sponsors, Bookkeeping, Review) stay and still work.
  2. API. Remove GET /api/mailing-exports and POST /api/mailing-exports/run. Do not keep a stub that returns empty configs.
  3. Backend. Delete backend/src/mailingExports/, backend/src/db/mailingExports.ts, backend/src/routes/mailingExports.ts, the handler scheduled branch, the mailing-bucket download branch, telemetry, and local/e2e Mailchimp simulators.
  4. Schedule and IAM. Remove the daily EventBridge rule/permission, mailing-bucket IAM, and Dapier GetItem IAM. DataOps must not call Mailchimp and must not read Dapier credentials.
  5. Stack parameters and deploy. Remove MailingExportsConfig, DapierCredentialsTableName, DapierCredentialsTableArn, related env vars, and the GitHub Actions required-var/parameter-override wiring.
  6. S3 resource. Remove MailingExportsBucket and MailingExportsBucketPolicy from the template. Keep DeletionPolicy: Retain through removal so CloudFormation orphans the physical bucket instead of failing or destroying contact archives. Do not add a runtime emptier.
  7. Recurring seed. Remove Backup MailChimp mailing list to Google Drive from BASELINE_RECURRING_CONFIGS so new deploys stop creating/updating that weekly task. Do not add a production migrator that deletes existing recurring configs.
  8. Docs. Delete docs/mailing-list-exports.md. Drop DataOps-owned weekly Mailchimp-backup language from JTBD and the conversational plugin inventory.
  9. Tests. Delete mailing-export product tests. Update shared route/capability/infra inventories so they no longer require the deleted surface. Treat the SAM identity-inventory change as an intentional resource deletion (update test_processed_template_keeps_baseline_resource_identity_inventory and schedule tests; do not keep a dummy EventBridge rule to satisfy Make SAM artifacts and scheduled-target previews identity-stable #217).

No compatibility shims, empty /mailing-exports page, disabled-config mode, or leftover Dapier reader.

Data safety

Mailing-export jobs live in TABLE_ARTIFACTS as MAILING_EXPORT#… items. Completed runs created private artifacts mailing-export-{id} whose storageUri points at the mailing bucket (mailing-exports/… ZIP keys) and may be referenced from a recurring task.

Decisions:

  • Do not add migration machinery to scan/delete those items on deploy. Once the code stops reading MAILING_EXPORT# and the mailing download branch is gone, leftover jobs are inert. Leftover artifacts remain ordinary Artifact records; private download must no longer special-case the mailing bucket (expect 409 like any other non-email private object).
  • Do not keep a dual-read path so old ZIPs remain downloadable from DataOps. Dapier is the source of truth.
  • Bucket removal is sequenced by Retain. This issue removes the bucket from the stack. CloudFormation leaves the physical bucket. Emptying or deleting that bucket, and deleting leftover DynamoDB items or artifacts, is [HUMAN] because it is destructive production data that can contain mailing-list contact archives.
  • GitHub Actions variables MAILING_EXPORTS_CONFIG, DAPIER_CREDENTIALS_TABLE_NAME, and DAPIER_CREDENTIALS_TABLE_ARN become unused after deploy. Deleting the org/repo vars is [HUMAN].
  • Live recurring config created from the Mailchimp backup seed is not auto-deleted. [HUMAN] may pause or delete it in Recurring if it still generates Thursday tasks.
  • Do not log, export, or attach archive contents, credentials, or raw bucket identity in this public issue or in tests.

Acceptance Criteria

  • Operations sidebar has no Mailing exports item. Newsletter, Calendar, Sponsors, Bookkeeping, and Review remain and open their current surfaces.
  • /#/mailing-exports is not a workspace route. Visiting it recovers to Home through the existing unknown-hash behavior. No mailing-exports heading, cards, or run actions remain.
  • GET /api/mailing-exports and POST /api/mailing-exports/run are gone. Unauthenticated calls are refused like other unknown /api/* paths. Authenticated calls 404. No empty-config stub.
  • backend/src/mailingExports/, backend/src/db/mailingExports.ts, and backend/src/routes/mailingExports.ts are deleted. The Lambda no longer handles dataopsAction: mailing-export.
  • Artifact private download no longer authorizes mailing-bucket keys. Email-document private download is unchanged.
  • EventBridge daily mailing-export rule and permission are gone. Ordinary backend cron and portable execution-export schedules remain.
  • Template has no MailingExportsBucket, MailingExportsBucketPolicy, MailingExportsConfig, DapierCredentialsTableName, DapierCredentialsTableArn, mailing env vars, mailing S3 IAM, or Dapier GetItem IAM. Deploy workflow no longer requires or passes those parameters.
  • docs/mailing-list-exports.md is deleted. Remaining docs do not tell operators or agents that DataOps still exports Mailchimp audiences.
  • Baseline recurring seed no longer includes “Backup MailChimp mailing list to Google Drive”.
  • Shared tests, capability catalogs, and SAM identity/schedule inventories no longer require mailing-exports. No new shims or skipped-but-still-present product tests.
  • Desktop and mobile screenshots prove the Operations nav without Mailing exports, /#/mailing-exports recovering to Home, and the remaining Operations items still loading.
  • [HUMAN] After deploy: confirm the daily mailing-export rule is gone in the live stack; delete unused GitHub Actions vars MAILING_EXPORTS_CONFIG, DAPIER_CREDENTIALS_TABLE_NAME, and DAPIER_CREDENTIALS_TABLE_ARN; optionally empty/delete the retained mailing-exports bucket and leftover MAILING_EXPORT# / mailing-export-* records; optionally pause/delete the live Mailchimp-backup recurring config. Do not put archive contents or credentials in this repo.

Test Scenarios

Scenario: Operations nav without mailing export

Given: a signed-in operator on the portal
When: they open the workspace sidebar on desktop and the mobile nav
Then: Operations lists Newsletter, Calendar, Sponsors, Bookkeeping, and Review, and does not list Mailing exports
And: each remaining Operations item opens its current surface without a load error caused by this change

Scenario: Old mailing-exports bookmark

Given: the mailing-exports route has been removed
When: the operator visits /#/mailing-exports
Then: the app recovers to Home (/#/) using existing unknown-hash recovery
And: no mailing-exports surface, heading, or API call is made

Scenario: Mailing API is gone

Given: the backend no longer registers mailing-export routes
When: an unauthenticated client calls GET /api/mailing-exports or POST /api/mailing-exports/run
Then: the call is refused before a handler can reveal whether the route exists
When: an authenticated operator calls the same paths
Then: the response is 404 Not found
And: no Mailchimp, Dapier-credential, or mailing-bucket I/O occurs

Scenario: Scheduled mailing export is gone

Given: the SAM template after this change
When: schedule and processed-template identity tests run
Then: there is no BackendFunctionDailyMailingExport rule or dataopsAction: mailing-export input
And: daily backend cron and portable execution-export rules remain
And: the identity inventory is updated for this intentional deletion rather than preserving a dummy mailing rule

Scenario: Artifact download no longer serves mailing ZIPs

Given: a leftover private artifact whose storage URI is on the former mailing-exports bucket
When: an operator requests its private download
Then: the mailing special-case is gone and download is not available for that artifact
And: email-document private downloads still work

Scenario: Shared catalogs drop mailing-exports

Given: frontend capability JSON, route-parity lists, pagination journeys, finance-surface tests, and infra privacy tests
When: the full verification workflow for this issue runs
Then: none of those suites still require /#/mailing-exports, /api/mailing-exports, MailingExportsBucket, or Dapier mailing IAM
And: remaining collection pagination, auth-table, and Operations surfaces still have coverage

Out of Scope

  • Dapier’s own Mailchimp export implementation, Dapier UI, and the Dapier-owned credentials table contents. DataOps only stops reading that table.
  • Source repos ../dtc-operations, ../datatasks, ../podcast-assistant, and ../aws-infra (mailing resources are in infra/template.full.yaml here).
  • Newsletter, Calendar, Sponsors, Bookkeeping, Review, Home, Tasks, Inbox, and other Operations/Knowledge items except removing mailing-export coupling.
  • Mailchimp campaign creation used by the newsletter workflow.
  • Historical spreadsheet/todo mapping text in backend/docs/ except where a test fixture would otherwise keep shipping a live mailing-export product.
  • Building a DynamoDB/S3 cleanup migrator, restore drill, or dual-read for old ZIPs.
  • Manually emptying/deleting the retained AWS bucket, GitHub Actions org vars, leftover artifacts, or the live recurring backup config (those are [HUMAN] after ship).
  • Closing or retitling Automate private Mailchimp account exports and attach them to recurring work #108 and Converge Newsletter, Calendar, and Mailing Export presentation #222 in GitHub (orchestrator follow-up below).

Related issues (orchestrator follow-up; do not close from this issue body)

#217’s SAM identity inventory will change because this issue deletes a scheduled rule. That is expected; update the inventory here. #217 is not a dependency.

Dependencies

  • No issue dependency. Dapier already owns the export; DataOps should stop in this change.
  • Infra deletion of the bucket resource is safe in the same deploy because DeletionPolicy: Retain. Do not wait for a human to empty the bucket before removing code, API, schedule, IAM, and the template resource.
  • After push, On-Call confirms the mailing EventBridge rule is gone and deploy no longer requires Dapier mailing parameters. Human cleanup of vars/bucket/leftover records does not block merge.

Required verification

Engineer may iterate with focused tests. Independent Tester runs the full relevant workflow on one frozen candidate and records commands, exit codes, and screenshot paths. Search-index rebuild is not required (no content/registry change).

node --test frontend/test/routing.test.mjs frontend/test/runtime-composition.test.mjs frontend/test/finance-surface.test.mjs
npm run test:frontend:unit
npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
node backend/scripts/verify-frontend-artifact.mjs --source frontend --artifact backend/dist
npm --prefix backend run test:e2e -- e2e/canonical-route-parity.spec.js e2e/canonical-capability-behavior.spec.js e2e/frontend-module-characterization.spec.js e2e/issue-208-pagination-journeys.spec.js --retries=0
npm --prefix backend run test:e2e
npm --prefix backend run test:e2e:frontend-parity
python -m pytest tests/infra/test_template.py
make sam-validate
make sam-build
make ci
git diff --check

If a command is skipped, Tester must say why. Do not skip the infra template tests: they currently pin mailing bucket, Dapier IAM, deploy parameters, and the daily mailing EventBridge rule.

Screenshots

Capture untracked native-size PNGs under .tmp/screenshots/issue-246/ (do not commit):

  • 01-operations-nav-no-mailing-desktop-1440x900.png
  • 02-operations-nav-no-mailing-mobile-390x844.png
  • 03-mailing-exports-hash-recovers-home-desktop-1440x900.png
  • 04-newsletter-still-works-desktop-1440x900.png
  • 05-calendar-still-works-desktop-1440x900.png
  • 06-bookkeeping-still-works-desktop-1440x900.png

Tester reads each image for leftover “Mailing exports” / “Mailing-list exports” copy, broken Operations nav, error pages, and layout overlap.

Activity

  1. alexeygrigorev
    added
    enhancementNew or improved functionality
    needs groomingRaw intake that needs PM grooming
    portalShared portal shell and UX
    frontendFrontend UI
    backendBackend/API
    infraDeployment and infrastructure
    P1Important
    on Oct 5, 2026

  2. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    Reporter screenshot of the Mailing exports nav item to remove.

  3. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    Reporter screenshot: Operations nav with Mailing exports underlined as the item to remove.

  4. alexeygrigorev
    added
    testingTests and QA
    dataData model, migration, storage
    and removed
    needs groomingRaw intake that needs PM grooming
    on Oct 5, 2026
  5. alexeygrigorev
    changed the title [-]Remove mailing export from DataOps; it now lives in Dapier[/-] [+]Remove mailing-list export from DataOps; Dapier now owns it[/+] on Oct 5, 2026

  6. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    GROOMED

    Researched the live mailing-export product and rewrote this into an agent-ready deletion spec.

    Decision: remove the capability completely from UI, authenticated API, backend service, daily EventBridge schedule, Dapier credential IAM, stack parameters, tests, and docs/mailing-list-exports.md. Dapier now owns the export; DataOps must not keep a hidden reader, empty /mailing-exports page, or disabled-config mode.

    Data safety: mailing jobs (MAILING_EXPORT# in the artifacts table) and private ZIPs become unused. Do not add a cleanup migrator. Remove MailingExportsBucket from the template while leaving DeletionPolicy: Retain so CloudFormation orphans the physical bucket. [HUMAN] after deploy: delete unused GitHub Actions vars, optionally empty/delete the retained bucket and leftover artifacts, and optionally pause the live “Backup MailChimp mailing list to Google Drive” recurring config.

    Orchestrator follow-up (not done in this grooming):

    Labels: removed needs grooming; added testing and data. Ready for Software Engineer.

  7. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    SWE implementation — #246 mailing-export deletion

    Complete capability deletion: DataOps no longer requests, stores, schedules, or presents mailing-list export. Dapier owns Mailchimp export. No shims, empty routes, disabled-config mode, dormant Dapier/Mailchimp credential reader, mailing-specific redirect, or runtime S3/Dynamo emptier.

    Worktree: /home/alexey/git/dataops/.tmp/worktrees/issue-246-remove-mailing-exports
    Branch: issue-246-remove-mailing-exports
    Base: 0e274de (origin/main at assignment)
    Not committed.

    Deleted / removed

    UI: Operations nav #mailing-exports-nav-button, hash route /#/mailing-exports, frontend/src/surfaces/finance/mailing.js (unwired from finance/index.js only), runtime wiring, operations title map, mailing-export CSS rules only, frontend-assets inventory.

    API/backend: backend/src/routes/mailingExports.ts, backend/src/mailingExports/, backend/src/db/mailingExports.ts, handler dataopsAction === 'mailing-export', mailing-bucket private-download special-case (email-document path stays), telemetry family mailing_exports, local/e2e mailing provider simulator.

    Infra/deploy (infra/template.full.yaml): parameters MailingExportsConfig, DapierCredentialsTableName, DapierCredentialsTableArn; MailingExportsBucket + policy (resource still had DeletionPolicy: Retain at removal so CFN orphans the physical bucket); BackendFunctionDailyMailingExport rule + permission; mailing S3 IAM and Dapier GetItem IAM; env DATAOPS_MAILING_EXPORTS_CONFIG, DATAOPS_MAILING_EXPORTS_BUCKET, DATAOPS_DAPIER_CREDENTIALS_TABLE; GitHub Actions deploy wiring. Daily backend cron and portable execution-export schedules remain.

    Seeds/docs: baseline recurring seed “Backup MailChimp mailing list to Google Drive”; docs/mailing-list-exports.md; DataOps-owned weekly Mailchimp-backup language in JTBD and conversational plugin inventory. Deploy seed gate now expects 6 recurring configs.

    Tests: deleted backend/tests/mailing-exports.test.ts; added backend/tests/mailing-exports-removed.test.ts; updated shared inventories. SAM identity inventory compares processed current template to frozen origin/main 0e274de minus the four mailing resources (BackendFunction / BackendFunctionRole excluded from exact equality because IAM/env change). No dummy EventBridge rule.

    Changed files

    .github/workflows/deploy-dataops-v1.yml
backend/e2e/canonical-capability-behavior.spec.js
backend/e2e/canonical-route-parity.spec.js
backend/e2e/canonical-task-workflow-behavior.spec.js
backend/e2e/frontend-capabilities.json
backend/e2e/frontend-module-characterization.spec.js
backend/e2e/helpers/isolated-capability-server.js
backend/e2e/issue-208-pagination-journeys.spec.js
backend/scripts/frontend-capabilities.test.mjs
backend/scripts/frontend-parity-target.mjs
backend/scripts/test-server.ts
backend/src/apiRequestTelemetry.ts
backend/src/db/mailingExports.ts (deleted)
backend/src/deploymentSeeds.ts
backend/src/docs/frontend-assets.json
backend/src/handler.ts
backend/src/mailingExports/* (deleted)
backend/src/router.ts
backend/src/routes/artifacts.ts
backend/src/routes/mailingExports.ts (deleted)
backend/tests/api-authorization.test.ts
backend/tests/api-request-telemetry.test.ts
backend/tests/collection-route-pagination.test.ts
backend/tests/deployment-seed-contract.test.ts
backend/tests/mailing-exports.test.ts (deleted)
backend/tests/mailing-exports-removed.test.ts (added)
docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md
docs/mailing-list-exports.md (deleted)
docs/operations-manager-platform-jtbd.md
frontend/DESIGN_SYSTEM.md
frontend/index.html
frontend/src/core/workspace.js
frontend/src/runtime/application.js
frontend/src/runtime/surface-composition.js
frontend/src/styles.css
frontend/src/surfaces/finance/index.js
frontend/src/surfaces/finance/mailing.js (deleted)
frontend/src/surfaces/operations-overview.js
frontend/test/finance-surface.test.mjs
frontend/test/routing.test.mjs
frontend/test/runtime-composition.test.mjs
infra/template.full.yaml
scripts/dev-portal-lib.mjs
tests/infra/test_template.py

    finance/index.js only unwires mailing. styles.css only removes mailing-export rules/selectors.

    Tests run

    Command Result
    node --test frontend/test/routing.test.mjs frontend/test/runtime-composition.test.mjs frontend/test/finance-surface.test.mjs pass 31 / 0 fail
    npm run test:frontend:unit pass 320 / 2 skipped / 0 fail
    npm --prefix backend run typecheck pass
    npm --prefix backend test pass 1089 / 1 skipped / 0 fail
    npm --prefix backend run build (includes verify-frontend-artifact.mjs) pass
    uv run --with pytest --with pyyaml python -m pytest tests/infra/test_template.py 32 pass, 8 fail (see below)
    mailing identity/schedule/removal infra tests pass (test_processed_template_keeps_baseline_resource_identity_inventory, test_five_schedules_are_explicit_identity_stable_resources_with_exact_behavior, test_mailing_export_capability_is_removed_from_template_and_deploy)
    sam validate --template-file infra/template.full.yaml pass
    git diff --check pass

    The 8 infra failures already mismatch origin/main 0e274de vs older SHA 6236865 / stale string contracts. They are not caused by this deletion (KnowledgeWorker/alarms, sandbox DataOpsEnvironment, GITHUB_OWNER on KnowledgeWorker, sponsor ConditionCheckItem, $body.templates.total == 11, AUTH_SESSION_LIFETIME_SECONDS: 28800). I did not rewrite those unrelated #217 tests.

    Screenshots

    Skipped. No portal process in this worktree, and Playwright is not installed here (backend/node_modules/@playwright/test missing). Tester should capture under .tmp/screenshots/issue-246/.

    Acceptance criteria

    • Operations sidebar has no Mailing exports; remaining Operations items stay (nav/HTML + routing tests). Visual proof pending Tester screenshots.
    • /#/mailing-exports is not a workspace route; unknown-hash recovery to Home (routing + runtime-composition + e2e specs). Visual proof pending Tester screenshots.
    • Unauthenticated /api/mailing-exports and /run → 401 { error: 'Unauthorized' }; authenticated → 404 { error: 'Not found' }. No stub.
    • mailingExports/ modules, DB, routes deleted; handler no longer special-cases mailing-export (falls through to cron).
    • Leftover private mailing ZIP download → 409 like other non-email private objects. Email-document path unchanged.
    • Daily mailing EventBridge rule/permission gone; cron + execution-export remain.
    • Template/deploy have no mailing bucket/params/env/IAM/Dapier GetItem wiring. Bucket resource kept Retain through removal.
    • docs/mailing-list-exports.md deleted; JTBD/plugin inventory no longer say DataOps exports Mailchimp audiences.
    • Baseline recurring seed no longer includes MailChimp mailing-list backup (6 remaining configs; deploy jq gate updated).
    • Shared tests/catalogs/SAM identity+schedule inventories no longer require mailing-exports. No dummy EventBridge rule.
    • Desktop/mobile screenshots — Tester.
    • [HUMAN] after deploy (rule gone, unused GHA vars, optional bucket/item/recurring cleanup).

    Skipped (Tester)

    • npm --prefix backend run test:e2e / focused e2e / test:e2e:frontend-parity — Playwright not installed in this worktree after npm ci.
    • make sam-build / make ci — full packaging/CI aggregate left for Tester on a frozen candidate.
    • python -m pytest without uv — this environment has no pytest; used uv run --with pytest --with pyyaml.
    • Search-index rebuild — issue says not required.
    • Screenshots — no running portal here.

    Out of scope kept: historical backend/docs/data.md, Trello import mapping, newsletter Mailchimp campaign links, SOP systems: ["mailing"].

  8. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    TEST PASS

    Tester independently verified the uncommitted #246 deletion in worktree issue-246-remove-mailing-exports (base 0e274de). Product code was not edited. Playwright was present (node_modules/@playwright/test v1.61.1, workspace-hoisted); Chromium was installed and e2e was run. SWE’s “Playwright missing after npm ci” skip is not accepted as a skip reason.

    Commands

    Command Exit Result
    node --test frontend/test/routing.test.mjs frontend/test/runtime-composition.test.mjs frontend/test/finance-surface.test.mjs 0 31 pass / 0 fail
    npm run test:frontend:unit 0 320 pass / 2 skipped / 0 fail
    npm --prefix backend run typecheck 0 pass
    npm --prefix backend test 0 1089 pass / 1 skipped / 0 fail; includes mailing-export capability is removed (1065ms)
    npm --prefix backend run build + node backend/scripts/verify-frontend-artifact.mjs --source frontend --artifact backend/dist 0 84 frontend assets; no mailing.js
    focused Playwright (canonical-route-parity, canonical-capability-behavior, frontend-module-characterization, issue-208-pagination-journeys, --retries=0) 1 38 pass / 5 fail (see pre-existing)
    npm --prefix backend run test:e2e -- --retries=0 1 90 pass / 13 fail / 103 tests / 15.5m (see pre-existing)
    npm --prefix backend run test:e2e:frontend-parity -- --retries=0 1 unchanged script vs 0e274de; see pre-existing
    python -m pytest tests/infra/test_template.py skipped env python is /home/alexey/git/sandbox-screenshots/.venv/bin/python with no pytest (No module named pytest)
    uv run --with pytest --with pyyaml python -m pytest tests/infra/test_template.py 1 this worktree: 8 failed, 32 passed
    same pytest on nested worktree at 0e274de with no #246 diff 1 origin/main: 9 failed, 31 passed
    make sam-validate 0 valid SAM template
    make sam-build 0 Node 20.20.2 required (build-sam-artifact.mjs); Node 24.13.1 failed the version assert. Build Succeeded; no mailing asset
    make ci (Node 20.20.2) first run 2 1 flake in e2e-global-setup.test.ts (“retains the last HTTP status…”); file is not in this diff
    isolated rerun of backend/tests/e2e-global-setup.test.ts 0 18 pass / 0 fail
    make ci rerun (Node 20.20.2) 0 backend 1089 pass / 1 skipped; transactions pass; SAM Build Succeeded; runtime boundary + artifact isolation 13 pass
    git diff --check 0 clean

    sam-build / make ci were run with PATH=/home/alexey/.nvm/versions/node/v20.20.2/bin:$PATH.

    Diff vs SWE handoff

    Uncommitted tree matches a complete deletion: UI nav/route/surface/runtime/CSS, API routes/service/db/handler/telemetry, EventBridge schedule, Dapier IAM + MailingExportsConfig stack params, deploy workflow params, MailChimp backup seed (recurring gate 7→6), tests, and docs/mailing-list-exports.md. No empty route, disabled-config mode, dormant Dapier reader, or mailing-specific redirect. Leftover “mailing” strings are absence tests, newsletter Mailchimp campaign links (out of scope), or historical SOP systems: ["mailing"] (out of scope).

    MailingExportsBucket on 0e274de already had DeletionPolicy: Retain and UpdateReplacePolicy: Retain. Current template has no mailing resources, so CloudFormation will orphan the physical bucket.

    Email-document private download remains in backend/src/routes/artifacts.ts. Leftover mailing ZIP download is 409.

    Acceptance criteria

    AC Result Evidence
    Complete deletion from UI/API/backend/schedule/Dapier IAM/stack params/tests/docs PASS 47-file diff, 111+/2392−; no product mailing strings; frontend-assets.json has no mailing
    No shims / empty routes / disabled-config / dormant Dapier reader / mailing-specific redirect PASS unknown hash uses existing recovery; API is unknown /api/*
    /#/mailing-exports recovers to Home PASS screenshot 03 (hash=#/, Today); frontend-module-characterization invalid-hashes test passed (includes #/mailing-exports); canonical-route-parity passed
    Unauthenticated /api/mailing-exports refused like other unknown /api/* PASS unit: 401 { error: 'Unauthorized' }
    Authenticated /api/mailing-exports is 404 PASS unit: 404 { error: 'Not found' }
    MailingExportsBucket removed with Retain at removal time PASS origin/main template Retain; current template has no resource
    Email-document private download stays PASS artifacts.ts still allows email-bucket sensitive objects; leftover mailing ZIP 409

    Screenshots (read)

    Untracked native-size PNGs under .tmp/screenshots/issue-246/ (not committed):

    • 01-operations-nav-no-mailing-desktop-1440x900.png (1440×900) — Operations = Newsletter, Calendar, Sponsors, Bookkeeping, Review. No “Mailing exports”.
    • 02-operations-nav-no-mailing-mobile-390x844.png (390×844) — same Operations list in the drawer.
    • 03-mailing-exports-hash-recovers-home-desktop-1440x900.png (1440×900) — Today / Home, no error page.
    • 04-newsletter-still-works-desktop-1440x900.png (1440×900) — Newsletter planner.
    • 05-calendar-still-works-desktop-1440x900.png (1440×900) — Operations calendar (October 2026).
    • 06-bookkeeping-still-works-desktop-1440x900.png (1440×900) — Bookkeeping + invoice ledger.

    Tester read each image. No leftover mailing copy, broken Operations nav, error page, or layout overlap.

    Pre-existing failures (not caused by this diff)

    Infra pytest: this worktree 8 failed / 32 passed; origin/main 0e274de 9 failed / 31 passed. Extra origin/main failure is test_processed_template_keeps_baseline_resource_identity_inventory (inventory this issue updated). Remaining 8 are stale #217 / SHA 6236865 contracts (DataOpsEnvironment sandbox vs prod, KnowledgeWorker alarms, GITHUB_OWNER, sponsor TransactWriteItems string, sandbox provenance extra params, $body.templates.total == 11, AUTH_SESSION_LIFETIME_SECONDS: 28800). test_mailing_export_capability_is_removed_from_template_and_deploy passed.

    Focused e2e 5 fails / full e2e 13 fails: mailing-scoped specs passed (route-parity, characterization unknown hash including #/mailing-exports, issue-208 files continuation in the full run). 11 of 13 failing spec files have zero diff vs 0e274de. The other two only delete mailing tests / noMailingConfig. Failures:

    1. Bookkeeping totals expect EUR 18.50, UI shows EUR Expenses 18.50 (Bookkeeping quick wins: ledger form and layout fixes (no data-model change) #243 money-direction; assertion identical on origin/main).
    2. Newsletter reload looks for a 2026-09-03 slot while default From is 2026-10-01 (spec unchanged).
    3. Isolated-server ECONNREFUSED / 30s timeouts (session/Settings, bookkeeping-production-portal) — flake; assistants/artifacts and templates/recurring failed in focused run and passed in full run.
    4. Unchanged specs: task-workflow overdue copy, required-proof equality, docs-outage empty corpus, editor-screenshots exact-pixel (664.078125 > 664.078125), issue-223 Showing 0 of 0, operating-model unavailable copy, runtime-template “private knowledge repository”, surface-feedback [data-summary-id="home"].

    frontend-parity: backend/scripts/frontend-parity.mjs is unchanged. Node 20.20.2: client.getClient is not a function during local schema --eval import. Node 24.13.1: 400 /knowledge/status on source/home-ready. Knowledge/status, not mailing.

    make ci first run: one flake in e2e-global-setup.test.ts (not in this diff). Isolated rerun 18/18; full make ci rerun exit 0.

    No #246 code fix required. Pre-existing e2e/pytest/parity failures belong on their own issues (#243 totals copy, newsletter date default, stale #217 infra SHA, knowledge/status parity).

  9. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    ACCEPTED

    User-perspective acceptance of #246 after Tester TEST PASS.

    Alexey asked to remove mailing export from the code and UI because Dapier owns it now. The operator journey matches the groomed complete-deletion spec.

    Operator experience

    • Operations nav (desktop and mobile) lists Newsletter, Calendar, Sponsors, Bookkeeping, and Review. There is no Mailing exports item.
    • /#/mailing-exports recovers to Home (/#/) through the existing unknown-hash behavior. No mailing heading, cards, or run actions remain.
    • Newsletter planner, Operations calendar, and Bookkeeping / invoice ledger still load. Remaining Operations items are intact; this change does not break them.
    • Copy is not misleading: remaining surfaces describe their own jobs, and docs no longer tell operators or agents that DataOps exports Mailchimp audiences.

    Scope vs implementation

    Uncommitted worktree issue-246-remove-mailing-exports is a complete capability deletion, not a hide/disable:

    • UI: Operations nav item, hash route, frontend/src/surfaces/finance/mailing.js, runtime wiring, CSS, and asset-list entry are gone.
    • API: GET /api/mailing-exports and POST /api/mailing-exports/run are unknown /api/* paths (unauthenticated 401, authenticated 404). No empty-config stub.
    • Backend: mailingExports/ service, DB, routes, scheduled dataopsAction: mailing-export branch, mailing-bucket download special-case, telemetry family, and local/e2e Mailchimp simulator are gone.
    • Infra/deploy: EventBridge daily mailing rule, mailing bucket/policy (Retain at removal so CloudFormation orphans the physical bucket), Dapier GetItem IAM, stack parameters, env vars, and GitHub Actions required-var/overrides are gone.
    • Seed/docs/tests: baseline “Backup MailChimp mailing list to Google Drive” seed removed; docs/mailing-list-exports.md deleted; shared catalogs no longer require the surface.

    No empty /mailing-exports page, mailing-specific redirect, disabled-config mode, or leftover Dapier credential reader. Remaining “mailing” strings are absence tests or out-of-scope historical mapping (Trello import, SOP systems: ["mailing"], newsletter Mailchimp campaigns).

    Evidence

    Tester ran the issue’s verification workflow on this frozen candidate (frontend unit, backend unit/typecheck/build, focused + full e2e, make sam-validate / make sam-build / make ci, infra pytest) and captured the six required screenshots. I read each PNG:

    • 01-operations-nav-no-mailing-desktop-1440x900.png — Operations = Newsletter, Calendar, Sponsors, Bookkeeping, Review. No mailing item.
    • 02-operations-nav-no-mailing-mobile-390x844.png — same list in the mobile drawer.
    • 03-mailing-exports-hash-recovers-home-desktop-1440x900.png — Today / Home after the old hash; capture notes hash=#/.
    • 04-newsletter-still-works-desktop-1440x900.png — Newsletter planner.
    • 05-calendar-still-works-desktop-1440x900.png — Operations calendar (October 2026).
    • 06-bookkeeping-still-works-desktop-1440x900.png — Bookkeeping + invoice ledger.

    No leftover “Mailing exports” / “Mailing-list exports” copy, broken Operations nav, error page, or layout overlap.

    Pre-existing e2e/infra/parity failures Tester listed are not caused by this deletion and do not block acceptance.

    [HUMAN] after ship (does not block merge)

    After deploy: confirm the daily mailing-export EventBridge rule is gone in the live stack; delete unused GitHub Actions vars MAILING_EXPORTS_CONFIG, DAPIER_CREDENTIALS_TABLE_NAME, and DAPIER_CREDENTIALS_TABLE_ARN; optionally empty/delete the retained mailing-exports bucket and leftover MAILING_EXPORT# / mailing-export-* records; optionally pause/delete the live Mailchimp-backup recurring config. Do not put archive contents or credentials in this repo.

    Orchestrator follow-up

    Ready for Software Engineer commit (Closes #246). Do not wait on the [HUMAN] cleanup.

  10. alexeygrigorev
    added 2 commits that reference this issue on Oct 5, 2026

    Remove mailing-list export from DataOps; Dapier owns it

    ee6fb16

    Merge issue-246-remove-mailing-exports: Remove mailing-list export fr…

    f38578e
  11. alexeygrigorev
    added
    humanCode done or issue blocked on human verification
    on Oct 5, 2026

  12. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    Shipped on main as f38578e (merge of ee6fb16).

    Mailing-list export is gone from the DataOps portal, API, schedule, Dapier IAM, and docs. Dapier owns Mailchimp export.

    [HUMAN] remaining after deploy

    Do not close this issue until these are confirmed in production:

    • Daily EventBridge rule BackendFunctionDailyMailingExport is gone after SAM deploy
    • GitHub Actions variables MAILING_EXPORTS_CONFIG, DAPIER_CREDENTIALS_TABLE_NAME, DAPIER_CREDENTIALS_TABLE_ARN deleted
    • Optional: empty dataops-v1-mailing-exports then delete the orphaned bucket
    • Optional: pause/retire recurring config recurring-config-mailing-list-backup if it still exists

    Related: closing #108 as superseded.

  13. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    On-call CI/CD after merge f38578e

    Watched the main push for f38578e (Merge issue-246-remove-mailing-exports).

    Deploy (the #246 gate): pass

    • Deploy DataOps V1 Lambda — success (checks 3m1s, deploy 2m57s)
      • Frontend coverage, backend tests, typecheck, SAM validate, seed, and smoke test all passed.
      • Live sam deploy did not pass MAILING_EXPORTS_CONFIG / DAPIER_CREDENTIALS_*. Those keys are gone from .github/workflows/deploy-dataops-v1.yml, infra/template.full.yaml, and the deploy parameter-overrides.
      • CloudFormation deleted the daily mailing EventBridge rule (BackendFunctionDailyMa AWS::Events::Rule DELETE_COMPLETE) plus its Lambda permission and MailingExportsBucketPolicy. MailingExportsBucket was DELETE_SKIPPED (S3 retention). HUMAN still owns live AWS confirmation that the rule is gone.

    Other workflows on this SHA

    Not a #246 mailing-export regression

    E2E failures are the same pre-existing main set (already failing on db06b87 / fda3592), none of them are mailing-export journeys:

    • canonical-capability-behavior Newsletter/Calendar (Server newsletter ${id} not visible)
    • canonical-task-workflow-behavior Tasks queue overdue empty copy
    • docs-outage empty corpus copy
    • editor-screenshots uploaded screenshot geometry
    • issue-223-workspace-redesign .home-attention-count empty text
    • runtime-template-readonly “private knowledge repository” copy
    • surface-feedback-states Home/Tasks summaries

    Planning-docs SOP lint already failed on 16bd8c5 and ad09a89. That job never reached infra template tests; deploy SAM validate covered the #246 template change.

    SWE: no #246 follow-up from this deploy. Do not treat the pre-existing E2E/planning-docs failures as mailing-export work. A separate session is already on production IAM ConditionCheckItem; this deploy did not collide with that.

    On-call: pass for #246 deploy/wiring. E2E + planning-docs remain red on main independently of this merge.

  14. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    HUMAN close-out done.

    • Daily mailing-export EventBridge rule is gone (confirmed earlier on deploy f38578e; still absent now).
    • Deleted unused GitHub Actions repository variables: MAILING_EXPORTS_CONFIG, DAPIER_CREDENTIALS_TABLE_NAME, DAPIER_CREDENTIALS_TABLE_ARN. Remaining repo var is only EMAIL_DOCUMENT_EXTERNAL_SOURCE_PREFIX.
    • Deleted both empty orphaned mailing-export buckets (dataops-v1-mailingexportsbucket-ftjb0jgjayfj and dataops-v1-mailingexportsbucket-nxsjwruqoigw). No current objects or versions.
    • Deleted leftover MAILING_EXPORT# job record (one failed run). No mailing-export-* artifact records remained.
    • Deleted live recurring config “Backup MailChimp mailing list to Google Drive” (RECURRING#8eabd05d-…). Archived its 6 leftover todo tasks so they leave the active queues. One already-done task was left in history.
    • Dapier’s dapier-poll-mailchimp-drive-backup rule was left alone.

    No further HUMAN work on this issue.

  15. alexeygrigorev
    removed
    humanCode done or issue blocked on human verification
    on Oct 5, 2026

  16. alexeygrigorev commented on Oct 5, 2026

    @alexeygrigorev
    alexeygrigorev
    on Oct 5, 2026
    MemberAuthor

    HUMAN production cleanup is done. Closing.

  17. alexeygrigorev
    closed this as completedon Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1ImportantbackendBackend/APIdataData model, migration, storageenhancementNew or improved functionalityfrontendFrontend UIinfraDeployment and infrastructureportalShared portal shell and UXtestingTests and QA

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions