Repository navigation
pm-stack databases (Kaneo Postgres, Grist, Pad) have no backup routine — docker volumes are the only copy #1514
Description
Gap (backup audit, 2026-10-27): pm-stack application databases have no backup routine at all
The dashboard's state files got a backup system today (#1459, closed), but the pm-stack application databases were not part of it:
- Containers
pm-stack-postgres-1/pm-stack-kaneo-1,grist-grist-1,pad-pad-1are running on the laptop (up 32h at audit time). Their data lives only in docker volumes:pm-stack_postgres_data,grist_grist_data,pad_pad_data. - Verified by grep on 2026-10-27: no
pg_dump(or any dump/snapshot step) exists in any laptop tool script, systemd unit (user or system), or pm-stack script. No timer or cron covers these volumes. - If the laptop disk or the docker volumes are lost, Kaneo's boards, Grist's documents, and Pad's data are gone — there is no second copy anywhere.
This is the implementation gap under the durability design in #1258 (stack state in backed-up durable storage): the aggregator state leg landed via #1459; the database leg never did.
Acceptance
- A nightly routine dumps the Kaneo Postgres database (and snapshots/exports Grist + Pad data) into an existing backed-up location (the dashboard-backup set or the restic repo-archive pattern), with an off-machine leg (R2 and/or Google Drive).
- One restore is actually performed and evidenced (restore the Kaneo dump into a scratch container and open it).
- The routine's freshness is visible somewhere the fleet already looks (watchdog or dashboard health), so a silent stop gets named.
Acceptance criteria
- A nightly routine dumps the Kaneo Postgres database and snapshots/exports Grist and Pad data into an existing backed-up location (the dashboard-backup set or the restic repo-archive pattern), with an off-machine leg (R2 and/or Google Drive).
- One restore is actually performed and evidenced: the Kaneo dump is restored into a scratch container and opened successfully.
- The routine's freshness is visible somewhere the fleet already looks (watchdog or dashboard health), so a silent stop gets named.
- No existing dashboard/atlas backup routine (ops(dashboard): daily backups of dashboard/orchestration logs + stores — laptop, Google Drive, R2 #1459, atlas backups: canonical local atlas.db has no routine backup (gitignored, ad-hoc copies only); nightly Turso atlas dump times out #1515) is broken or duplicated by the new routine.
Metadata
Metadata
Assignees
Labels
pm-stack (Pad/Grist) in developing-today/codeMedium: expected under 1 day of workInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveFiled directly by a humanQueue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Oct 27, 2026
CODE-575
dezren39 commentedon Oct 27, 2026
Manual backup run — done and verified (2026-10-27)
Gap #1 handled by hand today per Drew's order. The pm-stack databases are no longer single-copy.
What was backed up, and how
kaneoDB in pm-stack-postgres-1):pg_dump -Fc+pg_dumpall --globals-only, credentials expanded inside the container only (no values printed or written anywhere).home.sqlite3,grist-sessions.db, anddocs/czYHYKgG9SHcm9CPwiKEq1.grist: online backups via the sqlite backup API inside the running container (a raw copy of a live DB is not a backup).pad.dbvia the sqlite backup API from a one-off tool container (the pad image ships no sqlite tooling), plusencryption.key,config.toml,.pad.toml, andattachments/. The encryption key is required to restore pad.db (data is encrypted at rest), so it travels with the backup into the same private stores as the database itself.Bundle (dashboard-backup convention):
~/backups/pm-stack/2026-10-27/pm-stack-2026-10-27.tar.zst— 3,340,257 bytes, sha256a1ad3034ef06bdb86af3b3ff2c7e40579b7f98b75acea88215b622154b50e037, manifest.json covering 9 files, verified against the manifest at write time.Offsite copies, both verified
1IN6uOWsim2Ifn-VSk2JOBZUfgyemLd5l— size and md5 (faa798b680ccd90399f079367bc2f71f) match the bundle.supabase-backups, keypm-stack/2026-10-27/pm-stack-2026-10-27.tar.zst— HEAD-verified (size + stored sha256 metadata), and a GET back was byte-identical.Restore sanity, performed from the R2 copy
integrity_check= ok on all four stores (grist home 26 tables, sessions 1, doc 33, pad 87).pg_restoreof the offsite dump into a scratch database succeeded — 92 TOC table entries, 9,439 live rows after analyze; scratch DB dropped.Routine (scheduling handoff, no snowflake timer created): the dump+package step is a single idempotent script,
~/tools/pm-stack-backup.shon the laptop (dated bundle per day, 30-day prune of its own day dirs). The NixOS flake backup service (#1516 / dezren39/darwin-nix#976) should call/fold this script for the daily schedule. Two gotchas are documented in the script: the grist image entrypoint swallowsdocker run <img> python3, and a stdin-fed program over the laptop access path proved unreliable — the Pad step uses docker create + docker cp instead.dezren39 commentedon Oct 28, 2026
Intake review
Fleet-monitor intake pass (2026-10-28): verified still live — filed today from the backup audit with no labels at all; no merged PR or routine covers the pm-stack application databases (the dashboard state-file leg landed separately). Labels applied: kind/infrastructure, source/human, priority/high, effort/m, status/needs-triage, app/pm-stack.
## Acceptance criteriachecklist has been appended in that form (original body preserved verbatim above it). Priority set to high: these docker volumes are currently the ONLY copy of Kaneo/Grist/Pad data.dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
26 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · parked
pm-dashboard-backup.serviceExecStart is still the pre-fix Nix store build/nix/store/vdv5dq9gkv20bg3hnjrz9j0qpn0aif2b-pm-dashboard-backup/bin/pm-dashboard-backup, whose line 275 still readsDUMP_DIR="$HOME/backups/db-dumps/$DATE"— the PR fix(pm-dashboard-backup): point DUMP_DIR at pm-stack-backup.sh output dir #4651 fix (backups/pm-stack) is merged on main but NOT deployed. (2) The laptop primary~/git/codeis ahead 2 / behind 145 vs origin/main and still carries the untrackedpkgs/id/src/world/replica.rs(not this worker's dirt — left untouched). The next timer run (Tue 2026-11-03 04:36 CST) would therefore still execute the broken build. The last green run on record remains the manual 2026-11-02T23:14:56Z run (result complete, all steps ok)..last-run.jsonok + watchdog clean, and complete task 6593. Deploying a switch from a dirty, behind primary owned by other work is not taken unilaterally by this lane.dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up + parked
pm-dashboard-backup.service(user unit) ExecStart is still the pre-fix store build/nix/store/vdv5dq9gkv20bg3hnjrz9j0qpn0aif2b-pm-dashboard-backup, whose line 275 still readsDUMP_DIR="$HOME/backups/db-dumps/$DATE". (2) Laptop primary~/git/codeis ahead 2 / behind 172 vs origin/main with untrackedpkgs/id/src/world/replica.rs(not this worker's dirt — left untouched), so a unilateral switch is not taken. The 04:36 CST timer run will therefore still execute the broken build. Last green run on record remains the manual 2026-11-02T23:14:56Z run (result complete, all steps ok)..last-run.json+ watchdog, and complete only once the fixed unit is deployed (via a clean-primary switch) and a run ends result=ok with db-dumps in the manifest and verified off-machine copies.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 6593 (small_fix, parked, priority 70) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 4, 2026
agent-note:v1
task: 6593
lane: code-o
state: done (task scope)
note: Dump-dir handoff verified end-to-end on the laptop. Deployed pm-dashboard-backup is now store build jncrx23l (DUMP_DIR=$HOME/backups/pm-stack/$DATE, the #4651 fix; previously vdv5dq9g with db-dumps). Run 2026-11-04T06:16:06Z: result complete, exit 0, steps validate-date/credentials/db-dump/local-archive/r2-upload/drive-upload all ok, destinations local/r2/drive ok, archive_sha256 recorded; the 2026-11-04 tarball contains ./db-dumps/pm-stack-2026-11-04.tar.zst and ./manifest.json. Watchdog (laptop-backups-watchdog) latest verdict 2026-11-03 18:37 CST: OK jobs fresh within 26h. Issue #1514 stays OPEN: its remaining acceptance item is the evidenced Kaneo restore into a scratch container, which is outside this task's packet scope.
dezren39 commentedon Nov 8, 2026
Partial verification — staying open (2026-11-07 done-task-only audit, tracking: #7016).
Queue task(s) [2601, 6593] were marked done, and some work did land:
No merged change closes this issue's full scope, so the done status is not treated as completion. Remainder re-queued as triage task 11569.