Repository navigation
github-backup: estate mirror system (laptop mirrors + dedicated App + Worker webhooks + weekly sweep) #3215
Description
System of record: the docs in this repo — infra/github-backup/ (README + docs/RUNBOOK.md). This issue summarizes and tracks rollout; the repo teaches.
Related: #3205 (per-repo hygiene census / standard task).
What it is
Bare-mirror backup of Drew's GitHub estate on the laptop (~/backups/github/<owner>/<repo>.git), decided 2026-10-31:
- Scope: Tier C — all non-forks + forks <500 MB across 17 owners + force-includes (818 mirrors); nixpkgs family stored once via git alternates pool; 7 passive giants excluded by ruling (upstream is their backup); gaps recorded (dezren39/edsebooks HTTP 451, vertexinc SAML).
- Freshness: dedicated GitHub App hatch-repo-backup (metadata:read + contents:read; push + repository events; its own rate-limit pool) → Cloudflare Worker receiver (HMAC-verified, KV queue) → laptop drainer every 5 min (15-min per-repo coalescing).
- Floors: daily fetch of all mirrors; weekly reconciliation sweep (new repos auto-mirrored per config tier rules; new giant forks land as pending-Drew-review).
- Secrets: Bitwarden by name only (
github-app.backup-*), never in repo/tree/chat.
Rollout status
- System code + config + units + Worker + runbook — PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214
- PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214 merged; units installed on laptop from ~/git/code
- Initial clone fleet complete (818) + MANIFEST/FAMILIES verified
- Worker deployed (wrangler; KV namespace id into wrangler.toml)
- App created (one human click — manifest flow in RUNBOOK §3) + webhook secret set both sides
- App installed on dezren39 + admin orgs (install-report verified)
- End-to-end webhook proof (ping + one real push fetch)
- First weekly reconcile report reviewed
Restore/DR procedures: RUNBOOK §8–§9.
Acceptance criteria
- Every rollout checklist item above is complete: PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214 merged, units installed on the laptop from ~/git/code, the initial clone fleet (818 mirrors) verified against MANIFEST/FAMILIES, the Worker deployed, the hatch-repo-backup App created and installed, and an end-to-end webhook proof (ping + one real push fetch) recorded.
- The first weekly reconcile report has been reviewed and any gaps it names are filed or recorded here.
- Restore/DR procedures (RUNBOOK §8–§9) have been verified against the live mirrors.
Metadata
Metadata
Assignees
Labels
Large: multi-day or open-ended; expect staged check-insInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedNice to have; do when convenientQueue worker lifecycle stateFiled directly by a humanQueue worker lifecycle stateQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
dezren39 commentedon Oct 31, 2026
Rollout update (2026-10-31 16:05 CDT):
{"events": 1, "fetched": ["NixOS/nixpart"]}, exit 0. Secrets sourced from Bitwarden by name via the wrapper.github-app.backup-*) + Worker secret store.python3 bin/app_setup.py convert <code>→ set the webhook secret on the App (value in Bitwarden: github-app.backup-webhook-secret) → install on dezren39 + admin orgs →app_setup.py install-report. Until then the drainer simply finds an empty queue; daily fetch + weekly sweep already keep the tree current.dezren39 commentedon Nov 1, 2026
Status update (2026-10-31, verification close-out + Drew's naming/deploy directives):
Clone fleet: complete and verified — 818/818 mirrors on the laptop (29 GB tree), MANIFEST regenerated (818 mirrored, 0 missing), fsck sample 5/5 clean, working checkout cloned from an alternates member, fleet HEADs consistent (nix exact; hatch-personal/code are ancestor-stale by 35/44 live commits, closed by the daily fetch). Two retry passes with the system's own
clone_all.pycleared all 10 pass-1 failures (all network mid-transfer drops). Checklist boxes for PR merge, units installed, and clone fleet are therefore now true.Naming (Drew, 2026-10-31): the public name is
developing-today-backup. The GitHub App was never created under any name (API 404 for both slugs; no app-credential secrets in Bitwarden), so it will simply be created under the new name — the slug is available. The Worker WAS live ashatch-repo-backup-webhook; PR #3280 renamed it repo-side and redeployed it as developing-today-backup (live, version taggedgit-307a2501ca65), the laptop drainer was repointed via Bitwardengithub-app.backup-worker-urland verified on its next tick, and the old Worker was deleted (old URL 404s).Deploy method (Drew, 2026-10-31): before #3280 the Worker deployed by bare wrangler commands from the runbook. It now follows the repo deployment standard's shape:
just deploy-developing-today-backup→nix run .#developing-today-backup-deploy→ secrets from Bitwarden/SOPS → version upload (git-SHA tag) → separate deployment at 100%, withgithub-backup-units/github-backup-fetch/github-backup-reconcilerecipes alongside. Tool gap recorded in #3280: the deploy script drives wrangler's versions commands, not the standard's cf CLI, because this Worker is wrangler.toml-shaped and dependency-free while cf's flow requires the TypeScript cf/config format.Still open: the App creation click sequence (RUNBOOK §3, now for
developing-today-backup), App installs, the first genuine GitHub webhook delivery proof, and defects #3219 / #3229 (verified still present in deployed + main code).dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — routing note: the eight defect filings against this system from its first day (#3219, #3229, #3233, #3238, #3240, #3250, #3257, #3270) are unified under design #3282, "a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror" (design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace). The eight are adopted as #3282's phases (no parallels); a closing audit (#3283, gated) will post completion evidence here. One rollout interaction to expect: when Phase 1 (#3238's resolvability predicate) lands, the first reconcile/fetch afterwards will recount any bare-skeleton mirrors as missing and re-clone them — while the initial 818-mirror clone is still in flight that recount is the correction working, not a regression. This umbrella's rollout checkboxes are unaffected.
dezren39 commentedon Nov 1, 2026
Intake review
Still live: rollout is mid-flight — PR #3214 is the system PR named in the checklist, and merged PR #3280 (github-backup rename/deploy standard) is a follow-on, not the rollout completing. No merged PR closes this tracker.
Related: #3205 (per-repo hygiene census / standard task); PR #3214 (system code + config + units + Worker + runbook); PR #3280 (follow-on rename/deploy).
Notes / risks: the remaining checklist is mostly laptop-side state (units installed, 818-mirror fleet, Worker deployed, App installed, webhook proof) — verifiable only from the laptop and the Cloudflare dashboard, not from this repo. The App creation click and org installs are the human-gated steps.
Suggested approach: work the checklist in order; record the end-to-end webhook proof and the first weekly reconcile review as comments here.
Labels applied: kind/infrastructure, source/human, status/needs-triage, effort/l, meta/enriched (priority/low already present). Acceptance criteria appended to the body, drawn from the rollout checklist.
dezren39 commentedon Nov 1, 2026
App-creation attempt via RustDesk (2026-10-31 evening): blocked before any GitHub submission — the App does not exist yet.
What happened: the sandbox-side RustDesk client (1.4.5, nix build) cannot render its Flutter UI in this sandbox. Every GL configuration fails at GTK GLArea init — verbatim:
Failed to initialize GLArea: No available configurations for the given pixel format(X11) /No GL implementation is available(Wayland). Verified with an independent system-libGL probe: the sandbox's Xvfb (with GLX enabled) advertises GLX 1.4 with 80 fbconfigs and the Ubuntu Mesa client works against it — but every nix-linked GL client (the RustDesk closure, nix glxinfo) fails identically on all servers tried (system Xvfb, TigerVNC Xvnc, nixpkgs Xvfb ±iglx — the last two don't advertise GLX at all). Root cause: the nix Mesa/GLVND stack cannot create GL contexts in this sandbox (the RustDesk closure ships libglvnd with no usable vendor path here); system Mesa can't be mixed into nix-libc processes (GLIBC_PRIVATE symbol clash). A headless wlroots/cage + EGL attempt hit the same wall. No RustDesk session to the laptop was ever established, so no browser step ran and nothing was submitted to GitHub.Prepared and in place for a retry (e.g. from a client with a working GL stack, or Drew at the laptop):
dezren39 commentedon Nov 1, 2026
GitHub App created + installed + webhook proof — 2026-10-31 (CDP session on laptop)
App:
developing-today-backup, App ID 5562183, owner @dezren39. Created via the manifest flow (manifest verified byte-exact: metadata+contents read-only, events push+repository, webhookhttps://developing-today-backup.developing-today.workers.dev/github). Sudo cleared via GitHub Mobile approval (match number relayed live).Credentials — recovered from a convert crash (bug filed by this comment)
app_setup.py convertexchanged the code fine but crashed storing the private key:bws_setpasses the value as a bare CLI argument and the Bitwarden CLI's parser rejects values beginning with-(the PEM header) —error: unexpected argument '-----BEGIN RSA PRIVATE KEY-----'. app-id / client-id / client-secret were stored before the crash; the conversion PEM was unrecoverable (GitHub shows it once).Recovery, completed same session:
SHA256:j5HZDaG+2pWAuVbNdTwuNE4M8WpazhQqXcRZCNvD8lc=.github-app.backup-private-keyusing a--argument separator (probe secret round-tripped byte-exact first); stored value sha256 matches the downloaded file (43f2eb26…); download shredded.SHA256:8qYjT+…); exactly one key pair remains.app_setup.py: makebws_setinsert--before the value (or usebws secret edit/createvalue-safe form) soconvertcan't crash on PEM storage again.install-reportwas run as an equivalent stdlib+openssl RS256 mint →GET /app/installationsreturned 200.Webhook secret
app_setup.py webhook-secretrun (fresh value in Bitwarden), the same value set on the App (General → Webhook → Change secret; save confirmed), andjust deploy-developing-today-backupredeployed the Worker (versionacbb41a4, taggit-efc4e182d400) re-setting Worker secrets from Bitwarden./healthok.Visibility change — decision flagged
The manifest's
public: falsemakes the App installable only on the owner account (GitHub routes org install attempts back to the personal installation; the Install App chooser listed only dezren39). To complete the planned org installs the App was made public (Advanced → Make public). Consequence to note: any GitHub account can now install it on their own repos; such deliveries are signed with the App's webhook secret, so the Worker would accept and queue foreign repos. Recommend the drainer enforce the owner allowlist (the reconcile plan is already owner-scoped). Flip back to private is one click in Advanced if ever desired — but org installs require it to stay public.Installations — 16/16, all "All repositories" (verified via App JWT)
dezren39 177385379 · 3d0d 177388121 · Cloistered 177388175 · chip-rom 177388265 · chiplox 177388292 · Clustered-Computing 177388628 · IDEXX-UTDesign 177388652 · SpongeScribe 177388683 · developing-today 177388707 · developing-today-fork 177388738 · developing-today-forks 177388761 · dezren3 177388791 · dezren39-dev 177388812 · drewry-pope 177388833 · sleep-atomic 177388857 · twitter-base 177388879.
Sweep-only (no admin): NixOS, vertexinc — covered by reconcile polling as designed.
Webhook proof (timestamps CDT, 2026-10-31)
2bd5d85e-d5a4-11f1-82c5-8d7b8483596f(dezren39/openclaw-personal) 22:28:25 → Response 202 from the Worker (signature verified with the synced secret).Laptop helper servers (form :8766 / capture :8765) shut down. Session notes: the laptop network flapped repeatedly tonight (ERR_NETWORK_CHANGED) — several page reads and one install confirmation were lost to it, all recovered by retry; drewry-pope's first install attempt actually completed despite the error page (installation 177388833).
dezren39 commentedon Nov 1, 2026
Update (2026-10-31 23:12 CDT): the drainer owner allowlist flagged as owed at App creation is done — #3326, PR #3328 (merged
72e347609), deployed to the laptop, and proven live with a hand-injected foreign webhook (DRAIN-DROPlogged, no clone attempted, drain Result=success). The public-App foreign-install exposure is closed at the drainer; details on #3326.33 remaining items
developing-today commentedon Nov 8, 2026
state: handed_off
queue: repo-health-code-pilot
task: 12101 (triage)
by: lane code-k (worker repo-worker-code)
at: 2026-11-08
Picked up and revalidated fresh: issue #3215 is OPEN, no hard-stop labels (
held,user/owned,do-not-merge,do-not-auto-editabsent). Execution-lane assignment returnedopencode— handed off to the laptop opencode claimant, which launches the bound session that drives task 12101 to its terminal state. This lane does not work it in parallel.dezren39 commentedon Nov 8, 2026
Partial verification — staying open (2026-11-07 done-task-only audit, tracking: #7016).
Queue task(s) [8202] were marked done, and some work did land:
No merged change closes this issue's full scope, so the done status is not treated as completion. Remainder re-queued as triage task 11667.
developing-today commentedon Nov 8, 2026
🤖 Agent note · handed off
developing-today commentedon Nov 8, 2026
🤖 Agent note · handed off