Repository navigation
design(pm-stack): chaos/rotation drills — scheduled game-days (kill collector, age the feed, drop a service) verifying banners, watchdogs, and self-healing fire #1283
Description
Part of epic #1184 (PM stack coherence — wave 2, 2026-10-26).
The PM stack's failure modes are silent by default: a dead collector, an aged feed, a rejected orchestration section, or a down service have all presented as "everything is fine" at least once (#1110 being the worked example). Health pages and watchdogs only help if they actually fire — and that is only ever tested by accident. Scheduled chaos/rotation drills make it deliberate: on a cadence, a drill injects one known fault (kill the fleet-feed collector, artificially age the feed, drop one component service, corrupt one feed section), then verifies the expected signals fire — the degraded banner on the affected pages, the watchdog entry in NEEDS ORCHESTRATOR, the health page state — and files the drill result automatically as an issue comment (or a new issue when a signal does NOT fire). Drills run against the deployed stack in a maintenance window, restore everything they touched, and never run two faults at once.
Acceptance:
- A documented drill schedule exists (at least collector-kill, feed-aging, and one service-drop), runnable on demand and on cadence.
- Each drill verifies named expected signals (banner / watchdog / health state) and records pass/fail per signal.
- A failed signal files an issue automatically under epic epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184 with the drill evidence attached; a passed drill posts its result to the tracking issue.
- Drills restore all touched state (verified by a post-drill health read) and are interlocked against overlapping with each other or with a real ongoing incident flag.
Acceptance criteria
- A scheduled drill set exists (kill collector, age the feed, drop a service — at minimum these three) and each drill verifies the expected banner/watchdog/self-healing actually fires.
- A failed drill files an issue automatically under epic epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184 with the drill evidence attached; a passed drill posts its result to the tracking issue.
- Drills restore all touched state (verified by a post-drill health read) and are interlocked against overlapping each other or a real ongoing incident flag.
Activity
developing-today commentedon Oct 27, 2026
CODE-407
dezren39 commentedon Oct 27, 2026
Intake review
Related: epic #1184 (wave 2 parent), #1110 (the silent-failure worked example named in the body).
Notes: title marks this as design-first; the drill implementations can split into follow-up slices once the design (interlocks, restore verification, evidence format) is settled. Priority set low: valuable hardening, but the watchdogs it tests are already in place from earlier waves.
Risks: a drill that mis-restores state creates the silent failure it tests for — the post-drill health read and incident interlock are non-negotiable parts of the design, not polish.
Suggested approach: design note first (one page in planning/), then implement the cheapest drill (age the feed) end to end before generalising.
24 remaining items
dezren39 commentedon Nov 3, 2026
Second drill slice of this issue is implemented and open for review: #4920 (
queue/7919-1283-age-feed-drill), queue task 7919.pm-stack/aggregator/scripts/age-feed-drill.shages the feed'sgeneratedAtpastFEED_STALE_MSwhilefleet-feed-collector.timerstays active — the stale-but-alive case the merged kill-collector drill (pm-stack(drills): kill-collector chaos drill, first slice of #1283 #1823) cannot produce — polls/api/healthuntil the staleness signal fires (capturing the health payload and, best effort, thewatchdog:feedtier item on/api/attention.json), then restores the saved bytes and confirms the collector's next run makes the feed fresh.flock, incident-flag refusal, refusal on an already-stale feed or an inactive collector, and every refusal path leaves the feed byte-identical (the kill-collector drill: restore trap fires on precondition refusals — a refused drill still starts the collector (PR #1823) #2865 lesson).systemd/age-feed-drill.{service,timer}schedule it Wed 03:43 ±15 min — a different quiet window from kill-collector's Sun 04:17 — as laptop user units, where the collector/aggregator live since code#3362/pm-stack NixOS module: fleet-feed collector units are hand-written — fold service + timer into nixos/services/pm-stack #3363.test/age-feed-drill.test.ts: 7 pass / 0 fail, covering all five refusal paths, a full age → fire → restore run, and a collector rewrite mid-drill being re-aged rather than mistaken for freshness.Not done in this run: the on-demand live run against the deployed stack (packet test 3). That runs in the first scheduled/manual maintenance window after the merge; the drill always restores via trap, so a mid-drill interruption still leaves a fresh-feed path.
Remaining scope on this issue: the third slice, the service-drop drill.
dezren39 commentedon Nov 8, 2026
Partial verification — staying open (2026-11-07 done-task-only audit, tracking: #7016).
Queue task(s) [2538, 2667, 7690, 7919] were marked done, and some work did land:
No merged change closes this issue's full scope, so the done status is not treated as completion. Remainder re-queued as triage task 11561.
dezren39 commentedon Nov 8, 2026
🤖 Agent note · done (triage)
pm-stack/aggregator/scripts/kill-collector-drill.shand its tests are on main.status/in-reviewreflects it. Next action for the fleet: shepherd PR feat(pm-stack): age-the-feed chaos drill, second slice of #1283 #4920.packet valid (repo-task/v1)) and enqueue was attempted — the broker deduped it to done task 2667, the first-slice packet that owns the canonicalrepo-health:v1:fixkey for issue design(pm-stack): chaos/rotation drills — scheduled game-days (kill collector, age the feed, drop a service) verifying banners, watchdogs, and self-healing fire #1283, so no new task was created and no non-canonical key was forced. The packet is preserved athidden_filesof the goal workspace via this lane's shift records; producers can re-issue it under a remainder key if Drew's queue conventions allow. The untracked remainder is named here so it is not lost.