Skip to content

feat(pm-stack): age-the-feed chaos drill, second slice of #1283 - #4920

Open
dezren39 wants to merge 1 commit into
mainfrom
queue/7919-1283-age-feed-drill
Open

dezren39 wants to merge 1 commit into
mainfrom
queue/7919-1283-age-feed-drill

Conversation

@dezren39

@dezren39 dezren39 commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Second slice of #1283 (epic #1184), after the kill-collector drill merged in #1823.

What this adds

pm-stack/aggregator/scripts/age-feed-drill.sh — deliberately makes the fleet feed stale while the collector keeps running, the failure mode the kill-collector drill cannot reach: there the collector is dead, here the process tree looks healthy and only the data is old, so "everything is fine" is a lie nothing else contradicts.

Sequence:

  1. Interlocks (any failure = refuse, nothing touched): flock on the drill lock, incident-flag refusal, collector timer must be active (the premise is a running collector), feed must currently be fresh (an already-stale feed is a real incident).
  2. Saves the feed's exact bytes, then arms restore (EXIT/INT/TERM) immediately before the first write — the kill-collector drill: restore trap fires on precondition refusals — a refused drill still starts the collector (PR #1823) #2865 lesson from the kill-collector drill, so every refusal path leaves the feed byte-identical.
  3. Rewrites the feed's top-level generatedAt into the past (PM_DRILL_AGE_MS, default 20 min > the 15-min FEED_STALE_MS) leaving every other field intact. If the collector's ~5-min run rewrites it with a fresh stamp mid-drill, the drill re-ages it — that is the scenario.
  4. Polls /api/health until feed.ageMs crosses FEED_STALE_MS and degraded is true, saving the health payload as evidence; records (best effort) whether the impl(fleet): staleness watchdog - surface dead/stale stack inputs as NEEDS ORCHESTRATOR instead of silent decay #1148 watchdog's watchdog:feed tier item appears on /api/attention.json.
  5. Restores the saved bytes (never clobbering a fresh collector write) and waits up to PM_DRILL_RESTORE_WAIT_MIN for the collector's next run to confirm freshness via /api/health.
  6. Writes a JSON result and follows the kill-collector reporting contract: pass posts to design(pm-stack): chaos/rotation drills — scheduled game-days (kill collector, age the feed, drop a service) verifying banners, watchdogs, and self-healing fire #1283, a non-fired signal files a finding under epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184.

pm-stack/aggregator/systemd/age-feed-drill.{service,timer} — weekly user timer Wed 03:43 (±15 min), a different quiet window from the kill-collector drill's Sun 04:17, so the two never contend for the stack. The drill runs on the laptop, where the collector and aggregator have lived as user units since code#3362/#3363; no root is needed.

Tests

pm-stack/aggregator/test/age-feed-drill.test.ts runs the real script with stubbed systemctl/sleep, a stubbed health channel, and a real feed file:

bun test test/age-feed-drill.test.ts   # 7 pass, 0 fail
bash -n pm-stack/aggregator/scripts/age-feed-drill.sh   # ok

Pinned by the tests: all five refusal paths exit non-zero and leave the feed byte-identical; a real run ages → fires → restores; and a collector rewrite mid-drill is re-aged rather than mistaken for freshness.

Not run here: the on-demand live run against the deployed stack (the packet's third test) — that belongs to the first scheduled or manual maintenance-window run.

Queue task 7919 (repo-health-code-pilot).


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Ages the fleet feed's generatedAt past FEED_STALE_MS while
fleet-feed-collector.timer stays active, polls the aggregator's
/api/health until the staleness signal fires (recording the health
payload and, best effort, the watchdog:feed tier item), then restores
the saved bytes and confirms the collector's next run made the feed
fresh. Interlocks follow kill-collector-drill.sh: flock, incident-flag
refusal, refusal on an already-stale feed or inactive collector, and
every refusal path touches nothing. Weekly user timer in a different
quiet window (Wed 03:43) from the kill-collector drill (Sun 04:17).
@dezren39
dezren39 requested a review from a team as a code owner November 3, 2026 13:47
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-03T13:47:42.363097Z
We couldn’t complete this review. Comment @codex review to try again.
70c7b52 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 12 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0a96da74-058c-4b76-b8d3-edf42ad7f080
Commits

Reviewing files that changed from the base of the PR and between 5bc74ff and 70c7b52.

Files selected for processing: 4
  • pm-stack/aggregator/scripts/age-feed-drill.sh
  • pm-stack/aggregator/systemd/age-feed-drill.service
  • pm-stack/aggregator/systemd/age-feed-drill.timer
  • pm-stack/aggregator/test/age-feed-drill.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant