Skip to content

fix: verify monitoring delivery and recovery - #77

Merged
simonvanlierde merged 23 commits into
mainfrom
fix/monitoring-reliability
Nov 14, 2026
Merged

simonvanlierde merged 23 commits into
mainfrom
fix/monitoring-reliability

Conversation

@simonvanlierde

@simonvanlierde simonvanlierde commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes monitoring setup checks that could pass without proving delivery or recovery, and keeps notification credentials out of Grafana's Viewer API.

  • Backups unpause after interruption; restore checks recover markers in all five volumes and restart Grafana.
  • Smoke checks read fresh metrics, logs and traces, then require TargetDown to clear after recovery.
  • An internal relay holds notification destinations. Watchdog stops after delivery failures or lost Grafana scrapes, including newly created failure counters.
  • Browser errors contain technical metadata only; requests omit browser cookies and referrers. Export-failure signals use labeled solid/dashed/dotted strokes, and optional Cloudflare lookup errors fail explicitly.
  • Corrects password rotation and onboarding instructions, repairs linked-worktree secret scans, and hardens CI with fresh-demo and licence checks.

Tested: just check, just infra-validate, disposable smoke/restore, Viewer and notification probes, natural heartbeat failure/recovery, backend queue recovery and project-silence checks. Native Chromium confirms no cookies or Referer header. Dual independent rendered reviews verify signal styles and legends on desktop/narrow screens in light/dark themes. Regression controls reject interrupted cleanup, no-op restore, stale metrics, broken trace export and disabled buffered observation.

Not checked: production upgrades, credential rotation, full host-loss or disk-exhaustion drills, live browser ingress, or complete dashboard incident workflows. Previously exposed destination credentials need rotation and revocation during rollout.

…ility

* origin/main:
  build(deps): bump the hub group across 1 directory with 6 updates (#76)
  build(deps): bump python in /demo in the demo group (#70)
  build(deps): bump the demo group in /demo with 4 updates (#71)
  build(deps): bump curlimages/curl from 8.23.0 to 8.23.0 (#74)
  build(deps): bump grafana/alloy in /templates in the spoke group (#72)
  build(deps): bump github/codeql-action/upload-sarif (#75)
@simonvanlierde simonvanlierde changed the title fix: verify onboarding and monitoring checks fix: verify monitoring delivery and recovery Nov 14, 2026
@simonvanlierde
simonvanlierde merged commit 967cd56 into main Nov 14, 2026
7 checks passed
@simonvanlierde
simonvanlierde deleted the fix/monitoring-reliability branch November 14, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant