Skip to content

feat(alerts): deliver detection alerts server-side, with Expo and Web Push - #257

Merged
ralyodio merged 6 commits into
masterfrom
feat/cloud-alerts
Sep 26, 2026
Merged

ralyodio merged 6 commits into
masterfrom
feat/cloud-alerts

Conversation

@ralyodio

@ralyodio ralyodio commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #252 (and #253). Merge #253, then #252, then this. The branch now contains those commits; its own changes are the top two commits (7b3adfc, 4c45f74). The ingest-route hook is resolved against #252's rewrite: after(() => dispatchDetectionAlerts(newDetections)) runs only for rows the RPC inserted. Deduplicated repeats never alert, and two ingest route tests cover that (they fail if repeats are dispatched).

Based on #253 (fix/next16-async-params, commit c20badd). Until #253 merges, this PR's diff also shows that commit; my change is the single commit on top.

What

  1. Server-side alert dispatch (apps/web/src/lib/alerts/). dispatchDetectionAlerts(detections) (the contract's Alerts hook) loads the org's enabled alert_rules and alert_destinations. A rule fires when severity >= min_severity, server_scope is empty or contains the server, and both the rule and its destination are enabled and in the same org. Every attempt is written to a new alert_deliveries table (sent / failed / rate_limited, with the error). It never throws: a broken destination, a guard refusal or a missing provider key becomes a failed row. Until now rules were configurable but nothing sent them.
  2. Per-rule rate limit. rate_limit_per_hour counts the rule's sent + failed rows in the trailing hour. Once the limit is reached, further matches are recorded as rate_limited and not sent. A rule allows exactly N attempts per hour.
  3. Senders. Slack, Discord and webhook go through safeFetch (SSRF guard, unchanged). Detection text is escaped so a monitored host cannot <!channel> or @everyone. Email goes through Resend, which the web app already uses. PagerDuty uses Events API v2, with one incident per detection via dedup_key. Push goes to every push_subscriptions row of the org.
  4. Push senders (apps/web/src/lib/push/).
    • Expo: https://exp.host/--/api/v2/push/send in chunks of 100. Tokens that return DeviceNotRegistered are deleted. Optional EXPO_ACCESS_TOKEN.
    • Web Push: the web-push package with VAPID keys. Subscriptions that return 404/410 are deleted.
    • The payload is {title, body, url, tag}, which is exactly what public/sw.js renders. url is /org/<slug>/detections?detection=<id>, which CloudIngest's detections page scrolls to.
  5. Ingest hook. The current POST /api/ingest now selects each inserted detection and calls after(() => dispatchDetectionAlerts(inserted)), so the daemon's request is not delayed. This is kept minimal because CloudIngest (feat(web): cloud ingest v2, remediation claim/complete, live dashboard #252) rewrites this route. When merging, hook after(() => dispatchDetectionAlerts(newDetections)) after the detections RPC block; per CloudIngest its rows match AlertableDetection.
  6. Test-send route. In alert-destinations/[dest_id]/route.ts, only the default: branch changed. It used to answer "delivery not yet implemented server-side" for email, PagerDuty and push; it now calls sendTestAlert() through the real senders and returns 502 with the reason on failure. SecurityFixes owns the rest of that file.
  7. push-subscriptions route.
    • POST also accepts {kind:"webpush", subscription:{endpoint, keys:{p256dh, auth}}}. The endpoint must be https and on a known push service, and the keys must be base64url. The row is stored as keys = {provider:"webpush", p256dh, auth}.
    • DELETE accepts {endpoint}.
    • A new GET lists the caller's own devices in the org (no key material), so the page can tell whether this browser is registered for this org.
    • Expo behaviour and its tests are unchanged.
  8. PWA opt-in. The org alert settings page gets a "Browser notifications" card.
    • Enable calls pushManager.subscribe({userVisibleOnly:true, applicationServerKey}) and registers the subscription. Turn off unsubscribes and deletes the row.
    • Clear states for unsupported browsers (with an iOS home-screen hint), permission denied, and no VAPID key configured.
    • PwaLifecycle unsubscribes the browser on sign-out, so a shared browser stops showing the org's alerts. With no session left, the push service's 410 makes the server delete the row on the next send.
  9. Destination form now matches what the server delivers:
    • Added "Push (mobile app and browsers)".
    • Email asks only for recipients (comma-separated). The unused SMTP Host and From fields are gone.
    • The webhook secret is labelled as an HMAC signing secret.
  10. Docs. .env.example documents the VAPID keys and how to generate them, RESEND_API_KEY for email alerts and the optional EXPO_ACCESS_TOKEN. In apps/mobile/README.md, the sentence "Nothing sends pushes yet" is replaced (CopyAndDocs informed).
  11. Dependency. web-push@^3.6.7 plus @types/web-push (dev) in apps/web. The lockfile was updated with pnpm. pnpm also re-resolved the jiti peer in the existing vite/electron-vite entries (2.6.1 → 1.21.7); I didn't change that directly.

How verified

  • Migration applied to local Supabase (docker exec supabase_db_threatcrush psql < supabase/migrations/20260925110000_alert_deliveries.sql): table, indexes and policies created.

  • next dev -p 3441 with freshly generated VAPID keys (npx web-push generate-vapid-keys), a throwaway user/org/server and 7 destinations with 8 rules. Detections went in through the real POST /api/ingest with the user's bearer token. Observed alert_deliveries rows:

    • critical: slack/discord/webhook sent.
    • email failed: Email delivery is not configured (RESEND_API_KEY not set).
    • PagerDuty (real Events API, fake key) failed: PagerDuty HTTP 400: Invalid routing key.
    • Webhook to http://127.0.0.1:39441/internal failed: Requests to internal addresses are not allowed, and the receiver got nothing on /internal.
    • Push failed: All 1 registered devices had expired and were removed: a fake Expo token registered through the real route got DeviceNotRegistered from the real Expo API, and its row was deleted.
    • medium: only the "discord medium" rule fired. low: nothing. The disabled rule never fired.
    • Slack rule with rate_limit_per_hour = 2: the 1st and 2nd high detections were sent, the 3rd was rate_limited ("Rule limit of 2 alerts per hour reached").
  • How I exercised safeFetch-guarded delivery locally without weakening the guard. The next dev process ran with a NODE_OPTIONS=--require stub (smoke only, not committed) that replaced two things underneath the guard:

    • DNS: dns/promises.lookup answered hooks.receiver-smoke.example → 203.0.113.10, an address the guard treats as public.
    • Network: sockets to that host were routed to a local HTTP receiver.

    resolveSafeAddress/safeFetch ran unmodified, and the internal-URL destination above was still refused. The receiver captured the Slack text (mentions escaped, <url|View in ThreatCrush>), the Discord embed with allowed_mentions:{parse:[]}, and the webhook JSON with X-ThreatCrush-Signature. I recomputed HMAC-SHA256 over the captured body and it matched.

  • Real Web Push, two ways:

    • Mozilla autopush without a browser. A Node script acted as a push client over autopush's websocket, registered a channel for our VAPID key, and registered the subscription through POST /api/orgs/:id/push-subscriptions. After an ingested detection, it received the push from updates.push.services.mozilla.com and decrypted it (aes128gcm) to {title:"[HIGH] Web push smoke detection", body:"Server: web-1\nSource IP: …", url:"…/org/<slug>/detections?detection=<id>", tag:"detection-<id>"}. I then unregistered a second channel at autopush and ingested again: the result was sent to 1 of 2 devices and the dead subscription's row was deleted (404/410 cleanup).
    • Headless Chromium (Playwright's chromium-1243, persistent profile). On the settings page, clicking Enable subscribed via Chromium's push service and registered it; the status changed to "This browser receives alerts…". Chromium's endpoint host is jmt17.google.com, so I added it to the allowlist. An ingested critical detection then arrived as a real notification shown by sw.js (from registration.getNotifications()): title/body/tag match, and data.url is the detection link. The delivery row said sent to 2 of 2 devices. Turn off removed the local subscription and the server row. With notification permission denied, the card showed the "blocked" message and no button.
    • Separately, ServiceWorker.deliverPushMessage (CDP) with the server payload produced a notification with data.url set.
  • Test-send route on the dev server: Push → {"success":true}; Email → 502 Test failed: Email delivery is not configured (RESEND_API_KEY not set); PagerDuty → 502 Test failed: PagerDuty HTTP 400: Invalid routing key.

  • pnpm --filter @profullstack/threatcrush-web test: 61 files, 541 tests passed. The pre-commit hook built the CLI and web (TypeScript included) successfully.

  • Not verified:

    • Clicking a notification. I could not click an OS notification headlessly. sw.js's notificationclick opens data.url, which I confirmed is set to the detection link.
    • The "no VAPID key configured" state was not rendered.
    • A real email send: no Resend key here.

Blocked on the owner

  • Production env on the threatcrush.com box:
    • NEXT_PUBLIC_VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY, VAPID_SUBJECT. Generate once with npx web-push generate-vapid-keys and keep them stable: rotating them invalidates every browser subscription. The public key is inlined at build time, so set it before next build.
    • RESEND_API_KEY (email alerts).
    • EXPO_ACCESS_TOKEN, only if Enhanced push security is on for the Expo project.
  • Resend must allow sending from alerts@threatcrush.com (the domain already sends hello@threatcrush.com).
  • Apply supabase/migrations/20260925110000_alert_deliveries.sql to supabase.threatcrush.com.
  • Mobile FCM/APNs credentials (already listed in apps/mobile/README.md) are needed before Expo pushes reach real phones.

Decisions

  • Push audience. A push destination sends to every registered device (Expo + browser) of every member of the org. There is no per-user targeting yet.
  • Dedup across rules. If several matching rules route to the same destination, that destination gets one alert per detection.
  • What counts against the limit. Both sent and failed attempts count toward the rate limit; rate_limited rows don't. A null limit means the column default of 60.
  • Fail open on count errors. If the rate-limit count query errors, the alert is sent: a missed alert is worse than an extra one.
  • Webhook signing. Webhooks are signed as X-ThreatCrush-Signature: sha256=<hex HMAC-SHA256(secret, raw body)>. The secret itself is never sent. The CLI daemon's local webhook sends the raw secret in a header named X-Threatcrush-Signature, so the two differ.
  • Webhook payload. It is a superset of the old test payload: {event: "detection.created" | "test", title, severity, message, timestamp, url, detection?}.
  • Email source. Email is sent from ThreatCrush Alerts <alerts@threatcrush.com> to the destination's to (up to 10 comma-separated addresses). Customer SMTP settings were dropped from the form because nothing used them.
  • One public-key variable. Server and browser share NEXT_PUBLIC_VAPID_PUBLIC_KEY, the name .env.example already had, instead of adding a separate VAPID_PUBLIC_KEY.
  • Push-service allowlist.
    • fcm.googleapis.com: Chrome and Chromium-based browsers.
    • jmt17.google.com: open-source Chromium builds.
    • updates.push.services.mozilla.com: Firefox.
    • *.push.apple.com: Safari.
    • *.notify.windows.com: Edge.
    • The allowlist is enforced at registration and again at send time.
  • Retention. alert_deliveries has no retention job yet. Rate-limited rows accumulate under floods (CloudIngest's dedup reduces this).

Next 16 passes page params as a Promise. The org pages still read
params.slug synchronously, so every page under /org/[slug] handed its client
component slug=undefined and could not load the org.
The daemon -> cloud pipeline needs the web side of the v1 contract:

- Migration: detections gain occurrences/last_detected_at/dedupe_key with
  a partial unique index; ingest_detections() dedupes on
  (server, rule, source IP, minute) so resends are safe;
  ingest_hardening_findings() upserts atomically while respecting
  acknowledged/resolved; remediation_actions gains `executing` and
  claim_server_remediations() hands pending dashboard actions to the
  daemon with a 5-minute lease.
- /api/ingest: 1 MB streamed cap, per-event validation with a rejected[]
  list, one access lookup per batch, heartbeat hostname, daemon
  remediation rows, contract response; v1 bodies still accepted.
- New claim and PATCH (executing -> executed/failed, 409 otherwise) routes.
- SDK exports the contract's event/response types.
- Dashboard: detections poll every 30 s, show xN and last seen, and
  honour ?detection=<id> (highlight or pinned); remediations show
  executing/executed/failed, source and error; online/offline comes from
  last_seen (3 min); empty states say how to link a server.
A spool replay after a lost response resent remediation events and the
dashboard showed each daemon ban twice. The contract now has the daemon
attach a stable event_id; ingest_remediations() inserts with ON CONFLICT
DO NOTHING on a partial unique (server_id, metadata->>'event_id') index and
the skipped rows count as deduplicated. Events without event_id (older
daemons) insert as before.
@socket-security

socket-security Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedweb-push@​3.6.71001001008070
Added@​types/​web-push@​3.6.41001007380100

View full report

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

ThreatCrush Security Scan

16 finding(s)

HIGH/CRITICAL: 1 | MEDIUM: 7 | LOW: 8

Severity Rule Location
HIGH secret-aws-access-key prd/0003-detect-hardcoded-secrets-before-they-are-committed-or-served.md:126
MEDIUM sql-string-concatenation .github/workflows/migrate-dev2.yml:113
MEDIUM js-open-redirect apps/web/src/app/auth/login/page.tsx:66
MEDIUM js-unescaped-html-sink apps/web/src/app/hire/page.tsx:104
MEDIUM js-unescaped-html-sink apps/web/src/app/hire/page.tsx:108
MEDIUM js-open-redirect apps/web/src/components/funding/FundingClient.tsx:97
MEDIUM js-unescaped-html-sink apps/web/src/components/GuideReader.tsx:265
MEDIUM js-uninitialized-buffer packages/scan/src/node-rules.ts:456
LOW secret-generic-credential apps/web/src/app/api/auth/refresh/route.ts:17
LOW secret-generic-credential apps/web/src/app/api/auth/reset-password/route.ts:26
LOW secret-generic-credential apps/web/src/app/api/auth/reset-password/route.ts:27
LOW secret-generic-credential PRD.md:269
LOW tls-verification-disabled prd/0004-find-dangerous-code-patterns-without-pretending-to-be-a-compiler.md:121
LOW tls-verification-disabled prd/0004-find-dangerous-code-patterns-without-pretending-to-be-a-compiler.md:122
LOW sh-remote-script-execution scripts/smoke-test.sh:72
LOW secret-aws-access-key scripts/smoke-test.sh:150

Snippets are redacted; ThreatCrush never prints matched credential material.

… Push

Alert rules and destinations could be configured in the dashboard but
nothing ever sent them: the ingest route stored detections and stopped,
and the destination "Test" button answered "not yet implemented" for
email, PagerDuty and push.

- lib/alerts: dispatchDetectionAlerts() matches enabled rules (min
  severity, server scope) against newly ingested detections, enforces
  rate_limit_per_hour per rule and records every attempt in the new
  alert_deliveries table (sent / failed / rate_limited). It never throws;
  a broken destination or missing provider key becomes a failed row.
- Senders: Slack/Discord/webhook through safeFetch (SSRF guard), email
  through Resend, PagerDuty Events API v2, push to every device in the org.
- lib/push: Expo push (chunks of 100, DeviceNotRegistered tokens deleted)
  and Web Push via web-push with VAPID (404/410 subscriptions deleted).
- Ingest calls the dispatcher via after() so the daemon's request isn't
  delayed; the test-send route uses the real senders for the remaining types.
- push-subscriptions accepts browser Web Push subscriptions (known push
  services only) and lists the caller's devices; the org alert settings
  page gets a "Browser notifications" control, and sign-out unsubscribes.
Stacking alerts on ingest v2 left two Severity unions and a string-typed
RPC row, which failed next build's type check.
# Conflicts:
#	.env.example
#	apps/web/src/app/api/ingest/__tests__/route.test.ts
#	apps/web/src/app/api/ingest/route.ts
#	apps/web/src/app/api/orgs/[id]/alert-destinations/[dest_id]/route.ts
@ralyodio
ralyodio merged commit 39aaa95 into master Sep 26, 2026
12 checks passed
@ralyodio
ralyodio deleted the feat/cloud-alerts branch September 26, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant