Repository navigation
pm-dispatch: POST /v1/sessions and /v1/tasks answer HTTP 502 — daily upstream sync dead since 2026-11-04 #6992
Description
Problem
The pm-dispatch service answers HTTP 502 to both of its write endpoints, and has for at least four consecutive days. Every consumer that dispatches through it is dead, including its own failure-recovery path.
Evidence (verified fresh 2026-11-08T00:25Z, gh run logs)
Jules Daily Upstream Sync (OpenChamber) in developing-today-forks/openchamber, scheduled runs on main:
- 2026-11-04 run 41828572424:
pm-dispatch-start: pm-dispatch POST /v1/sessions failed (HTTP 502); failure-queue stepPOST /v1/tasksalso 502. - 2026-11-05 run 42028456274: identical 502 pair.
- 2026-11-06 run 42216683583: identical 502 pair.
- 2026-11-07 run 42367237603: identical 502 pair.
Setup and checkout pass; the dispatch step is the first failure. Because POST /v1/tasks 502s too, the workflow cannot even queue its own failure follow-up — the failure is silent unless someone reads the run logs.
Impact
- The daily upstream sync of developing-today-forks/openchamber has not run since 2026-11-03 (the Nov 3 run failed differently, on an Actions staging error that has not recurred; the workflow has since moved to local
./.github/actions/pm-dispatch-*actions, which load fine). - Tracked on the OpenChamber side as main CI failure: .github/workflows/jules-daily-upstream-sync.yml (run 41633212191) developing-today-forks/openchamber#1450 (queue task 7543, now at its attempt cap) and the workflow's own tracker chore(deps): automated upstream sync failing developing-today-forks/openchamber#1570.
- Any other pm-dispatch consumer hitting
/v1/sessionsor/v1/tasksin this window presumably failed the same way.
Acceptance
- Root cause of the 502s identified (which hop answers 502: proxy, aggregator, or upstream session service) and recorded on this issue.
-
POST /v1/sessionsandPOST /v1/taskssucceed again, proven by a green scheduledJules Daily Upstream Sync (OpenChamber)run (or an equivalent live dispatch). - On that proof, main CI failure: .github/workflows/jules-daily-upstream-sync.yml (run 41633212191) developing-today-forks/openchamber#1450 and feat(mcpx): schemaval core — JSON Schema 2020-12 validator + dialect policy (#1482) #1570 can close.
Filed by the openchamber queue worker (lane openchamber-b) escalating the repeated pattern, per the resume instruction recorded on openchamber#1450 (2026-11-05): "if dispatch 502s recur, escalate the repeated infrastructure pattern rather than editing this repository."
Activity
developing-today commentedon Nov 8, 2026
CODE-2221
dezren39 commentedon Nov 8, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 8, 2026
🤖 Agent note · done
Triage verdict: not reproducing on current state (2026-11-08T01:12Z), fault hop narrowed.
Verified fresh from the laptop, twice (local + public paths):
pm-dispatch.serviceis running (bun,src/server.ts) and listening on 127.0.0.1:4879 and the tailnet address. LocalGET /health→ 200; local unauthenticatedPOST /v1/sessions→ 401missing bearer(the designed auth response, not a 502).https://dispatch.security.cabviapm-tunnel.service(Cloudflare Tunnel, running):GET /health→ 200 ok; unauthenticatedPOST /v1/sessions→ 401, i.e. the request reaches the service.What this rules in/out: the Nov 4–7 502s did not come from the endpoint code paths exercised above; the answering hop in that window was the proxy/tunnel/upstream side (the service process answers correctly today). Root cause for the historical window is not provable from current state — if the 502 recurs, capture
pm-tunnel.serviceandpm-dispatch.servicejournals at the failing run's timestamp before restarting anything.Ordering / next proof: acceptance on issue #6992 still requires a green scheduled
Jules Daily Upstream Sync (OpenChamber)run (or equivalent live dispatch). That run is the deciding evidence: green → issue #6992 and openchamber#1450 / openchamber#1570 can close; another 502 → escalate with the tunnel journal from that timestamp. Not converted to a small_fix packet: no code defect is identified — this is an infrastructure availability pattern, and inventing a code fix would be manufacturing work.dezren39 commentedon Nov 8, 2026
🤖 Agent note · triage findings for task 11437
POST /v1/sessions/POST /v1/tasks.Which hop answers 502: the Cloudflare edge, when the tunnel has no live connector
The dispatch application never emits 502. Verified live 2026-11-07 ~19:00 CST from two networks (sandbox + laptop):
GET https://dispatch.security.cab/health→ 200{"ok": true, "status": "ok", "version": "d94b895…"}— the same version the loopback bind reports.POST https://dispatch.security.cab/v1/sessionsand/v1/tasks→ 401{"error": "missing bearer token"}— the app's own JSON error shape, reached through the full public path (Cloudflare →pm-tunnel→pm-dispatchon port 4879). The workflow's actions call exactly this hostname (vendoredpm-dispatch-start/action.ymlline 143).pm-dispatch.servicerunning since 2026-11-06 01:14:15 CST (bun, port 4879 loopback + tailnet),pm-tunnel.servicesince 01:14:11 CST. Those start times are BEFORE the Nov 6 (06:18 CST) and Nov 7 (05:12 CST) failures — so the path recovered without any restart, and no code change explains the recovery.In the failure windows, the requests never reached the laptop. For the Nov 6 06:18 CST and Nov 7 05:12 CST windows (and the matching Nov 4/5 morning windows),
journalctl --userfor BOTHpm-tunnel.serviceandpm-dispatch.servicecontains no entries at all — no request, no error, no reconnect. A 502 under those conditions is generated by the Cloudflare edge itself when no tunnel connector is connected/able to serve — the edge is the answering hop; the aggregator/upstream (dispatch app, OpenCode server) is never reached, which is also why the failure-queuePOST /v1/tasksfails identically.Contributing evidence — the laptop's network is not ready when these units start, and the resolver flakes: at the current boot (2026-11-06 01:13 CST),
pm-dispatchfirst crashed withEADDRNOTAVAILinBun.serve(server.ts:663 — the tailnet address100.99.113.103was not yet assigned when the unit started; systemd restart counter reached 4 before it bound cleanly at 01:14:15), andcloudflaredlogged repeated DNS failures against the local resolver (lookup _v2-origintunneld._tcp.argotunnel.com on 127.0.0.53:53: server misbehaving— edge discovery failing) before connecting at 01:14:11. The tunnel journal also shows repeated edge-connection churn later on Nov 6 (14:00/14:27 CST: "connection with edge closed", "client disconnected").Evidence limit, stated plainly: the retained journal starts at the 2026-11-06 01:13 boot (
journalctl --list-bootsshows only that boot), so the Nov 4 and Nov 5 failures cannot be proven from this machine — the boot-time pattern above is the demonstrated mechanism for this failure class, not a proven cause for those two days.What remains (acceptance)
Jules Daily Upstream Sync (OpenChamber)run — NOT yet proven; the path is up now but nothing was changed, so a recurrence at the next boot/network disruption is likely. This is the outstanding acceptance item.Routed fix (follow-up task enqueued from this triage)
Harden the pm-stack laptop units against network-not-ready startup and silent tunnel death, in
developing-today/code(nixos/services/pm-stack+pm-stack/dispatch): pm-dispatch should not crash-loop on a missing tailnet bind (wait/retry for the address, or bind after tailscale readiness), pm-tunnel should be resilient to the local resolver failing at start, and an OUTSIDE health probe ofhttps://dispatch.security.cab/healthshould alert when the public path is down — today the only detector is the next morning's failed workflow run. Task 11437 completes astriaged.dezren39 commentedon Nov 8, 2026
🤖 Agent note · correction to the task 11437 findings note above
triagedcompletion; whoever holds it now: the work is done on this issue — findings posted, follow-up task 11451 enqueued — and the task can be completed astriagedwith follow_up_task_ids [11451] without repeating the investigation.developing-today commentedon Nov 8, 2026
🤖 Agent note · handed off