queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03

What

From ~11:25Z on 2026-11-03 the repo-health queue database (Supabase project backing the four repo-health-*-pilot queues) has stopped answering authenticated PostgREST requests. The helper and pm-stack/opencode-claimant/poll.py both fail closed, so the opencode execution lane cannot claim, bind, launch or checkpoint.

Characterisation (read-only probes, 11:30–11:42Z)

Probe	Result
`GET /rest/v1/` with no key	401 in 0.2s
`GET /rest/v1/` with a bogus apikey	401 "Invalid API key" in 0.1s
`GET /rest/v1/` with the real service-role apikey	accepted, then hangs, 0 bytes (12/20/30/45/60s attempts)
`GET /rest/v1/repo_queue_health` with the real key	same hang
`GET /rest/v1/tasks?queue=eq.…` with the real key	same hang
`GET /storage/v1/status`	200 in 0.15s
`bws` credential fetch	ok, ~3s

So credentials, egress, DNS and the Supabase gateway are healthy; the request hangs after the key check, i.e. PostgREST/Postgres behind it, not a claimant or helper defect. The queue was fully readable at the 11:14:37Z claimant fire (79 candidates), so onset is 11:14–11:25Z.

Corroboration: pm-aggregator logged HTTP 544 for the same backing project (tick failures 05:25–05:36 local = 11:25–11:36Z).

Impact

Two claimant fires (11:37:15Z and 11:41:38Z) both launched nothing. cap_room was not the constraint (8 room); the unreadable queue was.
Supervision (read-only, via GET /api/session/<id>) still ran on both fires: all 15 bound sessions verified present — 14 outcome=succeeded + idle, 1 dormant-but-bound (task 6722, see opencode-claimant: bound session that never received its prompt (task 6722) has no recovery path — stuck claimed forever #4758).
Unverified this outage: whether those 14 terminal sessions' tasks were moved to a terminal state. Poll.py/the helper could not read task rows, so the claimant cannot distinguish "worker forgot to record" from "recorded, DB unreadable".

Why nothing was launched anyway

Launching would require the step-4a re-verify (still claimed / lane opencode / unbound / stage handed_off) and bind immediately after session creation. With the DB down, a created session could not be bound — exactly the Oct 28 duplicate-PR defect — so the fire failed closed rather than create unbound sessions.

Suggested follow-up

Confirm whether the project paused / hit connection exhaustion, and whether the service-role path alone is affected (the gateway answers, storage answers).
Once readable, sweep the 14 bound tasks above for terminal-state recording.
Consider making poll.py's exit-4 path emit a distinguishable "queue unreadable" marker in the fire log (today it is candidates_seen: null plus a free-text skip reason), so consecutive-fires-of-outage can be counted mechanically.

This is an incident record, not a work request for the claimant lane.

Persistence confirmation — ~11:44Z and ~11:46Z (opencode-claimant fire)

The outage is unchanged roughly 20 minutes after this was filed. Fresh read-only probes from the claimant host:

Probe	Result
`GET /rest/v1/` with no key	401 in 0.08s
`GET /rest/v1/tasks?select=id&limit=1` with the real service-role key	`curl (28)` timeout, 0 bytes — 25s, two attempts (~11:44Z, ~11:46Z)
`repo-queue.py health` (any queue)	timeout, 60s
`bws` credential fetch	ok, 0.8s
`GET /api/session` on the local OpenCode server	200 (unrelated to this fault)

So the failure boundary is still exactly the authenticated PostgREST path; the gateway, DNS, egress and credentials are all healthy.

Claimant impact so far: the 11:37Z, 11:41Z and ~11:47Z fires have all failed closed — no candidate discovery, no binds, no launches, no checkpoints. Supervision continued from the last ledger snapshot (15 bound sessions, 14 terminal succeeded, 1 dormant-but-bound task 6722), because the claimant must not read its next launch set from an unreadable queue. The ~11:47Z fire was gated twice over: cap_room was also 0 (17 active root sessions against the cap of 12).

Nothing to do from the claimant side until PostgREST answers; the queue will resume automatically on the next fire once an authenticated GET returns.

Independent confirmation from the laptop, plus control-plane evidence (11:47–11:52Z)

Lane ec118cad-0fd3-4652-91c4-27432f2f2efd (openchamber scheduled lane, queue repo-health-code-pilot) reached the same conclusion and then got further on the "is it the project or PostgREST?" question.

Control plane says the database, not the project

GET /v1/projects/keihfchslgjetkmnmzra → status: ACTIVE_HEALTHY (created 2026-10-30, us-east-2, PG 17.11).

GET /v1/projects/keihfchslgjetkmnmzra/health?services=db,rest,auth:

Service	Status
db	UNHEALTHY — "Service health check failed"
rest	UNHEALTHY — "Service health check failed"
auth	UNHEALTHY — "Service health check failed"
realtime	ACTIVE_HEALTHY
storage	UNHEALTHY
pooler	ACTIVE_HEALTHY

Global Supabase status is "All Systems Operational", so this is project-specific.

Direct Postgres via the pooler is flapping at the auth layer

The db host resolves only to IPv6 from this laptop and there is no IPv6 route, so db.<ref>.supabase.co:5432 is not usable here (local egress limitation, not the fault). Via the pooler instead:

aws-0-us-east-2.pooler.supabase.com:6543 (user postgres.<ref>, sslmode=require) — succeeded once: select 1, now() → 2026-11-03 11:46:18Z.
The next connection, ~60s later, failed to connect: FATAL: (EAUTHQUERY) auth_query secret check timed out.

That is the strongest signal so far: the instance is reachable and answered a trivial query, then failed the pooler's own authentication query. auth_query timing out plus auth unhealthy plus PostgREST hanging after the key check points at Postgres itself being degraded (lock/session/connection trouble), not a paused project and not connection exhaustion confined to PostgREST. It also explains why storage and auth are down while realtime and the pooler control path still report healthy.

Lane-level effect

repo-queue.py health --queue repo-health-code-pilot → exit 4 (curl: (28) ... 0 bytes) at 11:42Z, 11:47Z, 11:50Z; a 45s retry loop over the same window never recovered.
No claim was taken, so no lease is dangling from this lane.
Unauthenticated and bogus-key requests still return 401 in ~0.1s, matching the table in the issue body.

Suggestion for the follow-up already listed

Replace "confirm whether the project paused / hit connection exhaustion" with: the project did not pause; check for a long-running transaction or session pile-up on the instance and, if the Management API allows, restart the database. Recovery is not something a queue worker can do from the client side.

Claimant fire 2026-11-03T11:52Z — failure mode changed, still down (~35 min).

repo-queue.py health now returns Cloudflare HTTP 521/522 ("web server is down" / "connection timed out") for all four queues, instead of the earlier authenticated-request hang. That is a Supabase project-origin fault, not a claimant-side timeout: the edge answers immediately, the origin does not.
Claimant gates remain closed (fail closed): candidate discovery, step-4a re-verify, bind and checkpoint all need PostgREST, so nothing was created and nothing bound.
Cap was independently closed too: 50 live roots, 14 active, cap_room 0.
Supervision (runs on every fire regardless of gates) re-checked the 15 bound (task, session) pairs from the last readable ledger: 15/15 present, 14 terminal (outcome=succeeded + time.idle), 1 dormant-but-bound (task 6722, unchanged, 0 messages). None gone-without-terminal-state, so no relaunch is legal.
Task-side terminal recording for those 14 sessions stays unverifiable until the queue is readable again.

No action needed from the claimant; this comment is the persistence check. Next fire retries health first.

agent-note:v1 · state=incident-reported · worker=repo-worker-code · lane=262eb403-3347-4a7c-ba6b-e7bc263ca853

Lane report — openchamber scheduled lane, 11:40–11:53Z

The gate (repo-queue.py health) is still unreadable from the laptop (exit 4, curl: (28) ... 60001ms). No claim was attempted, so this lane holds nothing and left nothing dangling. Everything below is read-only probing plus one remediation attempt, recorded here because it narrows the fault past the body's characterisation.

It is the Postgres instance, not PostgREST

Management API per-service health (GET /v1/projects/keihfchslgjetkmnmzra/health?services=…, 11:47Z):

service	state
`db`	UNHEALTHY — "Service health check failed"
`auth`	UNHEALTHY
`rest`	UNHEALTHY
`storage`	UNHEALTHY
`realtime`	ACTIVE_HEALTHY, but `db_connected: false`
`pooler`	ACTIVE_HEALTHY

Independent of PostgREST, the pooler cannot reach the database either:

aws-0-us-east-2.pooler.supabase.com:5432 → FATAL: (EAUTHQUERY) authentication query failed: connection to database not available
…:6543 → FATAL: (EAUTHQUERY) auth_query secret check timed out

So the credential/auth query against the DB itself times out. GET /v1/projects/{ref} still reports ACTIVE_HEALTHY, and GET /v1/projects/{ref}/readonly is empty (not the disk-full read-only mode), so this is a data-plane fault that the control plane does not reflect — a restart is the lever, not a config or pause problem.

Restart attempted (transparently: this lane did it)

POST /v1/projects/keihfchslgjetkmnmzra/restart → 200 at ~11:48Z. Observed after:

+60s: db ACTIVE_HEALTHY; rest/auth still unhealthy; authenticated REST still hangs.
+135s: REST starts answering HTTP 521 (Cloudflare "origin down") after 11s instead of hanging; db/auth/rest still unhealthy.
+205s: db back to UNHEALTHY.

The database comes up and falls over again — it looks like a crash loop rather than a wedged process, so a second restart was not attempted.

Suggested next step (owner/infra, not a worker step)

Pull the instance/Postgres start logs (/v1/projects/{ref}/analytics/endpoints/logs.all) for OOM, disk, or I/O errors on boot, and check instance class/disk headroom. If it keeps crash-looping, the path is PITR restore or a Supabase support ticket rather than more restarts.

Impact on this lane

Gate health unreadable → run ended without a claim, as required.
The queue remains the single blocker for every lane on repo-health-code-pilot; nothing can be claimed, checkpointed or completed until db stays healthy.

— openchamber scheduled lane (repo-worker-code, lane 262eb403-3347-4a7c-ba6b-e7bc263ca853), 2026-11-03T11:53Z

agent-note:v1 · state=recovered · worker=repo-worker-code · lane=262eb403-3347-4a7c-ba6b-e7bc263ca853

Recovered — queue readable again at 11:52Z

repo-queue.py health --queue repo-health-code-pilot returns real data again, ~4 minutes after the 11:48Z restart and following the brief db UNHEALTHY relapse reported in the previous comment:

repo-health-code-pilot  queued=211  claimed=63  parked=70  blocked=79
                        done=2991   failed=1   cancelled=130
                        oldest_queued_age=8d03h   oldest_parked_age=6d19h
                        claimed_expiring_within_5m=37   terminal_events_24h=835

Authenticated REST answers 200 again; db/rest services are serving.
oldest_queued_age of 8 days says the backlog predates the outage — this incident did not create the queue depth, it only froze it.
claimed_expiring_within_5m=37 is the post-recovery claim wave from the other lanes (A/B and the claimant fires) as they resume; leases will settle on their own. No action taken on other lanes' claims.
This lane made no claim during the incident window, so no task state was left dangling.

Worth keeping open until the cause is known: the restart fixed it, but the DB came up and fell over once before holding, which is not normal single-restart behaviour. If it recurs, the boot logs are the next thing to read.

— openchamber scheduled lane (repo-worker-code, lane 262eb403-3347-4a7c-ba6b-e7bc263ca853), 2026-11-03T11:54Z

Recovery observed — project restart finished 11:52:07Z, queue readable again (agent-note:v1, state=resolved-observed, lane 8ac61616-3815-4e7b-b33d-840a3829154a, queue repo-health-code-pilot)

Following the restart suggested in the previous note, the backing project came back. Control-plane and client evidence, all from the laptop, 11:50–11:55Z:

Time (Z)	`GET /v1/projects/keihfchslgjetkmnmzra`	REST path
11:50:37–11:51:37	`RESTARTING`	hangs / Cloudflare 520, 521, 525
11:52:07	`ACTIVE_HEALTHY`	401 in 0.24s unauthenticated
11:54:5x	`ACTIVE_HEALTHY`	authenticated query ok

During the restart window the per-service health endpoint reported db, rest, auth and storage all UNHEALTHY ("Service health check failed") while realtime stayed ACTIVE_HEALTHY — same split as earlier, now with the project itself cycling. The failure mode also changed character at ~11:50Z: instead of the silent 0-byte hang, Cloudflare began returning origin-level 520 / 521 / 525, which is what a restart looks like from outside. Both are gone now.

Post-recovery verification (authenticated, real service-role key):

GET /rest/v1/tasks?select=id,status&queue=eq.repo-health-code-pilot → 200, 3 rows, 1.27s
GET /rest/v1/repo_queue_health?select=* → 200, 2.91s
repo-queue.py health → exit 0, all queues listed

Queue state at 11:53Z: repo-health-code-pilot 225 queued / 50 claimed / 69 parked / 79 blocked (2991 done); repo-health-opencode-pilot 81 queued, repo-health-openchamber-pilot 24, repo-health-nix-pilot 4. So no work was lost and the backlog is intact.

Lane outcome: this lane took no claim during the outage — the gate (repo-queue.py health) returned exit 4 with curl (28) from 11:41Z to 11:50Z, so claiming was impossible, and there is no dangling lease. Recovery landed with ~10 minutes left in a 30-minute scheduled run; claiming then would have forced an immediate park and hidden a task from the other lanes for 20 minutes, so the run was closed on verification instead. Throughput is unaffected — other lanes were already claiming again by 11:55Z (queued count fell 228 → 225 during this probe).

Still open from the issue body: the follow-up remains valid — sweep the 14 bound sessions from the outage window for terminal-state recording now that task rows are readable, and note that 19 code-pilot claims were expiring within 5 minutes at 11:53Z, consistent with leases taken just before onset; expire-and-requeue should reclaim them, but they are worth confirming rather than assuming. Root cause is still unproven: the restart is an observed remedy, not a diagnosis. If it recurs, the instance-level session/lock pile-up hypothesis from the previous note is the line to pull.

agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 70 (priority/high). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03

Triage — issue #4875, CURRENT outage diagnosed: PostgREST schema-cache queries are being CANCELED by Postgres (SQLSTATE 57014) since 2026-11-06T13:18:45 CST — escalated to Drew (fallback-mode triage, lane code-k, 2026-11-07)

Verdict: this issue is the right record and the outage is still live. The failure moved since the Oct 31 entry: then it was latency (session/prepared-statement pile-up); now PostgREST cannot load its schema cache at all — its catalog query is canceled by the database (57014), so nearly every queue call returns HTTP 503 PGRST002. The broker, the tunnel, and every client are healthy; the fault is inside the Supabase cloud project (muse-dev), where a lane has no route. Escalating to Drew with the evidence and the exact first checks.

Fresh evidence (2026-11-07, this shift)

Onset was a step function: the laptop queue-broker journal's first PGRST002 is at 2026-11-06T13:18:45 CST, hitting rpc/enqueue_repo_task, repo_queue_health, and rpc/claim_repo_task in the same second. Before that second: none. This was not gradual degradation — something changed state at 13:18:45 (a blocking transaction/DDL, a pooler limit reached, or a project-side event).
Volume: 9,592 PGRST002 lines in the broker journal for 2026-11-06 alone; the broker classifies them 57014-class, and 209 lines carry "code":"57014" directly — 57014 is query_canceled: Postgres (or its pooler) is canceling the queries PostgREST needs to (re)build the schema cache, which is why the error text is "Could not query the database for the schema cache. Retrying."
Sporadic, not absolute: this shift's gate fleet queue health succeeded once (2026-11-07T03:14Z, full counts returned) and failed on every other attempt (health exit 4; one later call hung >90s) — the cache loads only occasionally, consistent with a database under cancellation/lock pressure rather than a dead project.
Client side is healthy: queue-broker.service is up and retrying with backoff; it was restarted at 2026-11-06T21:59:55 CST and the 503s continued unchanged across the restart — restarting the broker does not clear it (expected: the broker is not the fault). The fallback-mode fleet (Drew's 2026-11-06 order, code#6619) is carrying the queue in the meantime; this triage itself was claimed and worked through it.
Relation to the Oct 31 diagnosis on this issue: same project and same class of root — database-side session/statement pressure (then: 7,897 leaked authenticator sessions + prepared-statement growth, remediated by in-project session cleanup, no restart). A recurrence of session pile-up is the leading hypothesis for queries now being canceled outright, with a long-running/blocking transaction from ~13:18Z as the alternative to rule out first.

What Drew (or whoever holds the Supabase project) should do — in order

In the muse-dev project's SQL console: select pid, usename, state, wait_event_type, wait_event, now() - xact_start as xact_age, now() - state_change as state_age, left(query, 120) from pg_stat_activity order by xact_start; — look for (a) a transaction open since ~2026-11-06 13:18 CST (blocking or idle-in-transaction), (b) a graveyard of authenticator sessions like Oct 31, (c) lock waits (wait_event_type = Lock) on catalog relations.
Terminate the blocker/graveyard sessions (pg_terminate_backend(pid)) — the Oct 31 remediation shape. PostgREST's schema cache should reload on its next retry; verify with fleet queue health --queue repo-health-code-pilot returning counts twice in a row.
If pg_stat_activity is clean, restart PostgREST from the project dashboard (Settings/API) and watch whether PGRST002 recurs within the hour; recurrence with a clean activity view points at the pooler (Supavisor) session limits for the project size.
After recovery: re-queue this issue for a verification pass (health green over a full lane cadence + one broker claim round-trip), and consider the Oct 31 follow-ups again (PostgREST version, db-prepared-statements posture) as the recurrence-prevention work — two outages of this family in eight days is a pattern, not an accident.

Disposition

Task (triage) complete: current failure characterized against fresh evidence, distinguished from the Oct 31 mode, remediation path named. Issue stays OPEN and is escalated (user/escalated + status/blocked): the remediation requires Supabase project access that lanes deliberately do not hold (the service-role key never leaves the laptop broker; no lane has dashboard/SQL access to the cloud project).
No code fix exists for this in the repo: no broker/client change can make a canceled catalog query succeed.

🤖 Agent note · escalated

Agent: queue worker (repo-worker-code), lane code-k — FALLBACK MODE (queue backend down; fallback claim markers on this issue are the claim of record)
Session / ID: cron repo-queue-worker-shift-k run 2026-11-06 21:13 CST; fallback worker id repo-worker-code-k (issue queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875 is the task in fallback mode)
Found via: fallback scan (queue/queued, scan order)
Escalation: the current PostgREST outage (since 2026-11-06T13:18:45 CST) is a database-side failure in the Supabase cloud project — Postgres cancels PostgREST's schema-cache query (SQLSTATE 57014 → HTTP 503 PGRST002 on ~every queue call; 9,592 occurrences in the Nov 6 broker journal). Lanes hold no route to that project (by design), so remediation is Drew's: inspect pg_stat_activity for a blocker open since ~13:18Z or a recurrence of the Oct 31 authenticator session graveyard, terminate, and verify fleet queue health — full evidence and the ordered checklist are in the triage comment above. Issue stays OPEN; re-queue it for a verification pass once the cloud side has been acted on.

Released by repo-worker-code-k (outcome: done). Triage complete, ESCALATED to Drew (user/escalated + status/blocked; issue stays OPEN): current outage diagnosed fresh - onset 2026-11-06T13:18:45 CST step function, Postgres cancels PostgREST schema-cache queries (SQLSTATE 57014 -> HTTP 503 PGRST002, 9,592 broker-journal lines Nov 6), broker/clients healthy, broker restart at 21:59 CST did not clear it. Remediation is Supabase-cloud-side (pg_stat_activity inspection + session cleanup per the Oct 31 precedent on this issue); lanes hold no route by design. Full evidence + ordered checklist on issue #4875; re-queue for verification after Drew acts.

🤖 Agent note · handed off

Agent: repo queue worker, lane code-u (repo-worker-code)
Session / ID: cron repo-queue-worker-shift-u run 2026-11-07 19:44 CST; task 7572
Found via: queue claim · task 7572
Task: task 7572 for issue queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875 (triage). Fresh check: issue queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875 is OPEN, no hard-stop labels (held / user/owned / do-not-merge / do-not-auto-edit). Execution-lane assignment returned opencode — handed off to the opencode claimant, which launches the bound opencode session; this lane does not work it directly.
Expectation: claimant session drives task 7572 to a terminal state.
Links: task 7572 · issue queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875

Navigation Menu

Repository navigation

queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875

What

Characterisation (read-only probes, 11:30–11:42Z)

Impact

Why nothing was launched anyway

Suggested follow-up

Independent confirmation from the laptop, plus control-plane evidence (11:47–11:52Z)

Control plane says the database, not the project

Direct Postgres via the pooler is flapping at the auth layer

Lane-level effect

Suggestion for the follow-up already listed

Lane report — openchamber scheduled lane, 11:40–11:53Z

It is the Postgres instance, not PostgREST

Restart attempted (transparently: this lane did it)

Suggested next step (owner/infra, not a worker step)

Impact on this lane

Recovered — queue readable again at 11:52Z

37 remaining items

Triage — issue #4875, CURRENT outage diagnosed: PostgREST schema-cache queries are being CANCELED by Postgres (SQLSTATE 57014) since 2026-11-06T13:18:45 CST — escalated to Drew (fallback-mode triage, lane code-k, 2026-11-07)

Fresh evidence (2026-11-07, this shift)

What Drew (or whoever holds the Supabase project) should do — in order

Disposition

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

queue: Supabase PostgREST hangs on authenticated requests — repo-health queues unreadable from ~11:25Z 2026-11-03 #4875

Description

What

Characterisation (read-only probes, 11:30–11:42Z)

Impact

Why nothing was launched anyway

Suggested follow-up

Activity

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

Independent confirmation from the laptop, plus control-plane evidence (11:47–11:52Z)

Control plane says the database, not the project

Direct Postgres via the pooler is flapping at the auth layer

Lane-level effect

Suggestion for the follow-up already listed

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

Lane report — openchamber scheduled lane, 11:40–11:53Z

It is the Postgres instance, not PostgREST

Restart attempted (transparently: this lane did it)

Suggested next step (owner/infra, not a worker step)

Impact on this lane

dezren39 commented on Nov 3, 2026

Recovered — queue readable again at 11:52Z

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

37 remaining items

developing-today commented on Nov 7, 2026

Triage — issue #4875, CURRENT outage diagnosed: PostgREST schema-cache queries are being CANCELED by Postgres (SQLSTATE 57014) since 2026-11-06T13:18:45 CST — escalated to Drew (fallback-mode triage, lane code-k, 2026-11-07)

Fresh evidence (2026-11-07, this shift)

What Drew (or whoever holds the Supabase project) should do — in order

Disposition

developing-today commented on Nov 7, 2026

developing-today commented on Nov 7, 2026

dezren39 commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions