fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route)

Summary

On 2026-11-05 ~22:55–23:56Z, 89 queue-lane sessions died on the Antigravity Claude/GPT quota wall ("Individual quota reached") even though their route (workhorse) had a healthy, serving member the whole time. In the same window, workhorse-primary chose openai-plan/gpt-6-luna 3,154 times and openai.main logged 3,149 successes through the workhorse route. The pools and the route work — sessions that fresh-draw rotate fine. Sessions that had already drawn the walling member were trapped: their sticky pin kept re-selecting it, and the failover walk never re-drew the untried healthy member of the origin pool, ending in no-candidates.

Drew's ruling when this was surfaced as "a wall": unless every single plan is tapped out, a session dying on one pool's quota instead of auto-choosing another provider is an error in our stack. It was not every plan: OpenAI plan was at ~87–90%, AG Gemini ~26–28%, DeepSeek API routes open, Mistral Vibe plan leg live.

The error, identified

Text: Individual quota reached. Please upgrade your subscription to increase your limits. Resets in 81h35m49s. — classified AI.Error.RateLimit by the AI layer.
Source: provider antigravity-direct, model claude-sonnet-5-5 (the Antigravity Claude/GPT bucket), drawn via workhorse → workhorse-primary → sonnet-plan (sonnet-plan's only live door; claude-code is disabled).
Reset math checks out: first logged at 2026-11-05T23:24:33Z with "Resets in 81h35m49s" → 2026-11-09T09:00Z, exactly the AG Claude/GPT bucket reset recorded in the route config on 2026-11-03.
591 occurrences in opencode.log that evening; 1,605 failed route_attempt rows against antigravity-direct in the 22:50–23:56Z window; 3,750 all-time for this error text.

Evidence trail (laptop opencode.db, 2026-11-05)

Sessions ses_e5e5213cb1dffehQZ9zzrdAcTU (opencode lane anomalyco#3026, created 22:55:00Z on route workhorse), ses_e5e51f8557cffe9remJl4d8rtR, ses_e5e51ea9667ffd2MPFISZCgsXb — all end idle_outcome=failed, cost 0.

Decision/attempt trace for the first session, first step:

workhorse (ordered) → workhorse-primary (round-robin) → sonnet-plan (round-robin) → drawn antigravity-direct/claude-sonnet-5-5 → failure RateLimit, retryable=1.
Failover fired once: workhorse → workhorse-ordered → deepseek-flash-plan → drawn mistral-plan/mistral-small-latest → mistral/mistral-small-latest → failure QuotaExceeded, retryable=0 (Mistral API pool had walled at 22:45Z). deepseek-flash-plan's other doors were legitimately out (cline-pass + opendesign pool-quota-exhausted, Go doors in credential cooldown).
Next resolution: workhorse → no-candidates. Exclusion detail lists workhorse-primary and workhorse-ordered as already-tried, and Luna's leaf excluded by two credential-cooldown-until reasons — while Luna's serving credential (route_attempt.credential_id for openai.main) had no health rows at all that evening and 4,765 successes on the day.
Every subsequent step repeats: workhorse-primary → sticky → sonnet-plan → sticky → AG → fail → no-candidates. ~10s cadence inside the runner, then ~4min claimant warm retries ("failed to warm session"), until drain failure. 763 no-candidates decisions across 74 sessions in the window.
Meanwhile route_health recorded 996 cooldown-start rows for antigravity-direct (reason retry-after, until = +24h each) — the target kept being attempted while its own cooldown was active; each new failure just re-wrote the cooldown.

Current state: self-quieted by 2026-11-06 ~03:00Z (claimant stopped re-warming the dead sessions; fresh draws rotate to healthy members). The defect is latent — the AG bucket stays walled until 2026-11-09T09:00Z, and any session that sticks to a member whose pool walls mid-flight re-enters this trap.

Narrowed candidate mechanisms (all in the resolution path)

Sticky pin is not broken by quota failure / cooldown. packages/core/src/model-route.ts — the sticky choice (chooser ~L2867, stickyTarget ~L1983) re-selects the session's pinned member on every step. Evidence says the pin survived both the target's quota failure and its recorded cooldown. Check what steady()/continuity validation actually consults (cooldown keyed by target+credential+location vs the AG target's credential-less attempts: AG route_attempt.credential_id is NULL for all 3,750 attempts).
Failover tried set is scoped to the whole resolution tree, not the leaf. In the step's failover resolution, both ancestor routes of the two attempted leaves were already-tried, so the walk could not re-enter workhorse-primary to draw its untried, healthy Luna leaf. nextTarget readiness is leaf-granular (openLeaves: ready && !tried per leaf, ~L2637), but the tried set accumulated during the chain (~L2964) appears to mark ancestor positions at each level (~L2758). Failover should exclude leaves already attempted, not pools partially attempted.
A leaf's unavailable verdict may be judged by its worst credential. Luna's leaf was excluded citing cooldowns of credentials that were not serving it (one already expired, one ~19h out — the quota-gated alternate credential cycles 30-min gates in route_health). A multi-credential leaf should be unavailable only when every credential is unavailable.
Secondary: cooldown ignores the provider-stated reset. AG said "Resets in 81h35m49s" (in the error body); the recorded cooldown was +24h. model-route.ts ~L465 caps cooldown at quotaResetAt only when a reset was extracted — the AG reset lives in message text and apparently is not extracted.
Secondary: inconsistent quota classification. The identical condition (pool exhausted) classifies as RateLimit/retryable=1 for Antigravity but QuotaExceeded/retryable=0 for Mistral. retryable drives the runner's same-target Retry outcome (step.ts SessionRunnerRetry.isRetryable), so AG failures invite ~10s same-target retries against an 81-hour wall. Quota-exhaustion should be uniformly failover-eligible and never same-target-retryable on a short cadence.

Acceptance

A session whose drawn member returns quota-exhausted is served by another eligible member of its route (same step or next step) whenever one exists — including a member of the same pool it just drew from.
A quota failure breaks the session's sticky pin on the failed member; the member is not re-attempted while its quota/cooldown state says it is out.
Regression test: two-member pool, wall member A mid-session, assert the session completes on member B without a no-candidates resolution.
Related doctrine: ensure(core+openchamber): repeated errors of ANY class remove a model from rotation for escalating time, or until its expected resolution #1014 (a declined retry cascades while eligible candidates remain), feat(routes): round-robin weights as priority tiers — rotate within a tier, fall through when it is exhausted #1122, design(core): one quota signal, three consumers — shared QuotaSnapshot and weight-layer contract for #566/#567/#568 #587.

Acceptance criteria

When a drawn pool member returns a quota-wall RateLimit, failover re-draws untried healthy members of the origin pool before declaring no-candidates; a session is never stranded while any member of its route can serve.
A sticky pin to a quota-walled member is released (or bypassed) on the wall, so pinned sessions fail over instead of re-selecting the walled member.
A regression test reproduces the traced 2026-11-05 scenario (pinned session, walled drawn member, healthy untried member) and passes; route_attempt logging shows the healthy member being drawn.

Merge pull request #1807 from developing-today-forks/mock-provider-ha…

Mock-provider test harness landed: PR #1807, merged as 1a24c0ce.

What it is (packages/core/test/):

fixture/mock-provider.ts — scripted outcomes per target, optionally per credential: text, tool call, HTTP failure, transport failure (ENOTFOUND/ECONNREFUSED), never-settling timeout. HTTP failures are built by the production RequestExecutor.httpFailure itself, so classification is byte-identical to real provider answers — including the captured Antigravity wall (429, plain text, Resets in 81h35m49s.) and the Mistral insufficient_quota JSON.
fixture/mock-route-driver.ts — drives a routed step through the real resolution path: the resolver's eligibility filter over ModelRoute's own predicates, ModelRoute.nextTarget picks, and the runner's outcome accounting (quota classification → observeQuotaFailure, failureHint, bench probes, credential failover before route failover, first-token timeout in the runner's exact error shape).
model-route-mock-provider.test.ts — 22 pass, 1 skip, 0 fail; affected suites together 252 pass / 1 skip / 0 fail; typecheck clean.

Passing today (route altitude): the acceptance scenario — two-member pool, serving member walls mid-session, step completes on the survivor; AG-shaped wall fails over in-step and is not re-attempted next step; quota failure breaks an ordered route's pin; failover excludes leaves, not pools; auth/DNS/timeout rotation; per-credential quota switching on one leaf; stated-reset cooldowns (stated 2h benches 2h; the 81h reset caps the bench at 24h while the provider quota gate holds to the stated reset).

Ready to un-skip when the router fix lands: one test, test.skip with its body intact — "route chains: a wall inside a nested route does not strand the parent route's untried members." The incident's fatal resolution happened in the resolver service's cascade (parent tried-set seeded by target ref from the chain), one altitude above the route driver; it needs a resolver-service harness and is the acceptance core of the fix itself. Every route-level scenario passing against current code localizes the defect to that cascade, consistent with the mechanism analysis above.

Feeder triage — gap confirmed on current main

Agent: issue sweep (issue-sweep-rotating, Make feeder)
Checked: 2026-11-06T07:30Z, fresh gh calls verified twice. Repo main c81b9d816c5c16f552b8e08cac8025758ba61f62 (verified x2); this issue OPEN with no labels and 0 comments before this note (verified x2) — no live queue task, no work/* label, not triage-spent.

What I checked (reads only, gh api raw — no builds/tests):

packages/core/src/model-route.ts on current main (3136 lines, blob 685dabde460cf24086164ade34744f2c0818d668):
- Sticky chooser at lines 1983-1984: const sticky = state.sessions.get(sessionID); if (sticky !== undefined && candidateIndexes.includes(sticky)) return sticky — the sticky return consults only candidate membership, no quota/cooldown state in this chooser.
- Sticky decision path at lines 2867 and 2898: a priorSticky in the pool is honoured as decision reason sticky.
- openLeaves at lines 2640-2655 is leaf-granular (ready && !tried per leaf), while the exclusion detail at line 2758 reports already-tried — the ancestor-vs-leaf scoping of the tried set (issue mechanism 2) is exactly what the lane must pin down in code.
- Cooldown cap at line 465: state.cooldownUntil = quotaResetAt !== undefined && quotaResetAt > now ? Math.min(until, quotaResetAt) : until — matches secondary mechanism 4: no extracted reset, no cap.
The issue's own evidence is primary for the runtime failure: 89 sessions dead 2026-11-05 22:55-23:56Z, 763 no-candidates decisions, 996 cooldown-start rows for the walled target, while the same route served 3,149 successes through its healthy member.

Verdict: still live / latent — the code sites named by the issue are present in substantially the cited form on current main, and nothing on this issue or in its labels shows prior triage or a fix in flight. This is not already-fixed and not a duplicate on the evidence above (related doctrine: #1014, #1122, #587 — related, not duplicates).

What "done" looks like (from the issue's Acceptance):

A session whose drawn member returns quota-exhausted is served by another eligible member of its route whenever one exists — including a member of the same pool it just drew from.
A quota failure breaks the session's sticky pin on the failed member; the member is not re-attempted while its quota/cooldown state says it is out.
Regression test: two-member pool, wall member A mid-session, assert the session completes on member B without a no-candidates resolution.

Enqueue follows in the same pass (triage packet, queue repo-health-opencode-pilot); a second agent note will name the task id, or, if the queue backend outage (fleet-wide since ~04:44Z, code#6450) blocks the enqueue, the prepared packet will be recorded in the fleet action outbox instead of being lost.

🤖 Agent note · triage packet prepared, enqueue outboxed

Agent: issue sweep (issue-sweep-rotating, Make feeder)
Session / ID: issue-sweep-rotating run 2026-11-06 01:24 CST
Found via: feeder rotation (prev rotating repo dezren39/darwin-nix → developing-today-forks/opencode)
Task: Enqueue a repo-task/v1 triage packet (kind triage, priority 70, dedupe repo-health:v1:triage:developing-today/opencode:issue:1806 — queue registry pre-move repo name, as validated) into queue repo-health-opencode-pilot. Packet: hidden_files/packet-opencode-1806.json (validated repo-task/v1 this run).
Evidence: feeder triage comment fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806 (comment) (gap confirmed on main c81b9d816c5c; sticky chooser model-route.ts:1983-1984, openLeaves:2640, exclusion :2758, cooldown cap :465), updated for PR test(core): mock provider harness for route failover scenarios (#1806) #1807's localization (MERGED 1a24c0ce443a, main 8cf16377a7c7, model-route.ts blob unchanged): defect sits in the resolver service cascade; the skipped nested-route-chain test is the acceptance core.
Enqueue status: fleet queue enqueue failed this run — session proxy read failed (failed to fill whole buffer), exit 3 (queue backend outage since ~04:44Z, code#6450; queue health also timed out). The fully prepared enqueue is recorded as OB-2026-11-06-004 in the fleet action outbox for the processor to execute when the backend recovers; the queue/queued label is applied with this note so the fleet monitor's reconciliation also sees the owed enqueue. No task id exists yet — do not treat any id as assigned until the enqueue returns one.

🤖 Agent note · correction to the enqueue status in comment 6679749543

The enqueue attempt's true terminal result was a 35s timeout (EXIT:124, no task id returned), not the "exit 3 / session proxy read failed" reported in my previous note — that error text came from a faulty interim delivery and is retracted. What stands: queue health timed out three times this run (EXIT:124), the packet is validated, no task id exists from this run, and the prepared enqueue is outboxed as OB-2026-11-06-004 (its last_error is corrected to match). The enqueue is idempotent via its dedupe key if the timed-out attempt created the task server-side.

🤖 Agent note · filed

Agent: issue-sweep-rotating
Session / ID: action-outbox-processor run 2026-11-06 02:04 CST — enqueue executed by the outbox processor from prepared outbox entry OB-2026-11-06-004; queue task 10911
Found via: filed this run
Task: Triage task 10911 enqueued to repo-health-opencode-pilot for issue fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806 (route failover traps sessions on a quota-walled member); opencode queue signal refreshed (generation 1512).
Expectation: A queue worker claims task 10911 and triages issue fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806 per the packet's acceptance criteria.
Links: Queue task 10911 · outbox entry OB-2026-11-06-004

Triage verdict — task 10911 (queue repo-health-opencode-pilot)

Agent: repo queue worker (repo-worker-opencode, lane opencode-a, direct execution)
Checked: 2026-11-06T08:14–08:22Z, fresh gh + laptop origin/main reads. Main 8cf16377a7c77fb6e2af013faecbb942fba7fc45 (PR #1807 merged). Issue OPEN, labels at claim: only queue/queued — no held/user-owned/do-not-* hard stops.

Which of the five mechanisms are load-bearing

Mechanism 3 — CONFIRMED load-bearing (resolver altitude). packages/core/src/model-resolver.ts:657-665 builds credentialByProvider as one active credential per provider, and the ready filter at :800-828 excludes a leaf when that single credential is in credential-cooldown or quota-blocked — it never asks whether another stored credential for the same provider could serve. Credential failover (:949+) only runs after a target is picked and loaded, so it cannot rescue a leaf excluded at eligibility. This matches incident step 3 exactly: Luna's leaf excluded by credential-cooldown-until reasons while its serving credential had no health rows that evening.

Mechanism 2 — load-bearing at the cascade altitude, untested. Within one definition the tried set is leaf-granular (model-route.ts openLeaves :2640-2655). Above ModelRoute.nextTarget, the only exclusion path is the resolver cascade: triedRefs keyed by provider/model ref (model-resolver.ts:639 cascadeRefKey; accumulated :1090-1093; parent seed :1130-1134) plus visitedRoutes (:1128-1129, origin route pre-visited :1178). That path has no test coverage: the mock-route-driver states in its header that it deliberately does not reproduce the cascade, and the acceptance test is test.skip at packages/core/test/model-route-mock-provider.test.ts:395. Whether cascade seeding independently strands untried leaves — beyond mechanism 3's eligibility exclusion — is exactly what the routed fix must prove with a resolver-service harness.

Mechanism 1 (sticky pin) — NOT load-bearing at route altitude. PR #1807's harness passes the pin scenarios on current code (quota failure breaks an ordered route's pin; a sticky choice is honoured only among ready candidates — model-route.ts:1983-1984, :2867/:2898). Contributing factor for the cross-process re-attempts in incident step 4: route health benches are in-memory only (model-route.ts:267), so a bench does not survive a process restart (claimant warm retries ran in fresh processes). Recorded for the fix PR to disposition as a possible separate issue; not in the fix scope.

Mechanism 4 (cooldown vs stated reset) — NOT load-bearing. model-route.ts:465 caps a quota bench at an extracted quotaResetAt, and #1807's tests prove the stated-reset behaviour (a stated 2h reset benches 2h; the 81h reset caps the bench at 24h while the provider quota gate holds the leaf to the stated reset).

Mechanism 5 (quota classification inconsistency) — NOT confirmed load-bearing. AG's RateLimit/retryable=1 plausibly fed the 996 cooldown-start rows via same-target retries, but in-step failover eligibility already works on current code per #1807; per the packet scope, classification is not changed unless the fix's harness proves it required.

Routed fix

Follow-up task 10919 (kind small_fix, queue repo-health-opencode-pilot, dedupe repo-health:v1:fix:developing-today/opencode:issue:1806, packet packet-opencode-1806-fix.json): build the nested-chain scenario in the existing resolver-service harness (packages/core/test/model-resolver.test.ts), un-skip/port the :395 test, fix multi-credential eligibility (a leaf is unavailable only when every credential that could serve it is unavailable), and fix any cascade seeding defect the harness proves. Acceptance is the issue's own: a session whose drawn member quota-walls is served by another eligible member of its route — including the same pool — with no no-candidates resolution.

Merge pull request #1827 from developing-today-forks/queue/10919-1806…

agent-note:v1 — state: completed via PR #1827 merged 2026-11-06; task 10919 complete. No Codex findings on #1827; merge rested on diff/PR description vs triage route (mechanism 3) + PR-recorded 237 resolver/route tests pass / typecheck clean.

agent-note:v1 — state: completed via PR #1827 merged 2026-11-06; task 10919 complete. No Codex findings on #1827; merge rested on the PR description vs triage route (mechanism 3) + PR-recorded 237 resolver/route tests pass / typecheck clean.

Drew directive 2026-11-06 08:10 CST — CRITICAL, high urgency

"sessions shouldn't error should just move to next provider this is a critical high urgency bug"

Evidence from this morning's orchestration pass: 50 of 60 laptop OpenCode sessions in an errored state — 25 terminated on claude-code provider.auth (not signed in), 24 on the Antigravity individual-quota wall, 1 model-unavailable. In each class the session died on its leaf's error instead of moving to the next candidate in its route.

Requirement: within a session attempt, a candidate's error (any availability class) advances to the next untried candidate; the session errors only when all candidates are exhausted. The availability classification/benching (#1832) is landed — this issue's remaining resolver work (tried-set, sticky pin, worst-credential, the skipped harness cascade test) is the session-level half and is now the fleet's top-priority fix. A fix lane has been dispatched against this issue.

Merge pull request #1853 from developing-today-forks/fix/1806-session…

Session-level cascade — the remaining class gate is fixed (PR #1853, merged `5e9b135f`)

Per Drew's 2026-11-06 08:10 CST directive ("sessions shouldn't error should just move to next provider"), the last resolver-remainder hole on this issue is closed.

Root cause (packages/core/src/session/runner/step.ts, the step failover gate): failover was admitted only for retryable failures, Authentication, and QuotaExceeded. Any other target-indicting class killed the session with untried candidates remaining — even though the broader health-accounting gate still benched the leaf for future sessions. Proven by a step-level probe on pristine main 9a936e0924 (healthy fallback attached): Authentication/QuotaExceeded/ProviderInternal → Failover; unclassified InvalidRequest → session died, fallback() never called (a provider 404 "model not found" classifies as unclassified InvalidRequest — this morning's "model unavailable" session); target-side Timeout → session died, fallback() never called.

Fix: a cascadeFailure predicate (Timeout, UnsupportedOperation, NoRoute, unclassified InvalidRequest) admitted to the same failover gate; classified InvalidRequest (context-overflow, payload-too-large — the only classifications that exist) stays out, since size recovery owns the request and every candidate would reject it identically. Tried-set/exhaustion semantics unchanged (resolver-owned): each candidate attempted at most once; the session errors only when every candidate is exhausted, with the failing candidate's error.

Already-landed remainder, verified not redone: mechanism 3 via #1827; sticky-pin/tried-set behavior is covered by the passing resolver + mock-driver suites (the pin is honored only among ready positions; the ported #1806 nested-wall scenario passes in model-resolver.test.ts).

Tests (real counts): 6 new step-level tests in session-step.test.ts (auth/quota/InvalidRequest/Timeout → Failover; context-overflow → no failover; exhaustion → single error) — file now 40 pass / 0 fail (34 baseline). Affected suites: 566 pass / 0 fail across 11 files; adjacent runner suites 66 pass / 0 fail; typecheck + oxlint clean.

Not deployed: production still runs 22c0a6d; this lands on Drew's next nixos-rebuild switch, after which the standing tripwire (re-run the fastlane probe legs + watch the errored-session count) verifies it live. Leaving this issue open for that production verification.

claude-code 404 — root cause found, production GREEN on `5e9b135` (2026-11-06)

Root cause. The stale external plugin @openchamber/opencode-claude@1.3.8 in the global config's plugins array. It starts an in-process loopback proxy, folds its provider settings (baseURL: http://127.0.0.1:<port>/v1) into the config-seated native claude-code provider, and registers a session.model.request hook that rewrites every claude-code request's baseURL to that proxy. The native provider then composed POST http://127.0.0.1:<proxy>/v1/v1/messages and the proxy answered its own plain-text 404 Not Found. Proven by recording-listener test (listener received nothing while sessions 404'd in ~5 ms — the hook overrode the configured baseURL). The fork binary, the credential, the config baseURL, and the UA were all innocent: builds 22c0a6d and 5e9b135f failed identically because the plugin is external to both. Hardening follow-up filed as #1858.

Cutover. Production now runs fork main 5e9b135f via the fastlane drop-in (/nix/store/mldzbxia2djz7wy2s0kkzjk5qrhwv7d5-opencode-2.0.30+5e9b135-wrapped); rollback path to the 22c0a6d build is recorded in the drop-in. The 404 persisted identically on the main build before the plugin removal — confirming the defect was not the binary — and cleared the moment the plugin was removed.

Green evidence (session level). Both acceptance probe legs return PROBEOK with error: None, finish: stop, served by claude-code/claude-haiku-5-5#low; route_attempt rows show provider claude-code, outcome success, ~1.25 s real Anthropic round trips (vs 2.7–7 ms loopback 404s before). Fresh verification after the probe fix (sessions ses_e5e4e782140ffeVAcgIAgCO1Ev direct, ses_e5e4e781a8effefSI0fW0VSxBl route): result: GREEN on both legs, SUMMARY direct: True route: True.

Stopgap + durable fix. Stopgap: removed the single plugin line from ~/.config/opencode/opencode.jsonc (a symlink to the code repo's .opencode/opencode.jsonc); backup at /tmp/opencode.jsonc.bak-plugin-20261106; service restarted. Durable: developing-today/code PR developing-today/code#6538 merged (merge commit 9c06d0c8a4e20629dbcd883f9f8338f54daed1d5); the merged file blob 885d480582 is byte-identical to the production-proven live file, and the primary checkout has been fast-forwarded to the merge — so the pending anomalyco#5647 switch can no longer resurrect the plugin via this config.

Probe fix. /tmp/fastlane-probe.py mis-parsed the current server message shape (role/parts vs the returned type/content), printing NOT-GREEN while healthy. Parser updated to accept the type/content shape; the run above is its verbatim GREEN output. Backup at /tmp/fastlane-probe.py.bak-20261106. The post-switch tripwire can trust the script again.

This issue stays open for the remaining post-switch production verification and the errored-session census.

Reopened 2026-11-06 — the closure was mechanical, not ordered

Drew states plainly: he never ordered or made this closure, and this issue was being held open deliberately for post-cutover production verification.

What the record shows (verified 2026-11-06 via the timeline API + PR readback):

PR #1827 merged at 2026-11-06T11:56:54Z. Its body opened with "Fixes fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806" — but the PR fixed mechanism 3 only (multi-credential leaf eligibility; queue task 10919).
The issue's close event landed at 2026-11-06T11:56:57Z, 3 seconds after the merge, actor dezren39, no commit association — the signature of GitHub's auto-close-on-merge keyword, executed under the account the fleet merges with. No human close decision was made; the "Fixes" keyword in a partial-fix PR body closed the whole issue.

Restated requirement (Drew, 2026-11-06)

The error trace on this issue shows a single error blocking entire route pools. The required behavior:

When a real route fails, block that individual route (mark it unavailable) and keep failing over route-by-route until a working route serves the session.
A session must not error, and a pool must not be taken down, while untried candidates remain.

Attached here

The post-cutover production verification + errored-session census for the 5e9b135 cascade-fix build (fastlane cutover 2026-11-06 ~09:38 CST, claude-code green since the plugin removal) follows in the next comment. This issue stays open until that verification is reviewed.

Production verification — post-cutover errored-session census (2026-11-06)

Scope: production opencode (laptop) running fork 5e9b135 (the #1853 cascade-fix build) via the fastlane drop-in since the cutover 2026-11-06 15:38 UTC (09:38 CST), claude-code green since the plugin removal. Census read from the server's own store (session_v2 / session_message / route_attempt) at ~17:25 UTC. Method note: headline counts were read twice and agree; an earlier automated pass produced conflicting figures (75 vs 89 sessions) and was discarded — every number below comes from direct queries cross-checked at read time.

Sessions created since the cutover

89 total (84 top-level, 5 child)
Terminal outcomes: 83 succeeded · 2 failed · 4 still running (no idle outcome yet)

The 2 failed sessions — NOT the #1806 defect shape

Both were single-model pinpoint test prompts ("Reply with exactly: OK") with the model pinned at creation to one direct provider model — no route pool involved, no failover candidates existed:

ses_e5e4e3eb4ffffepRSajRwAZET7 — pinned mistral-plan/mistral-small-latest → provider.no-route "Model unavailable: mistral-plan/mistral-small-latest" at resolution.
ses_e5e4e3e4872ffeFcijpTMB2nkw — pinned openai-plan/gpt-5.5 → provider.no-route "Model unavailable: openai-plan/gpt-5.5" at resolution.

Both target pools were quota-walled at the time. A completeness sweep over all sessions (any creation date) found these same 2 as the only idle-failed events since the cutover.

Route attempts inside those 89 sessions

2,430 attempts: 2,402 success · 14 failure · 14 timeout
All 88 attempt-bearing sessions recorded at least one success; zero sessions exhausted their attempts without a success, and zero died on a first failing route with untried candidates left.
18 sessions show the positive cascade signature: a failed/timed-out attempt followed by a success on a different provider/model.
Only 3 assistant messages carried an error object; all were provider.transport "Session WebSocket closed", and all 3 sessions still completed successfully.

Concrete examples

ses_e5e4e44c13cffeXClDUJu2HhWb (route workhorse): attempts hit openai.main/gpt-6-luna Transport failure → mistral/mistral-small-latest QuotaExceeded → opencode-go.popesdrew/muse-spark-1.3-contributor QuotaExceeded → openai.main/gpt-6-luna success. Three failures across three providers; session completed (idle: succeeded).
ses_e5e4e327218ffeW6PhVdCOddQp (route workhorse): vibe-direct/mistral-small-latest RateLimit failure, failed_over_to = openai.main/gpt-6-luna, then served; session completed (idle: succeeded).

Verdict

On the 5e9b135 build, in the ~1 h 50 m since the cutover, the record shows the behavior Drew required: individual routes fail, get marked, and the cascade moves to the next candidate; no session has errored with untried pool candidates remaining, and no pool has been taken down by a single error. This issue stays open for Drew's review of this census — it is not closed by this comment.

Navigation Menu

Repository navigation

fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806

Summary

The error, identified

Evidence trail (laptop opencode.db, 2026-11-05)

Narrowed candidate mechanisms (all in the resolution path)

Acceptance

Acceptance criteria

Feeder triage — gap confirmed on current main

Triage verdict — task 10911 (queue repo-health-opencode-pilot)

Which of the five mechanisms are load-bearing

Routed fix

17 remaining items

Drew directive 2026-11-06 08:10 CST — CRITICAL, high urgency

Session-level cascade — the remaining class gate is fixed (PR #1853, merged `5e9b135f`)

claude-code 404 — root cause found, production GREEN on `5e9b135` (2026-11-06)

Reopened 2026-11-06 — the closure was mechanical, not ordered

Restated requirement (Drew, 2026-11-06)

Attached here

Production verification — post-cutover errored-session census (2026-11-06)

Sessions created since the cutover

The 2 failed sessions — NOT the #1806 defect shape

Route attempts inside those 89 sessions

Concrete examples

Verdict

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(core): route failover traps sessions on a quota-walled member — sticky pin + fallback never re-draws the origin pool (89 sessions died 2026-11-05 while Luna served 3,149 successes in the same route) #1806

Description

Summary

The error, identified

Evidence trail (laptop opencode.db, 2026-11-05)

Narrowed candidate mechanisms (all in the resolution path)

Acceptance

Acceptance criteria

Activity

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Feeder triage — gap confirmed on current main

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Triage verdict — task 10911 (queue repo-health-opencode-pilot)

Which of the five mechanisms are load-bearing

Routed fix

17 remaining items

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Drew directive 2026-11-06 08:10 CST — CRITICAL, high urgency

dezren39 commented on Nov 6, 2026

Session-level cascade — the remaining class gate is fixed (PR #1853, merged 5e9b135f)

developing-today commented on Nov 6, 2026

claude-code 404 — root cause found, production GREEN on 5e9b135 (2026-11-06)

dezren39 commented on Nov 6, 2026

Reopened 2026-11-06 — the closure was mechanical, not ordered

Restated requirement (Drew, 2026-11-06)

Attached here

dezren39 commented on Nov 6, 2026

Production verification — post-cutover errored-session census (2026-11-06)

Sessions created since the cutover

The 2 failed sessions — NOT the #1806 defect shape

Route attempts inside those 89 sessions

Concrete examples

Verdict

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Session-level cascade — the remaining class gate is fixed (PR #1853, merged `5e9b135f`)

claude-code 404 — root cause found, production GREEN on `5e9b135` (2026-11-06)