feat(opencode): fall back to a configured sibling model when a provider is down by goleary · Pull Request #42 · replohq/replopencode

goleary · 2026-09-09T00:41:08Z

Do not merge. Opened so we can look at the actual size of doing model fallback inside opencode instead of around it.

What this does

When a step's LLM call fails with a status the coordinator lists (or never reaches a provider at all), and the same-provider retries are spent, the step swaps to the mapped sibling model and continues. The failing provider is marked degraded for the configured cooldown, so later steps in the same sandbox start on the healthy route instead of paying the retries again. The assistant message is updated to name the model that actually answered.

Config comes from REPLO_OPENCODE_FALLBACK_CONFIG, which the coordinator already injects into every sandbox (apps/cloudflare-agent-coordinator/src/lib/opencode-fallback-config.ts in andytown). Nothing in this fork read it before. No env var means no behaviour change.

Also fixes a caching bug seen in prod: applyCaching marked an empty trailing text part with cache_control, which Anthropic rejects (cache_control cannot be set for empty text blocks). An empty tail now falls through to the message-level marker.

Shape of the change

File	Lines	What
`src/session/fallback.ts` (new)	+120	Parse config once, `qualifies(error)`, `next()`, degraded-provider map with cooldown, `healthy()` chain walk with cycle guard
`src/session/retry.ts`	+28 / -11	`policy()` gains a provider getter, an optional same-provider attempt cap, and a `fallback` hook that runs once retries are exhausted or the error is non-retryable
`src/session/processor.ts`	+41 / -5	Resolves the fallback model via `Provider.Service`, streams with `ctx.model` instead of the input model, rewrites `providerID`/`modelID` on the assistant message, logs `[model-fallback]`
`src/session/prompt.ts`	+7 / -1	Each step resolves `SessionFallback.healthy()` before `getModel`
`src/provider/transform.ts`	+4	Empty-text guard
tests	+315 / -3	`test/session/fallback.test.ts` (config, qualification, cooldown, cycle), a retry-policy handoff test, and an end-to-end processor test: two 503s from the test LLM server, third call answered by the fallback provider, message stamped `fallback/fallback-model`, `test` marked degraded

What happens when both routes are down

The coordinator map is bidirectional for the Claude transport pair (openrouter/anthropic/claude-sonnet-5 <-> anthropic/claude-sonnet-5). That cannot loop:

Inside one step the number of swaps is capped by maxFallbackAttempts (2 in the coordinator config). With both routes down the sequence is: OpenRouter fails, 3 same-route retries at 2 s / 4 s / 8 s, swap to direct Anthropic, 3 retries, swap back to OpenRouter (second and last swap), 3 retries, then the error surfaces as an ordinary failed turn. Roughly 45 s of waiting, then it stops. Set the cap to 1 to skip the bounce back.
Across steps the healthy-route walk keeps a visited set. When every route in a cycle is cooling down it returns the model that was asked for, so the next step starts where it would have without this change and fails the same way.
Same-model retries are unchanged when no fallback config is present, so upstream opencode behaviour is untouched.

Both cases are covered by test/session/fallback.test.ts (cycle) and the retry-policy test (swap cap, fresh budget after a swap).

Decisions worth a look

Fallback lives in the retry schedule, not a plugin. The schedule already owns "what happens after a failed attempt", so the hook is one extra branch there. The old model-fallback plugin approach needed the plugin to re-run the whole step.
Per-step swap, per-sandbox cooldown. The swap decision is local to a step (bounded by maxFallbackAttempts); the cooldown is module state so the next step skips the dead provider. It is process memory, so a sandbox restart forgets it, which is fine.
Which errors qualify: only statuses listed in fallbackOnErrors, whether they arrive as an HTTP status or as OpenRouter's in-stream {code} chunk, plus retryable errors with no status (fetch failed, connection reset). Invalid prompts, quota errors, context overflow and aborts never swap. 402 is not special-cased; the coordinator lists it (replohq/andytown#27790).
Same-provider retries keep today's behaviour when maxUpstreamRetryAttempts is unset, so upstream opencode semantics are unchanged without our env var.

Partial output from the failed route

Reasoning and text parts a failed attempt already wrote are removed from the assistant message when the step switches models, so the fallback never replays another model's signed reasoning as its own. Tool parts stay, since they record effects that really happened. Covered by a stubbed-provider test that streams partial output and then fails with OpenRouter's in-stream {code: 503}.

Verification

bun run typecheck clean.
bun test test/session/fallback.test.ts test/session/retry.test.ts test/session/processor-effect.test.ts: 63 pass.
bun test test/provider/transform.test.ts: 296 pass, including a wire-level check that the OpenRouter provider never serializes cache_control on an empty text block.
bun test test/provider: one pre-existing failure (chunkTimeout raises a response stream error when SSE body stalls) that fails identically on dev.

Live run on a sandbox (2026-09-16)

Ran on one of our own internal agent sandboxes with this branch's linux-x64 build swapped in for the fleet binary. OpenRouter traffic was routed through a local stub returning 503, and the fallback map pointed the primary route at direct Anthropic. Prompts were sent straight to opencode's HTTP API on the box.

Check	Result
Startup: `s6-svc` restart to `/global/health` 200	1.2 s with this build, 1.3 to 1.5 s with the fleet binary on the same box. No startup tax; the removed harness plugin cost about 52 s at boot, this is compiled into the binary and does one JSON parse of the env var on first use
Memory	no new dependency, no new process, no plugin loader; the module is about 120 lines
Turn 1, primary route down	retries at 2 s, 4 s, 8 s on the primary, then `[model-fallback] switched model`, real completion from direct Anthropic, 19 s total (nearly all of it the three retries the coordinator config asks for)
Turn 2, same session	no retries, started directly on the fallback route, 3 s total
Stored assistant message	`providerID`/`modelID` name the model that answered, on both turns
Prompt cache	first fallback turn wrote the prefix to the new route's cache (one extra cache write), later steps read it

The box was restored afterwards: fleet binary, coordinator env, stub removed.

Review round 1 (Codex, 2026-09-09)

All four findings applied in 19e6164:

Fallback model had no retry budget. The schedule now counts attempts since the last swap, so the fallback route gets its own retries and backoff. Covered by the retry-policy test.
Cooldown was per provider. Keyed by provider/model route now, so the coordinator's Claude to GPT to Gemini chain on OpenRouter walks correctly. Covered by a test using those exact routes.
Empty-text cache fix did not reach the wire. The skip is gone; Claude behind OpenRouter now goes through the same empty-content filter as direct Anthropic. Verified by serializing through the real provider with a stub fetch.
Test isolation. Fallback tests reset module state before and after each test.

Follow-ups in andytown (not here)

replohq/andytown#27790: OpenRouter ↔ direct Anthropic pairs in both directions, 402 in fallbackOnErrors, the eval opt-out, and the pin bump to this release.
Flip the coordinator default so agent Anthropic traffic routes through OpenRouter.

🤖 Generated with Claude Code

Summary by cubic

Adds a configured sibling-model fallback inside session retries when a provider fails, and fixes two prompt-caching bugs that Anthropic and OpenRouter reject. Reads REPLO_OPENCODE_FALLBACK_CONFIG; without it, behavior is unchanged.

Fallback

After same-provider retries, listed HTTP or in-stream provider errors and retryable transport failures move the step to a mapped route with a fresh retry budget.
Failed routes cool down per provider/model, so later steps start on a healthy route.
Fallback rebuilds prompt history for the new model, including compaction; the assistant message names the model that answered, and a failed fallback leaves the requested model on the row.
Partial reasoning, text, and pending tool parts from the failed route are removed on a swap, cleared tool calls are settled so cleanup doesn't wait out their timeout, and output from earlier steps stays.
Context overflow, aborts, non-retryable statusless errors, invalid targets, swap limits, a tool that already ran, and failed part reads prevent switching, but the failed route is still marked degraded.
Config is re-read when the env var changes and cooldowns are cleared; a swap budget of zero disables the feature, including the same-provider retry cap.

Caching

Empty text blocks are no longer sent with cache markers for Claude through Anthropic or OpenRouter.
Signed OpenRouter reasoning details survive message filtering.

^{Written for commit f4f5b69. Summary will update on new commits.}

Reads REPLO_OPENCODE_FALLBACK_CONFIG, which the coordinator already ships into every sandbox, and uses it inside the session retry schedule: once same-provider retries are spent on a listed status or a transport error, the step swaps to the mapped model, marks the failing provider degraded for the cooldown, and later steps start on the healthy route. The assistant message records the model that actually answered. Also stops marking an empty trailing text part for prompt caching, which Anthropic rejects. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions · 2026-09-09T00:41:18Z

Hey! Your PR title Fall back to a configured sibling model when a provider is down (do not merge) doesn't follow conventional commit format.

Please update it to start with one of:

feat: or feat(scope): new feature
fix: or fix(scope): bug fix
docs: or docs(scope): documentation changes
chore: or chore(scope): maintenance tasks
refactor: or refactor(scope): code refactoring
test: or test(scope): adding or updating tests

Where scope is the package name (e.g., app, desktop, opencode).

See CONTRIBUTING.md for details.

github-actions · 2026-09-09T00:41:46Z

The following comment was made by an LLM, it may be inaccurate:

…-fallback

…real empty-text fix The retry schedule now counts attempts since the last swap, so a fallback model gets its own retries and backoff instead of inheriting an exhausted counter. Cooldown is keyed by provider/model route rather than provider, because the coordinator chains several OpenRouter models and one failing route must not degrade the others. Empty text parts are filtered for Claude behind OpenRouter the same way they are for direct Anthropic; the earlier skip did not survive the provider copying the message-level cache marker onto the last text part, which a wire-level test now checks. The fallback tests reset module state before each test as well as after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The Claude-via-OpenRouter branch reused the direct-Anthropic filter, which only spares a blank reasoning part when the signature sits under providerOptions.anthropic. OpenRouter carries it in providerOptions.openrouter.reasoning_details, so a signed whitespace-only thinking block was dropped on replay and the provider sent the turn back without its signature. A blank part now survives when it carries reasoning_details, verified through the real OpenRouter serializer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…retryable, validate config A route that fails after the swap budget is spent is now still put on cooldown, so the steps that follow do not walk straight back into it. Statusless errors only switch models when the provider marked them retryable; an invalid prompt or exhausted quota reported inside a stream stays on the requested model. The config is decoded from the JSON string with the schema, which now requires non-negative integers. Compaction steps take the route the step loop resolved instead of reopening the user message's model, and Claude-behind-OpenRouter detection uses the same substring the caching path does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A map value that is not "provider/model" used to produce a route with an empty provider and a model that does not exist. It is skipped now, and the single-use parse helper is folded into fallbackFor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A fallback target the sandbox cannot resolve no longer fails the next step: the prompt loop falls back to the requested model instead of dying on a config mistake. recordFailure says in its name that it writes the cooldown, route says it returns a route rather than a verdict, and ref builds the route key from a model in one place. The processor's process() no longer accepts a model it would ignore, the retry option is called retries and its zero and absent cases are documented, the retry status counts across swaps so retry ids stay unique, and 402 is no longer added to the configured status list by the code. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…y for the new model OpenRouter reports an upstream 429 or 5xx as a { code, message } chunk inside a 200 stream. The retry classifier already read that code; the fallback classifier only looked at HTTP statuses, so the most common OpenRouter failure retried and never swapped. Both now read the same JSON. On a swap the processor asks the prompt loop to convert the history again for the new model, so signed reasoning and tool metadata from the failed model are downgraded the way they are for any other model change. A swap budget of zero now also leaves the cooldown alone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goleary · 2026-09-16T23:45:07Z

/devin review

devin-ai-integration · 2026-09-16T23:45:11Z

Starting Devin Review.

…action fallbacks The assistant row now takes the model from the step that actually finished, so a fallback route that also fails leaves the requested model on the row with the error. Compaction passes the same history converter the prompt loop does, so a swap during compaction rebuilds the summary input for the new model. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Reasoning and text parts written by the failed attempts stay on the assistant message, and once the fallback finishes the step the row names the fallback model, so the next step would replay the failed model's signed reasoning as the fallback's own. On a swap those parts are now removed; tool parts stay because they record real effects. Covered by a stubbed-provider test that streams partial output, fails with OpenRouter's in-stream 503, and expects only the fallback's text to remain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The fallback config is re-read whenever its env string changes, since a harness reload rewrites process.env in place. A config with a swap budget of zero now turns the feature off entirely, including the same-route retry cap, so the coordinator can opt a project out by writing a config rather than deleting a file its env rotation never removes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The swap cleanup now removes only reasoning and text written after the step began, so output already on the message survives. A change to the fallback env var clears the cooldown map, and route() reads the config before consulting it, so a replaced or re-enabled config never steers a step from stale state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

goleary · 2026-09-17T17:27:11Z

/devin review

devin-ai-integration · 2026-09-17T17:27:23Z

Starting Devin Review.

The retry history is built before the step, so a tool the failed attempt already executed is missing from what the fallback model sees, and it could run the tool again. A swap is now declined when the failed attempts wrote any tool part, and the error surfaces as it would without fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A tool part still pending had only its input streamed and never ran, so it no longer declines the fallback; it is cleared with the rest of the dead route's partial output instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Removing a pending tool part left its entry in the processor's in-flight tool map, so cleanup waited out its 250 ms timeout for a call that would never finish. The entry is settled along with the part now, and the partial-output test asserts the step finishes inside that window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

goleary · 2026-09-17T17:46:38Z

/devin review

devin-ai-integration · 2026-09-17T17:46:42Z

Starting Devin Review.

…ig path The history converter is required, so a new caller cannot switch models with history built for the old one; tests that never switch pass a converter that fails if reached. Reading the message's parts no longer swallows a database error, which would have marked earlier output as the failed attempt's. The retry policy takes the provider as a getter only and the fallback hook gets just the error. Tests drive the config through the env var, so the module has no injection path, and qualifies is private. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

It sat on the cleanup timeout boundary and would flake on slow hosts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

goleary · 2026-09-17T17:53:12Z

/devin review

devin-ai-integration · 2026-09-17T17:53:16Z

Starting Devin Review.

devin-ai-integration

Devin Review found 1 new potential issue.

cubic-dev-ai

All reported issues were addressed across 7 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

The snapshot of earlier parts is only taken when fallback is configured, and a failed read on either side of the swap now declines the switch rather than aborting the step or treating earlier output as the failed attempt's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions Bot added the needs:title label Sep 9, 2026

goleary and others added 2 commits September 16, 2026 12:18

Merge remote-tracking branch 'origin/dev' into gabe/openrouter-direct…

c3a65f7

…-fallback

goleary marked this pull request as ready for review September 16, 2026 22:48