Repository navigation
fix(core): tell Code Mode models built-in tools stay top-level - #57627
Conversation
There was a problem hiding this comment.
The change looks right overall. On the base, the new test fails with the old Unknown tool 'read'. error and passes with this PR. codemode tests (1416) and core codemode tests (23) pass, and both packages typecheck. I couldn't check that the models themselves behave differently: that needs a real provider, and the simulated model only replays scripted calls.
Public API: the published @opencode/codemode package changes behaviour. Unknown tool paths whose first segment is one of the hardcoded names now fail with an opencode-specific message instead of the fuzzy "Did you mean" suggestion. Nothing is added or removed from its API, and opencode's SDK, server, config and plugin API are unchanged.
One concern, inline: the hardcoded list of built-in tool names in packages/codemode would be better passed in from core, using the tools core keeps out of Code Mode (codemode: false).
081ff9d to
93bdc95
Compare
7f5ecbe to
12c308e
Compare
12c308e to
949f41f
Compare
Issue for this PR
Fixes #53623
Type of change
What does this PR do?
Problem. Weaker models (Gemma 4, MiMo v2.6 Flash) write
await tools.read(...),tools.shell(...)and similar insideexecute. The Code Mode prompt says the listed tools "cannot be called directly" under## Available tools, and nothing says the model's other tools are still normal tool calls.Change. Only the Code Mode prompt in
packages/core/src/codemode/instructions.ts. It doesn't name any specific tool, since which tools are direct and which are in Code Mode is configurable.With a complete catalog the first line drops ", and
search," and the catalog line stays "The catalog is complete. Do not guess tool names."Evidence.
codemode-routingeval (https://github.com/anomalyco/evals/pull/18) on dev 97fd7ac, Gemma 4 31B, Gemma 4 26B and MiMo v2.6 Flash, 6 reps. Trials that called a built-in insideexecute:Success rates are unchanged (439/468 vs 438/468), and mean tokens drop from 63k to 40k because the recovery loops go away. Haiku 5.5, DeepSeek V4.1 Flash, MiniMax M3 and MiMo v2.6 Pro (2 reps) show no regressions. An earlier version of this PR that listed
read,edit,shellandgrepby name scored the same (3/468), and a shorter wording ("Call your other tools directly as usual") did worse (10/468). Without "cannot be called directly", a few models (Gemma 4 26B, Haiku 5.5) started calling Code Mode tools orsearchas direct tool calls; with it, a rerun on OpenRouter dropped those from 8 to 2 while keeping the built-in fix (dev 39, this PR 4).Cache effect. The Code Mode block is part of the instruction baseline, which is re-rendered on every request, so existing Code Mode sessions miss the prompt cache once after deploy.
Out of scope. Gemma 4 26B with a large partial catalog still guesses tool names instead of searching in about a third of trials.
How did you verify your code works?
bun typecheckinpackages/core: passbun test test/codemodeinpackages/core: 23 passScreenshots / recordings
Not a UI change.
Checklist