Skip to content

fix(core): tell Code Mode models built-in tools stay top-level - #57627

Merged
rekram1-node merged 4 commits into
devfrom
codemode-builtins-wording
Oct 29, 2026
Merged

rekram1-node merged 4 commits into
devfrom
codemode-builtins-wording

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Fixes #53623

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Problem. Weaker models (Gemma 4, MiMo v2.6 Flash) write await tools.read(...), tools.shell(...) and similar inside execute. The Code Mode prompt says the listed tools "cannot be called directly" under ## Available tools, and nothing says the model's other tools are still normal tool calls.

Change. Only the Code Mode prompt in packages/core/src/codemode/instructions.ts. It doesn't name any specific tool, since which tools are direct and which are in Code Mode is configurable.

 # Code Mode

-Use the `execute` tool to call the tools listed below. They cannot be called directly, and neither can `search`. Both only work inside code you pass to `execute`.
+The tools listed below, and `search`, cannot be called directly: they only work as JavaScript inside the `execute` tool. Every other tool you have is a normal tool call: call it directly, never inside `execute`.

-The catalog is partial. Inside `execute`, use `search(...)` to find a tool, then call it by the `path` in the result. Do not guess tool names.
+The catalog is partial. Before calling a tool that is not listed below, find it with the global `search(...)` function. Do not guess tool names. Each result's `path` is the complete call expression: write it out as code, for example `await tools.ns.tool_name({ ... })`. Do not use it as a key such as `tools[path]`.

-## Available tools
+## Code Mode tools (call only inside `execute`)

With a complete catalog the first line drops ", and search," and the catalog line stays "The catalog is complete. Do not guess tool names."

Evidence. codemode-routing eval (https://github.com/anomalyco/evals/pull/18) on dev 97fd7ac, Gemma 4 31B, Gemma 4 26B and MiMo v2.6 Flash, 6 reps. Trials that called a built-in inside execute:

catalog dev this PR p
no MCP server 36/72 0/72 <0.001
4 tools (complete) 25/198 1/198 <0.001
84 tools (partial) 16/198 2/198 0.001

Success rates are unchanged (439/468 vs 438/468), and mean tokens drop from 63k to 40k because the recovery loops go away. Haiku 5.5, DeepSeek V4.1 Flash, MiniMax M3 and MiMo v2.6 Pro (2 reps) show no regressions. An earlier version of this PR that listed read, edit, shell and grep by name scored the same (3/468), and a shorter wording ("Call your other tools directly as usual") did worse (10/468). Without "cannot be called directly", a few models (Gemma 4 26B, Haiku 5.5) started calling Code Mode tools or search as direct tool calls; with it, a rerun on OpenRouter dropped those from 8 to 2 while keeping the built-in fix (dev 39, this PR 4).

Cache effect. The Code Mode block is part of the instruction baseline, which is re-rendered on every request, so existing Code Mode sessions miss the prompt cache once after deploy.

Out of scope. Gemma 4 26B with a large partial catalog still guesses tool names instead of searching in about a third of trials.

How did you verify your code works?

  • bun typecheck in packages/core: pass
  • bun test test/codemode in packages/core: 23 pass
  • The eval runs above

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks right overall. On the base, the new test fails with the old Unknown tool 'read'. error and passes with this PR. codemode tests (1416) and core codemode tests (23) pass, and both packages typecheck. I couldn't check that the models themselves behave differently: that needs a real provider, and the simulated model only replays scripted calls.

Public API: the published @opencode/codemode package changes behaviour. Unknown tool paths whose first segment is one of the hardcoded names now fail with an opencode-specific message instead of the fuzzy "Did you mean" suggestion. Nothing is added or removed from its API, and opencode's SDK, server, config and plugin API are unchanged.

One concern, inline: the hardcoded list of built-in tool names in packages/codemode would be better passed in from core, using the tools core keeps out of Code Mode (codemode: false).

Related: #51949, #55087

Comment thread packages/codemode/src/tool-runtime.ts Resolved
@rekram1-node
rekram1-node force-pushed the codemode-builtins-wording branch from 12c308e to 949f41f Compare October 29, 2026 01:40
@rekram1-node
rekram1-node merged commit aaa0d7c into dev Oct 29, 2026
15 checks passed
@rekram1-node
rekram1-node deleted the codemode-builtins-wording branch October 29, 2026 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Code Mode" instructions are confusing Gemma-4-31B

1 participant