diff --git a/AGENTS.md b/AGENTS.md index 8fb138ceccb5..30fda9e8bcd4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -176,5 +176,6 @@ This repo is the Alterspective fork (`origin` = `Alter-Igor/opencode`, `upstream - **Symlinks**: Windows has no symlink privilege here, so repo symlinks check out as text placeholders. The two typecheck-blocking ones (`packages/app/src/custom-elements.d.ts`, `packages/enterprise/src/custom-elements.d.ts`) are replaced with local hardlinks hidden via `git update-index --skip-worktree` (list them: `git ls-files -v | Select-String '^S'`). ~60 asset symlinks (favicons etc.) remain placeholders — harmless for typecheck/TUI, but app/web builds serving those assets need Developer Mode enabled first. - **Small model context protection & delegation** (`packages/opencode/src/plugin/small-model-delegation.ts`): protects models with small context windows (default threshold <= 300,000 tokens, e.g. local models running on RTX 5090 / `coder-svr`) from context exhaustion. Uses a two-tier mechanism: (1) prompt guidance via `experimental.chat.system.transform` directing the primary agent to delegate broad file searches and code exploration to the `explore` subagent; (2) active tool bouncer via `tool.execute.before` blocking direct calls to `grep` and `read` in primary sessions, guiding the model with an exact `task` tool invocation example. Subagents (`parentSessionID` present, max depth 1) run unrestricted with full tool access and return concise syntheses to keep primary context clean. Configurable via `OPENCODE_SMALL_MODEL_THRESHOLD` (number of tokens), `OPENCODE_SMALL_MODEL_DELEGATION` (0 to disable), or `opencode.jsonc` `experimental.small_model_context_threshold`. Supported across Web UI, TUI, CLI, `coder-svr`, and Docker sandbox (`opencode-delegate` env passthrough). Tests: `bun test test/plugin/small-model-delegation.test.ts`. - **Tool-output secret redaction** (#26, fork-only `packages/opencode/src/plugin/secret-redaction.ts`): high-confidence credentials (OpenRouter, OpenAI, Anthropic, Stripe, GitHub, GitLab, npm, Hugging Face, AWS, Azure `AccountKey`, Google, Slack, Synapse `gpapp_`, JWT, opaque `Bearer`, URL passwords, PEM private keys) become `[REDACTED:]` in two places. (1) `Session.updatePart` (`session/session.ts`, a three-line hook) redacts every stored copy of a tool part, running, completed or failed: input arguments, output, title, metadata (the shell tool's streamed output) and error text, but not attachments. The `PartUpdated` event is durable, so this is what keeps a key out of the database and out of a shared session, with or without the fork plugins. (2) The Synapse plugin's `tool.execute.after` hook redacts the result object core hands back, including an MCP result's `content` text and embedded resource text (image and blob data untouched). The observer's diagnostics redaction uses the same patterns. Not covered: the model's own text and reasoning; the overflow file that output truncation writes to disk; the experimental v2 core runner, which stores tool results without `updatePart`; sessions brought in with `opencode import`; and the delegate box, which runs its own opencode build. Known trade-off: a model that reads a file holding a real key sees `[REDACTED:]`, and if it rewrites that file the key is lost. A placeholder such as `postgres://u:${PASS}@` is redacted too. Because stored inputs are redacted, the doom-loop guard in `session/processor.ts` (a two-line hook) compares them with `redactedCopy` of the live input. Each rule runs only when the text holds one of its literal prefixes, which keeps the cost per streamed shell chunk well under a millisecond. After every upstream sync run `bun test test/plugin/secret-redaction.test.ts test/session/secret-redaction-store.test.ts` from `packages/opencode`. +- **Observer MCP failures** (#18, fork-only `packages/opencode/src/plugin/observer.ts` and `synapse.ts`): the session retrospective's `MCP_TOOL_FAILURE` check works with opencode's real MCP tool names (`_`, `mcp/catalog.ts` `toolName`). A tool is MCP when its `tool.execute.after` result has the MCP shape (`content` items), or when it was never announced through the `tool.definition` hook, which core fires only for native and plugin tools (`tool/registry.ts`). An MCP error result never reaches `tool.execute.after` (core throws it, `mcp/catalog.ts`), so a failure is read from the failed tool part's `message.part.updated` event; only calls the observer saw start count, so a fork or compaction replaying old parts is ignored, and a call the user stopped or refused, or a permission rule blocked, is not counted. The text of a successful MCP result is not searched for error words (a log or search tool returns them as data). The old `mcp__` / `mcp` name check stays as a fallback. Not covered: an MCP call made inside code mode's `execute` tool. After every upstream sync run `bun test test/plugin/observer.test.ts` from `packages/opencode`, and recheck those three upstream facts plus the failed part's shape in the `message.part.updated` event (`part.sessionID`, `callID`, `tool`, `state.status`, `state.error`). - **Synapse context overflow** (#196, #200, fork patch in three upstream files): Synapse refuses an over-long request with HTTP 400, `error.code` and `error.reason` both `context_overflow`, and the message "exceeds the largest feasible context window" (KB-AI-021 §2). `packages/llm/src/provider-error.ts` matches that message and a JSON `"code"` or `"reason"` of `context_overflow` (never plain text); `packages/opencode/src/provider/error.ts` treats that code or reason as context overflow on both the API-call and stream paths; `packages/llm/src/protocols/openai-responses.ts` also classifies the code on the Responses-API stream path (#200). The session then compacts and retries instead of stopping on an unretryable API error. After every upstream sync run `bun test test/provider-error.test.ts test/provider/openai-responses.test.ts` from `packages/llm` and `bun test test/session/message-v2.test.ts test/provider/error.test.ts` from `packages/opencode`. - The remediation plan and full revalidation evidence live in `specs/arch-remediation-plan.md`. diff --git a/packages/opencode/src/plugin/observer.ts b/packages/opencode/src/plugin/observer.ts index dc567f43cbe2..bf5b40663ef1 100644 --- a/packages/opencode/src/plugin/observer.ts +++ b/packages/opencode/src/plugin/observer.ts @@ -14,6 +14,48 @@ export interface ToolExecutionRecord { outputSnippet?: string isError?: boolean errorMessage?: string + /** Set when the result had the MCP shape (`content` items); see `isMcpRecord`. */ + mcp?: boolean +} + +/** A raw MCP `CallToolResult`, which core hands to `tool.execute.after` for MCP tools. */ +function isMcpResult(value: unknown): value is { content: unknown[]; isError?: unknown } { + return !!value && typeof value === "object" && Array.isArray((value as { content?: unknown }).content) +} + +/** + * #18: error texts core writes when the user stops or refuses a call, or a permission rule blocks + * it (`processor.ts` abort, `core/src/v1/permission.ts`, `question/index.ts`). + */ +const NOT_A_FAILURE = [ + "Tool execution aborted", + "The user rejected permission to use this specific tool call", + "The user has specified a rule which prevents you from using this specific tool call", + "The user dismissed this question", +] + +/** #18: a `message.part.updated` event for a tool part that failed, as the observer needs it. */ +export function failedToolFromEvent(event: { + type: string + properties?: any +}): { sessionID: string; callID: string; tool: string; error: string } | undefined { + if (event.type !== "message.part.updated") return undefined + const part = event.properties?.part + if (!part || part.type !== "tool" || part.state?.status !== "error") return undefined + const sessionID = part.sessionID ?? event.properties?.sessionID + if (typeof sessionID !== "string" || typeof part.callID !== "string" || typeof part.tool !== "string") return undefined + return { sessionID, callID: part.callID, tool: part.tool, error: typeof part.state.error === "string" ? part.state.error : "" } +} + +function mcpText(result: { content: unknown[] }): string { + return result.content + .flatMap((item) => { + const entry = item as { type?: unknown; text?: unknown; resource?: { text?: unknown } } + if (entry?.type === "text" && typeof entry.text === "string") return [entry.text] + if (entry?.type === "resource" && typeof entry.resource?.text === "string") return [entry.resource.text] + return [] + }) + .join("\n") } export interface SessionAnomaly { @@ -116,6 +158,7 @@ class SessionObserverManager { private sessionStartTimes = new Map() private latestRetrospectives: SessionRetrospective[] = [] private diagnosticLogs: DiagnosticLogEntry[] = [] + private nativeTools = new Set() public logDiagnostic(entry: DiagnosticLogEntry, workspaceDir?: string) { const safeEntry: DiagnosticLogEntry = { @@ -167,12 +210,54 @@ class SessionObserverManager { this.trajectories.set(sessionId, records) } + /** + * #18: opencode names MCP tools `_` (`alterspective-rag_rag_search`), so a name + * check alone misses them. Every native and plugin tool is announced through `tool.definition` + * first; MCP tools never are. + */ + public markNativeTool(toolId: string) { + this.nativeTools.add(toolId) + } + + private isMcpRecord(record: ToolExecutionRecord): boolean { + if (record.mcp) return true + if (record.tool.startsWith("mcp__") || record.tool.includes("mcp")) return true + return this.nativeTools.size > 0 && !this.nativeTools.has(record.tool) + } + + /** + * #18: a failed tool never reaches `tool.execute.after` (an MCP error result is thrown by core), + * so the failure is taken from the stored tool part instead (`message.part.updated`, status error). + * Only a call this observer saw start is updated: a fork or compaction re-publishes old failed + * parts, which must not be counted again. A call the user stopped, refused or blocked by a rule is + * not a service failure (QA on #18). + */ + public onToolFailed(sessionId: string, callId: string, tool: string, error: string) { + const record = this.trajectories.get(sessionId)?.find((r) => r.callId === callId && r.tool === tool) + if (!record) return + if (NOT_A_FAILURE.some((prefix) => error.startsWith(prefix))) { + record.endTime ??= Date.now() + record.durationMs = record.endTime - record.startTime + return + } + record.endTime ??= Date.now() + record.durationMs = record.endTime - record.startTime + record.isError = true + record.errorMessage = error.slice(0, 200) + } + public onToolAfter(sessionId: string, callId: string, tool: string, output: unknown) { - const text = typeof output === "string" ? output : output == null ? "" : JSON.stringify(output) ?? "" + // #18: an MCP result carries its text in `content` items, not in `output`. + const mcp = isMcpResult(output) + const text = typeof output === "string" ? output : mcp ? mcpText(output) : output == null ? "" : JSON.stringify(output) ?? "" const records = this.trajectories.get(sessionId) ?? [] const record = records.find((r) => r.callId === callId) - const isError = - text.includes("Error:") || + if (record && mcp) record.mcp = true + // A real MCP error result is thrown by core and arrives through onToolFailed; the words in a + // successful MCP result (a log or search tool's text) are data, not a failure (QA on #18). + const isError = mcp + ? output.isError === true + : text.includes("Error:") || text.includes("error:") || text.includes("Failed to") || text.includes("Unauthorized") || @@ -228,7 +313,7 @@ class SessionObserverManager { } // 2. Check for MCP Tool Failures - const mcpFailures = records.filter((r) => (r.tool.startsWith("mcp__") || r.tool.includes("mcp")) && r.isError) + const mcpFailures = records.filter((r) => this.isMcpRecord(r) && r.isError) for (const failure of mcpFailures) { anomalies.push({ type: "MCP_TOOL_FAILURE", diff --git a/packages/opencode/src/plugin/synapse.ts b/packages/opencode/src/plugin/synapse.ts index 07eab7077560..ace6f35f3b3e 100644 --- a/packages/opencode/src/plugin/synapse.ts +++ b/packages/opencode/src/plugin/synapse.ts @@ -8,7 +8,7 @@ import { InstallationVersion } from "@opencode-ai/core/installation/version" import { Global } from "@opencode-ai/core/global" import { Flock } from "@opencode-ai/core/util/flock" import { FSUtil } from "@opencode-ai/core/fs-util" -import { INJECTED_LEARNINGS_LIMIT, learningStorePaths, sessionObserver, sanitizeJsonSchemaForOpenAI } from "./observer" +import { failedToolFromEvent, INJECTED_LEARNINGS_LIMIT, learningStorePaths, sessionObserver, sanitizeJsonSchemaForOpenAI } from "./observer" import { redactToolResult } from "./secret-redaction" import { EscalationTracker, declaredTier, classifyFailure, malformedToolCallFromEvent } from "./synapse-escalation" import { @@ -1815,6 +1815,9 @@ export async function SynapseAuthPlugin(input: PluginInput, options?: SynapsePlu } // AIESC follow-up (#15): completed sessions free their tracker slot. if (sessionID && event.type === "session.deleted") escalations.release(sessionID) + // #18: a failed tool call skips tool.execute.after; its stored part carries the error. + const failed = failedToolFromEvent(event as { type: string; properties?: any }) + if (failed) sessionObserver.onToolFailed(failed.sessionID, failed.callID, failed.tool, failed.error) // AIESC-02: a malformed native tool call (200 response, garbage tool name) is // invisible to the HTTP hooks; charge it here, deduped per part id. const hit = malformedToolCallFromEvent(event as { type: string; properties?: any }) @@ -1830,6 +1833,11 @@ export async function SynapseAuthPlugin(input: PluginInput, options?: SynapsePlu escalations.recordFailure(hit.sessionID, "malformed-output") } }, + // #18: every native and plugin tool is announced here; MCP tools never are, which is how the + // observer tells an MCP tool (`_`) from a native one. + "tool.definition": async (definitionInput) => { + sessionObserver.markNativeTool(definitionInput.toolID) + }, "tool.execute.before": async (toolInput, output) => { // 1. Policy Enforcement: Block raw browser scraping of corporate email / M365 if (toolInput.tool.includes("playwright") || toolInput.tool.includes("browser")) { @@ -1886,7 +1894,13 @@ export async function SynapseAuthPlugin(input: PluginInput, options?: SynapsePlu redactToolResult(output) // #95: core calls this hook with no output when a tool fails (a failed subtask, for one). // Reading output.output then threw inside the failure path and lost the error metadata. - sessionObserver.onToolAfter(toolInput.sessionID, toolInput.callID, toolInput.tool, output?.output) + // #18: an MCP tool's result is the raw MCP result, with no `output`; pass it whole. + sessionObserver.onToolAfter( + toolInput.sessionID, + toolInput.callID, + toolInput.tool, + typeof output?.output === "string" ? output.output : output, + ) }, "experimental.chat.system.transform": async (_input, output) => { // #95: these instructions are about Synapse (its gateway, on-prem tools and text tool-call diff --git a/packages/opencode/test/plugin/observer.test.ts b/packages/opencode/test/plugin/observer.test.ts index b94fe102a9b3..51e90e424856 100644 --- a/packages/opencode/test/plugin/observer.test.ts +++ b/packages/opencode/test/plugin/observer.test.ts @@ -1,5 +1,6 @@ import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test" -import { defaultDiagnosticsLogDir, learningStorePaths, sessionObserver } from "../../src/plugin/observer" +import { defaultDiagnosticsLogDir, failedToolFromEvent, learningStorePaths, sessionObserver } from "../../src/plugin/observer" +import { SynapseAuthPlugin } from "../../src/plugin/synapse" import { tmpdir } from "../fixture/fixture" import * as fs from "fs/promises" import * as path from "path" @@ -401,3 +402,112 @@ describe("observer paths (#83)", () => { expect(text).toContain("obs-83-fallback") }) }) + +// #18: opencode names MCP tools _, so the old name check never matched them, and a +// failed call never reaches tool.execute.after at all. +describe("MCP tool failures with real remote tool names (#18)", () => { + const mcpAnomalies = async (sessionId: string) => { + const retro = await sessionObserver.finalizeSessionRetrospective(sessionId) + if (!retro) throw new Error("no retrospective found") + return retro.anomalies.filter((a) => a.type === "MCP_TOOL_FAILURE") + } + + test("error words in a successful MCP result are data, not a failure (QA on #18)", async () => { + const sessionId = "obs-18-shape" + sessionObserver.onToolBefore(sessionId, "c1", "seqlogs_search", {}) + sessionObserver.onToolAfter(sessionId, "c1", "seqlogs_search", { + content: [{ type: "text", text: "app error: ECONNRESET (recovered); Error: retry timed out" }], + }) + expect(await mcpAnomalies(sessionId)).toEqual([]) + }) + + test("a call the user stopped or refused, or a rule blocked, is not an MCP failure (QA on #18)", async () => { + const sessionId = "obs-18-refused" + const errors = [ + "Tool execution aborted", + "The user rejected permission to use this specific tool call.", + "The user rejected permission to use this specific tool call with the following feedback: not now", + 'The user has specified a rule which prevents you from using this specific tool call. Here are some of the relevant rules []', + ] + errors.forEach((error, i) => { + sessionObserver.onToolBefore(sessionId, `c${i}`, "alterspective-rag_rag_search", {}) + sessionObserver.onToolFailed(sessionId, `c${i}`, "alterspective-rag_rag_search", error) + }) + const retro = await sessionObserver.finalizeSessionRetrospective(sessionId) + expect(retro?.totalToolCalls).toBe(errors.length) + expect(retro?.anomalies.filter((a) => a.type === "MCP_TOOL_FAILURE")).toEqual([]) + }) + + test("an MCP result with isError set is flagged even when its text looks clean", async () => { + const sessionId = "obs-18-iserror" + sessionObserver.onToolBefore(sessionId, "c1", "alterspective-rag_rag_search", {}) + sessionObserver.onToolAfter(sessionId, "c1", "alterspective-rag_rag_search", { + content: [{ type: "text", text: "quota reached" }], + isError: true, + }) + expect((await mcpAnomalies(sessionId)).map((a) => a.tool)).toEqual(["alterspective-rag_rag_search"]) + }) + + test("a failed call is flagged from its stored part when the tool was never announced as native", async () => { + sessionObserver.markNativeTool("read") + const sessionId = "obs-18-failed" + sessionObserver.onToolBefore(sessionId, "c1", "alterspective-rag_rag_search", {}) + sessionObserver.onToolFailed(sessionId, "c1", "alterspective-rag_rag_search", "MCP error -32001: Request timed out") + sessionObserver.onToolBefore(sessionId, "c2", "read", {}) + sessionObserver.onToolFailed(sessionId, "c2", "read", "File not found: /tmp/x") + const found = await mcpAnomalies(sessionId) + expect(found.map((a) => a.tool)).toEqual(["alterspective-rag_rag_search"]) + expect(found[0]?.detail).toContain("Request timed out") + }) + + test("a failure for a call the observer never saw start is ignored (a fork or compaction replays old parts)", async () => { + const sessionId = "obs-18-replay" + sessionObserver.onToolFailed(sessionId, "old-call", "alterspective-rag_rag_search", "Error: old failure") + const retro = await sessionObserver.finalizeSessionRetrospective(sessionId) + expect(retro?.totalToolCalls).toBe(0) + expect(retro?.anomalies.filter((a) => a.type === "MCP_TOOL_FAILURE")).toEqual([]) + }) + + test("failedToolFromEvent reads only failed tool parts", () => { + const part = { type: "tool", sessionID: "s1", callID: "c1", tool: "x_y", state: { status: "error", error: "boom" } } + expect(failedToolFromEvent({ type: "message.part.updated", properties: { part } })).toEqual({ + sessionID: "s1", + callID: "c1", + tool: "x_y", + error: "boom", + }) + expect(failedToolFromEvent({ type: "message.part.updated", properties: { part: { ...part, state: { status: "completed" } } } })).toBeUndefined() + expect(failedToolFromEvent({ type: "message.part.updated", properties: { part: { ...part, type: "text" } } })).toBeUndefined() + expect(failedToolFromEvent({ type: "session.idle", properties: { part } })).toBeUndefined() + }) + + test("wiring: the Synapse plugin's hooks feed a failed MCP call into the retrospective", async () => { + const hooks = await SynapseAuthPlugin({ + client: {} as never, + project: {} as never, + directory: "/test/dir", + worktree: "", + experimental_workspace: { register() {} }, + serverUrl: new URL("https://example.com"), + $: {} as never, + } as never) + const sessionId = "obs-18-wiring" + await hooks["tool.definition"]?.({ toolID: "bash" }, { description: "", parameters: {} }) + await hooks["tool.execute.before"]?.({ tool: "imprint_imprint_get_instructions", sessionID: sessionId, callID: "c1" }, { args: {} }) + await hooks.event?.({ + event: { + type: "message.part.updated", + properties: { + part: { + type: "tool", + sessionID: sessionId, + callID: "c1", + tool: "imprint_imprint_get_instructions", + state: { status: "error", error: "MCP error: Unauthorized" }, + }, + }, + } as never, + }) + expect((await mcpAnomalies(sessionId)).map((a) => a.tool)).toEqual(["imprint_imprint_get_instructions"]) + }) +})