[jpr5]

Refs #505

Summary

Codex CLI talks to models over the OpenAI Responses API. It routes MCP and sub-agent tools by (namespace, name), and its apply_patch tool is a freeform custom tool. aimock could not script either shape, so a Codex session against aimock failed on its first tool call. This is PR 1 of 2. Recording is PR 2.

• Emission. A fixture tool call can carry a namespace, emitted as function_call.namespace. A toolCalls entry with type: "custom" (or a customToolCall block) emits a custom_tool_call item with response.custom_tool_call_input.delta / .done events. Both work over HTTP SSE, non-streaming and WebSocket. Fixtures that use neither field produce byte-identical output.
• Matching. Function and custom tools inside namespace tools, custom tools, and tools delivered in additional_tools and tool_search_output input items are now visible to toolName and predicates. A new match.toolNamespace key selects by exact namespace, alone or paired with toolName. custom_tool_call and custom_tool_call_output history items count as tool rounds.
• Validation and errors. File loading validates the new shapes up front. At request time, every other wire rejects a custom call before writing any content, with a 500 aimock_unsupported_tool_call in that wire's own error envelope (an error frame on Realtime and Gemini Live). A malformed fixture tool call gets aimock_invalid_fixture_tool_call the same way, from toolCalls or blocks, static or factory. Fixtures that loaded and served before keep working.
• Misbehavior (feat: add deterministic model misbehavior faults #508). Custom tool calls are never fault targets and pass through faulted output unchanged. A fixture the wire cannot serve is never faulted, so it gets the same coded error and journal entry as without a misbehavior config. Schema faults and tool-unknown-name resolve a Responses call by namespace and name.
• Docs and CHANGELOG. Responses API page (shapes, event order, matching, a Codex recipe), fixtures reference, WebSocket and multi-turn pages, the write-fixtures skill, and CHANGELOG entries, including the TypeScript type changes. The README now lists the feature and links to the Responses API docs section.

Spec: https://www.notion.so/3f43aa381852819c9dc0db15a4b40523

Design notes and decisions

• Codex's MCP namespace is mcp__<server> with no trailing __. The docs follow real Codex 0.161.0 output.
• An empty namespace name gets a 400, as in the OpenAI spec (NamespaceToolParam.name has minLength: 1).
• toolNamespace matches any request tool that carries a namespace. Only the Responses API sets one from namespace tools.
• On non-Responses wires a namespace is dropped and a custom call is rejected.
• Legacy toolCalls load as on main, with warnings (legacy type: "toolCall", numeric id, non-string name, input on a function call). The new fields are strict. When non-empty blocks are served, legacy toolCalls are ignored.
• Three shapes that loaded in 1.44.0 are now load errors: an empty namespace, type: "custom" with arguments, and type: "customToolCall" in toolCalls.
• ToolCallResponse.toolCalls becomes FixtureToolCall[] (a breaking type change, noted in the CHANGELOG). AudioResponse.toolCalls stays ToolCall[].
• Custom input deltas are split on code points, so no lone surrogates.
• A malformed custom_tool_call_output becomes empty text (200, as on main).
• Invalid-arguments envelopes on Ollama, Gemini and Bedrock gain an aimock code. Existing fields are unchanged.
• Misbehavior: custom calls are never fault targets or the tool-call-id-duplicate destination. Output-clearing faults with tool=<custom name> are not applicable. stop-length-mid-tool always cuts a function call: custom calls before it complete, custom calls after it are dropped, and a custom-only fixture gets 501 not-applicable.

Wire-format sources: the OpenAI OpenAPI spec, openai-node responses.d.ts (7.30/7.31), Codex rust-v0.161.0 (protocol/src/models.rs, codex-api/src/sse/responses.rs), and captured Codex 0.161.0 requests. A trimmed responses-lite request is checked in as src/__tests__/fixtures/codex-responses-lite-request.json.

Commits

1. 9c0cf717 test: pin Responses tool-call wire output. A golden of the exact HTTP, SSE and WS bytes for tool-only, content plus tools, and ordered blocks, captured before any change.
2. 4a41f44f feat: emit namespaced function calls and custom tool calls. Fixture types, one emission path for all Responses paths, custom input deltas, coded errors on every other wire, and load-time validation.
3. 319573b7 feat: match namespace, custom and additional_tools tools. Request tool flattening, the toolNamespace key (router, journal, validation, aimock-pytest), history conversion, and 400s for malformed tool collections on HTTP and WS.
4. 047db367 fix(misbehavior): keep faults off custom tool calls and unservable fixtures. Also the namespace-aware schema lookup.
5. 16423c12 test: drift legs for namespaced and custom calls, streaming and non-streaming.
6. af56cad0 docs: Responses API, fixtures, WebSocket and multi-turn pages, and the write-fixtures skill.
7. efd33ae3 docs(changelog): the Feature Request: namespaced function calls and custom tool calls in Responses API fixtures #505 entries, including the type changes that can break code compiled against 1.44.0.
8. 5e36eb2c fix(misbehavior): judge tool-unknown-name within the call's own namespace.
9. 3acf0955 docs(readme): mention namespaced and custom Responses tool calls. One README feature bullet that links to the Responses API docs section.

Verification

Local gates

Final run on the pushed head (9 commits): prettier, eslint, typecheck, build and test:exports pass; pnpm test 13318 passed, 3 skipped (run after build, as CI does); aimock-pytest 237 passed; commitlint passes on all 9 commits.

The 9th commit changes only the README, so the test, build and typecheck results above still apply. For that commit, the prettier check, commitlint and the CLI test that reads the README passed.

Not run: the live drift legs (no OPENAI_API_KEY locally). They skip with a stated reason when the model does not produce the call.

Codex CLI 0.161.0 end to end

Run on the final tree with no misbehavior faults. Model gpt-5.5, isolated CODEX_HOME, approval_policy = "never", aimock with --validate-on-load, one aimock process per scenario. Turn 2 fixtures match on toolCallId. The MCP server is a small stdio server, so the namespace is mcp__stdiodocs.

#	Scenario	Transport	Codex exit	Observed
1	custom_tool_call apply_patch	HTTP	0	hello.txt written with the fixture's content. 2 requests; turn 2 carried the custom call output and matched. Final message "done: patch applied".
2	custom_tool_call apply_patch	WebSocket	0	Same file and content. WS turn 1 and turn 2 both 200. No HTTPS fallback.
3	namespaced function_call mcp__stdiodocs/lookup_doc	HTTP	0	The MCP server received tools/call lookup_doc with {"id":"505"}. The output came back and matched turn 2. Final message "done: MCP round complete".
4	namespaced function_call mcp__stdiodocs/lookup_doc	WebSocket	0	Same MCP call received. WS turn 1 and turn 2 both 200. No HTTPS fallback.

On main before this PR (checked before the rebase), --validate-on-load rejects the custom fixture, and Codex fails the MCP call with unsupported call: lookup_doc.

Over WS, Codex's empty warm-up response.create gets a 404 (same on main), and each run spent about 15 s before turn 1.

Red/green evidence

Core feature, on a real server, main vs this branch (measured before the rebase onto #508):

• Custom call fixture on Responses: 500 "arguments" must be a string → 200 with output_item.added, input deltas, .done, output_item.done.
• toolName on a tool offered only inside a namespace: 404 → 200, namespace on every item, over HTTP, SSE and WS.
• toolNamespace + toolName pair with the wrong namespace registered first: 404 → 200 from the right fixture.
• History custom_tool_call plus output with hasToolResult: true: 404 → 200.
• The new test files on main: 111 failed, 21 passed (the passes are the byte-identity golden and unchanged-behaviour guards).

Review found the defects below in this branch. Each one was reproduced on a real server before the fix and checked again after it, with a test that fails before and passes after.

• Custom input deltas split surrogate pairs: 4/5 → 5/5 chunking cases clean.
• Factory toolCalls with non-object entries: uncoded 500 on Responses, 200 nameless call on Chat → coded aimock_invalid_fixture_tool_call, same as static.
• Array toolCalls entry: Chat 200 function: {}, Responses uncoded 500 → coded 500 on every wire. Array block in the loader: wrong message → "must be an object".
• Mistyped custom type with no string arguments on Responses: uncoded 500 → coded error on HTTP and WS.
• Duplicate-fixture key joined with |: false shadow warning → no warning; the true duplicate still warns.
• Non-chat custom-call load warning: text false for 16 of 18 endpoints → text true for all 18.
• Load warning read ignored legacy toolCalls when blocks were served → warns only on what is served.
• FunctionFixtureBlock not exported: TS2305 in a consumer → compiles.
• Ollama /api/generate misreported unknown blocks as tool calls (400) → same 500 as /api/chat.
• /api/generate block errors journaled with a null body → body and fixture journaled; response bytes unchanged.
• /api/generate with a tool call plus an unknown block: 400 → the same 500 as /api/chat, in both orders.
• Misbehavior planner threw on an unservable block fixture: Realtime and Gemini Live unjournaled, Gemini Interactions null-body entry → skipped and journaled like a run with no config.
• Invalid JSON arguments on wires that parse them: 54 off/on mismatches (faults served 200 or journaled applied: true) → 0.
• tool-call-id-duplicate could land on a custom call: 22 violations → 0.
• Output-clearing faults with tool=<custom name>: 16/16 cleared the custom call → 0/16; reported as not applicable.
• Schema faults on a namespaced call used another namespace's schema (served valid arguments) → the call's own schema (arguments violate it).
• tool-unknown-name checked the replacement name across all namespaces: wrong 501 and a skipped _v2 name → 200, renamed within the call's namespace.
• Docs: claims about tools validation, check order, object arguments and /api/generate were probed against a real server and corrected to match.

Out of scope / follow-ups

Review raised more than this PR fixes. What is left falls into three groups: behaviour that is the same on main, work for another subject or PR 2, and minor polish in this feature area. None of it blocks Codex use. The full list is below.

A. Same behaviour on main

• WebSocket Responses does not check the shape of input. A null item or object input throws a TypeError, sent as server_error and not journaled.
• An array function_call_output is not matched by toolResultContains.
• An additional_tools developer item becomes an empty system message.
• A null entry in toolCalls makes normalizeResponse throw a TypeError at file load, not a validation error.
• A text, unknown-type or non-object entry in blocks throws a plain Error: uncoded 500 and no journal error. Since this PR only tool-call blocks get the coded error.
• On Gemini Live and Realtime the same malformed (non-tool-call) block errors are journaled as 200.
• The Gemini Live history rollback does not cover malformed-block errors, so an unanswered user turn stays in history; it also leaves the fixture match counter advanced.
• A comment says resolveFixtureBlocks requires string arguments, but it stringifies objects.
• Function-call arguments of null, a number or a boolean pass load validation, because JSON.parse accepts them.
• A function call with no arguments fails load as "not valid JSON: undefined", while the runtime treats it as {}. The same message is given for a mistyped type, without naming it.
• Invalid function toolCalls arguments on the Responses API return an uncoded 500 with journal error null.
• Gemini Interactions and Anthropic streaming send malformed tool arguments as they are, while non-streaming rejects them.
• Gemini Interactions streams when the request omits stream. A request rejected for invalid tool arguments still uses up an interaction id.
• Streamed Responses events carry no sequence_number.
• Text, function-argument and reasoning deltas can split a UTF-16 surrogate pair. This PR fixed it only for custom input deltas.
• An item_reference with no type (or type: null) is not recognised, so the next tool output merges into the previous turn.
• Ollama /api/generate rejects a text-only blocks fixture with 400; for {content, blocks:[text]} it returns content while /api/chat returns the block text; a valid tool fixture adds a misbehavior summary {applied:false}.
• Realtime: a programmatic fixture with object arguments streams no argument deltas; a function call with no arguments crashes mid-stream and is journaled as 200.
• Object toolCalls arguments make the tool-args-invalid-json fault crash with an uncoded 500.
• In the duplicate-fixture key an absent field and "" collide (this PR corrected only the comment). describeMatch reports the wrong location.
• Errors from factory clean-up are labelled "factory threw".
• FixtureFileBlock is not exported from the package entry point.
• Skill and docs: the Reasoning row of the provider matrix disagrees with the fixtures doc; the endpoint list omits openai-live; the "HTTP SSE Text Response" example does not send stream: true.
• Tests: a leftover console.log, weak assertions and console noise in older tests, and a stale comment in the Responses drift file.

B. Another subject or PR 2

• aimock-pytest pins server 1.44.0 by default, which ignores toolNamespace. Bump it at the next aimock release.
• WebSocket Responses keeps no previous_response_id state. A follow-up turn that sends only new items cannot match on earlier content, and an outputs-only input makes placeholder tool calls with an empty name in the journal. Same on main with plain function tools. Storing responses by id fixes both.
• Codex's empty warm-up response.create over WebSocket gets a 404 (on main).
• Record mode drops custom tool calls and namespace, so a recorded Codex session replays as "unsupported call" (PR 2).
• Recording a non-streaming Responses body loses the output array (PR 2).
• Codex sub-agent tasks arrive as agent_message input items, which aimock does not convert.
• The openai dev dependency (4.104.0) has no types for custom tool calls or namespace, and its ResponseStream ignores output_item.done and response.incomplete. Bump it.

C. Minor follow-ups in this feature area

Behaviour and validation:

• Any /api/v3/ request is labelled "BytePlus ModelArk" in error text, even when BytePlus is not configured.
• A non-array Gemini audio toolCalls is dropped with no error; make it a coded error.
• Inside a namespace tool, non-object and other-type entries are dropped silently, and tools with no string name are accepted.
• A falsy top-level tools counts as absent, but input[i].tools: null gets 400.
• In history items, a non-string namespace is dropped and a non-string custom input becomes "" with no warning.
• A namespace tool with no description or an empty tools array is accepted; the OpenAI spec rejects both.
• A stray input is a warning on a toolCalls function entry but an error on a toolCall block.
• The toolNamespace load warning says only Responses requests carry namespaces, but a Chat Completions tool with a namespace also matches.
• UnsupportedToolCallError does not give the toolCalls[i] index and can name an empty tool.
• A factory entry with a mistyped type and object arguments is served as a function call, while the same static fixture gets a coded 500. A factory that returns a non-object is spread with { ...raw }.
• A fixture tool-call error on /api/generate uses the OpenAI envelope while /api/chat uses Ollama's; a custom call gets 400 on generate but 500 on chat.
• aimock emits status on custom_tool_call items; openai-node's type has no such field. Verify against the real API.
• Bedrock streaming routes use the non-streaming wire label in error text.
• FixtureToolCallError, ResponsesInputItem and ResponsesToolDef are not exported.

Types and names:

• FixtureFileToolCall is one interface with optional fields, not a union.
• ResponsesToolDef has a catch-all member that stops narrowing and forces casts.
• Normalized fixture types can still hold null or numeric names and object arguments; the resolveFixtureBlocks overload types are loose.
• invalidToolArgumentsEnvelope now serves three error types, so the name is misleading.
• Realtime hasToolCalls reads the raw toolCalls, not the resolved calls, and has a redundant alias.
• isPlainObject in src/helpers.ts is another private object guard; the repo already has several (isObject in jsonrpc.ts and misbehavior.ts, isPlainObject in mcp-fakes.ts). Consolidating them is a separate cleanup.

Misbehavior:

• When a fixture cannot be served and misbehavior is skipped, no summary or log is written, even when a header asked for a fault.
• servesFixture treats any error as "cannot serve", which can hide a helper bug.
• tool= cannot tell apart two tools with the same name in different namespaces, and the journal target omits the namespace.
• Blocks rebuilt for a fault drop extra fields.
• validateFixtureMisbehavior accepts a fixture fault that can never fire on wires that reject invalid-JSON object arguments.
• The schema lookup for a non-namespaced call on a non-Responses wire, whose Chat request tools carry a namespace, is not namespace-aware.
• The docs do not say that stop-length-mid-tool drops custom calls after the cut.

Drift tests:

• isModelNotFound treats a 400 saying "does not exist" as a retired model, so a real shape rejection can show as skipped.
• The non-streaming Responses leg calls JSON.parse with no guard; the model-listing fetch is not tagged as infra; response.in_progress is dropped from the SDK expectations.

Unit tests:

• The Gemini audio legacy-compat test never checks that the audio part is present.
• A fixture-error-parity predicate looks for "PATCH" but the input is {a:1}, so it never matches.
• The custom-input chunking test does not create the mock with logLevel: "silent".
• The other-wires test has no rows for Azure streaming, the compatibility-prefix paths, or the BytePlus route.
• The byte-identity test's 3 s WebSocket deadline can flake and misses frames after response.completed.
• The emission truncation test pins exactly one delta based on same-tick timing.
• The invalid-object-arguments misbehavior test table has gaps; streaming pass-through is not tested.
• Namespaced schema-fault tests cover only missing-required; no wrong-type, extra-property, enum, WS or tool_search_output case.
• Mutant M07 (key order) survives; the fixture-loader test only checks narrowing.

Base and related PRs

This branch is rebased onto main after #508 (model misbehavior), and it integrates with the misbehavior planner as described above. #509 is still open. If it lands first, this branch will be rebased onto it.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@511

commit: b7eaf3f