feat: add deterministic model misbehavior faults by jpr5 · Pull Request #508 · CopilotKit/aimock

jpr5 · 2026-10-09T06:05:13Z

Model misbehavior fixtures now exercise application failures inside valid provider responses: malformed tool arguments, schema violations, unknown tools, duplicate IDs, length stops, empty answers, refusals, filtering, and reasoning-only output. For example, times: 1 can deliver an invalid argument string on the first request and the original valid tool result on retry, with journal evidence showing which fault was prepared.

Current status: PR #508 merged at 83556d04095fb7b3fd9ec1d80d4cc01944e48685 after explicit user admin-merge approval. Reviewed head 39c48d4bee77f2e1c15f6da8fe60c53a947c5471 had 33 successful checks, 2 skips, no failures or pending checks. Local quality gates and final independent reviews are accepted. This single P1 initiative covers approved stages 1–8. Hosted OIDC/manual daily-workflow validation and scheduled recurrence remain open; no release or version bump was requested.

Provider faults and controls

Add a shared nine-fault catalog, validation, deterministic seeded rates, per-test firing budgets, and explicit applicability handling. Configuration precedence is request header → fixture → runtime test override → server baseline; an empty fault list opts out.
Render supported faults across OpenAI Chat (including Azure/OpenRouter), Responses HTTP/WebSocket, Realtime GA, Anthropic, Bedrock InvokeModel/Converse, Gemini generateContent/Vertex, Gemini Live/Interactions, Cohere, and Ollama chat. Fault and mode restrictions remain explicit; this does not promise all nine faults on every wire.
Add LLMock.setMisbehavior() / clearMisbehavior(), the runtime control/catalog routes, optional journal evidence and metrics. Invalid configurations fail during loading/programmatic registration. Preserve original fixture responses, caller count-map identities, ordinary rendering, and the accepted chaos/proxy/reset contracts.
Add SDK and recurring comparison coverage. Enabled native configuration failures propagate through the existing drift collector instead of appearing as successful skips.

Native fidelity and modeled exceptions

Local SDK decoding, captured native output, and hosted recurrence are separate evidence. The reference preserves the seven original modeled exceptions: Realtime length-stop; Anthropic reasoning-only; Developer Gemini length-stop and reasoning-only; Interactions streaming length-stop; Converse reasoning-only; and Invoke reasoning-only. Cohere object length-stop is separately modeled; its streaming TOOL_CALL/tool closure/[DONE] contract is captured. Four separately approved Vertex length-stop/reasoning-only mode cells remain modeled.

Approved valid NOT_TRIGGERED outcomes do not establish target fidelity or excuse transport, authentication, malformed-response, or local SDK failures. These exceptions do not expand unsupported wires. Subsequent observations remain mode-specific, including Gemini non-streaming reasoning-only. Hosted OIDC exchanges, manual daily-workflow validation and scheduled recurrence remain post-PR/post-merge acceptance work.

Documentation and consumer changes

Document the support matrix, scope/reset behavior, retry usage, SDK observations, native provenance, and Unreleased compatibility. Older builds can silently ignore the new fixture field. Update provider references, existing fixture/control/chaos/recording guidance, the fixture-authoring skill, and homepage capability/provenance material. FEATURE_WATCH PR #507 is merged upstream and integrated here through the accepted interface 3A handoff. Its peer scope was accepted under explicit user direction, not represented as zero-finding peer CR. Our integration passed independent review and 411 affected-consumer tests. The final C-S40 MockServer fault-capability registration passed independent review, 55 watch/HTTP tests and three actual boundary checks; it requests human review without modifying homepage/state during dry-run. Fixture-authoring instructions now direct readers to the support matrix and distinguish local support from native fidelity.

The Showcase product fix is separate: CopilotKit PR #7729. Its three responsive layouts are not counted as the three P1 fault/wire acceptance cells.

Validation and real RED → GREEN evidence

The essential retained local RED/GREEN results and reproduction commands are included below. Independent final review accepts their source-bound reuse; historical module passes are not presented as a fresh final full-suite result. No additional public artifact host is required.

Actual surface	Retained result and evidence
Local HTTP → public caller-count Map	Same selector: 8 failed → 8 passed; 101 unrelated tests deselected on each run. Three registration cells were value-checked before/after; 277 focused tests passed.
Cohere SDK and raw streaming output	Same focused cases: 5 failed / 2 passed → 7 passed. At cuts 0.25/0.5/0.75, all three cells retain incomplete arguments while changing `MAX_TOKENS`/no sentinel to captured `TOOL_CALL`/one sentinel. Six semantic mutations were killed; ordinary/misbehavior suites total 129 passes.
Actual homepage suite	Collection-time `row not found` / no tests → 43 passed after allowing whitespace around the exact row label; production HTML/scanner unchanged.
Vertex runner → actual drift collector	Same enabled/missing-token setup: four skipped → four failed; collector exit 0 → 5, with 68 local passes retained. This verifies failure propagation, not hosted authentication.

Retained provider evidence includes 548 SDK/guard passes, 550 related-control passes, 531 activation passes, and 40 captured-drift passes with 18 explicitly credential-gated skips. The built-CLI Python boundary has 199 passes and zero skips. Original P1 Chat K1, Anthropic K5, and Responses K3 UI cells are 3/3 independently accepted, including visible failure/recovery and journal evidence. These module receipts and unchanged proofs are reused within their recorded scope, not summed into a new exact-head suite claim.

Final local verification and pending acceptance

Accepted: I1–I7 have zero mandatory findings; I8 accepts unchanged boundary-proof reuse. The later unused-literal deletion has independent zero-finding confirmation.
Local quality: full formatting, full lint and explicit production/test-tree/scripts TypeScript checks passed at the final 757-path source manifest. Reuse the accepted fresh build because all 107 production/config/package build inputs are unchanged. Four original signed purpose commits plus the signed upstream merge, documentation correction, C-S40 registration and one workflow correction preserve peer history and authorship. Four commit bodies were wrapped without changing their words to satisfy commitlint; all eight signatures verify.
Composed test evidence: the unfiltered run completed 341 files with 12,614 passes, 2 failures, and 3 retained skips. Both failures were stale built-CLI prerequisites in one file. After the fresh build, the entire affected CLI file passed 33/33. Reusing the unchanged 340 passing files yields 12,616 passes, 0 failures, and 3 retained skips. This is composed evidence, not a second all-green full-suite run.
Cleanup accepted: nine owned Showcase proof fixtures disappeared with their owned in-memory aimock process; unrelated resources were preserved.
Integration accepted: current main is an ancestor (current main remains an ancestor). Our reviewed union preserves P1 provider-auth wiring and homepage claims. The added integration 411-test and C-S40 55-test results are overlapping affected-consumer evidence, not added to the composed runtime count.
Post-merge acceptance pending: hosted OIDC/native workflow acceptance and scheduled recurrence. These are not draft-creation blockers; local signed SDK and infrastructure readback are not substitutes.
Publication status: the reviewed replacement head is pushed, the PR is no longer a draft, and exact-head CI is green (33 successes, 2 skips). The user explicitly authorized admin-merge; merge is confirmed at 83556d04095fb7b3fd9ec1d80d4cc01944e48685. Essential raw evidence is embedded below. Changed external links were checked: 12 unique resources return HTTP 200; the new model-misbehavior page returns 404 before deployment; npm returns 403 to this checker. The two font preconnect origins return 404 at their root paths, while the actual stylesheet returns 200. These observations do not establish remote anchor availability.

Implementation and CI gates are accepted; explicit user admin-merge approval was received and the PR is merged. Hosted post-merge acceptance remains open. This status update performs no merge, provider call, version bump or release.

Essential retained RED/GREEN output

Commands below use Node 20 and the installed dependencies from the matching source snapshot. They are reproduced here with repository-relative paths. No proof was rerun during this checklist.

Caller identity, same real HTTP selector before and after the fix:

node node_modules/vitest/vitest.mjs run src/__tests__/llmock.test.ts --maxWorkers=1 --silent=false -t 'caller fixture count identity|clears fixture match-counts, rewinding sequence position|clearRequests.*empties the journal but keeps fixture match-counts'

RED (exit 1):   Test Files  1 failed (1)
               Tests  8 failed | 101 skipped (109)
GREEN (exit 0): Test Files  1 passed (1)
               Tests  8 passed | 101 skipped (109)

The 101 cases were deselected by the identical focused selector. Actual HTTP returned 200 in each of three registration cells (addFixture, addFixtures, prependFixture): caller count changed 0→1, has(original) false→true, canonical caller key false→true; Map size stayed 1. Repeated registration recovered one public entry; duplicate sibling sequence recovered counts [1,2] instead of three unrelated entries.

Cohere actual SDK → aimock HTTP at cut values 0.25, 0.5 and 0.75:

RED: AssertionError: expected 'MAX_TOKENS' to be 'TOOL_CALL'
     Tests  5 failed | 2 passed | 45 skipped (52)
GREEN: Tests  7 passed | 45 skipped (52)

The same focused K5 cases were replayed before/after; all three cuts preserved incomplete arguments and gained captured TOOL_CALL/closure/one [DONE] framing. Object MAX_TOKENS controls remained green. Six actual SDK/raw semantic mutations were rejected. The complete direct-consumer command is:

env -u COHERE_API_KEY node node_modules/vitest/vitest.mjs run src/__tests__/misbehavior-cohere.test.ts src/__tests__/cohere.test.ts --maxWorkers=1 --no-file-parallelism
# Retained full direct-consumer result: 129 passed.

Homepage real HTML collection, identical command before/after:

node node_modules/vitest/vitest.mjs run src/__tests__/competitive-matrix-homepage.test.ts --maxWorkers=1 --no-file-parallelism --retry=0

RED exit 1: Error: row not found: Model misbehavior faults (tool-call JSON, schema, unknown tool, stop reasons)
            Test Files  1 failed (1); Tests  no tests
GREEN exit 0: Test Files  1 passed (1); Tests  43 passed (43)

Vertex enabled configuration-error reporting used the real Vitest JSON report with the existing collectDriftEntries / computeExitCode consumer. With the token deliberately absent, RED produced four skipped native rows and collector exit 0; GREEN produced four failed native rows and collector exit 5. All 68 local cases remained passing, zero provider sends. This verifies fail-closed reporting, not successful native generation.

The unfiltered suite's two later failures had the same concrete prerequisite: dist/aimock-cli.js was older than integrated src/llmock.ts. A fresh pnpm build exited 0, followed by the entire affected real CLI file:

node node_modules/vitest/vitest.mjs run src/__tests__/mcp-fakes-llmock.test.ts --maxWorkers=1 --no-file-parallelism --retry=0 --reporter=json
# GREEN: 33 passed, 0 failed, 0 skipped; exit 0.

Separately, full ESLint RED identified only an unused pure literal named unavailable. Removing that unused declaration produced full ESLint exit 0, with independent confirmation that runtime proof remains applicable. Full formatting, pnpm typecheck (all three configurations), and pnpm build also exited 0.

Final upstream and C-S40 consumer evidence

The same integration command before/after conflict resolution:

node node_modules/vitest/vitest.mjs run src/__tests__/competitive-matrix.test.ts src/__tests__/competitive-matrix-summary.test.ts --maxWorkers=1 --minWorkers=1

RED: two Vitest collection failures from actual merge markers; exit 1.
GREEN: 157 tests passed; exit 0.
Expanded affected consumers: 13 suites, 411 tests passed; exit 0.

This is collection RED→GREEN, not 157 individually observed assertion failures. Independent review confirmed both P1 and peer assertions survive the union.

C-S40 used an actual owned HTTP fixture server with the same probe and input before/after registration. Essential raw observations:

{"phase":"RED","cS40Registered":false,"reports":[{"id":"mockserver-lr-sessions","status":"no change","claims":["C-S4","C-S13"]}],"needsReview":false,"fetches":2,"filesUnchanged":true,"exit":1}
{"phase":"GREEN","cS40Registered":true,"reports":[{"id":"mockserver-lr-sessions","status":"no change","claims":["C-S4","C-S13"]},{"id":"mockserver-lr-chaos","status":"changed","claims":["C-S40"]}],"needsReview":true,"fetches":2,"filesUnchanged":true,"exit":0}

The direct committed consumer tests are reproducible with:

node node_modules/vitest/vitest.mjs run src/__tests__/competitive-watch-http.test.ts src/__tests__/competitive-watch.test.ts --maxWorkers=1 --minWorkers=1
# Test Files  2 passed (2)
# Tests       55 passed (55)

Three independently accepted boundaries cover source-linked C-S40/manual-review output, preservation of the actual MockServer Partial cell and dry-run files, and unchanged/error controls. The real HTTP result flows through the existing matrix updater and summary renderer. No new native/provider calls are claimed by this proof.

First-CI corrections and current status

The first PR head (559ec2ea) recorded 32 successful checks and two failed checks. Those historical failures remain preserved. Both corrections are independently accepted, and replacement head 39c48d4b now has 33 successful checks and 2 skips, with no failures or pending checks.

Commitlint: the actual CI range validator rejected four purpose-commit body paragraphs longer than 100 characters. Only line wrapping changed; subjects/words, authors/committers/dates, all source bytes and peer history were preserved. Backup history was retained and all replacement commits were signed.

node node_modules/@commitlint/cli/cli.js --from d50b5fd7d7099e0b78db8d3a28fd7f3bb5367556 --to HEAD

RED at 559ec2ea: body's lines must not be longer than 100 characters [body-max-line-length]
                four affected commits; exit 1
GREEN after metadata correction: no findings; exit 0
GREEN at local replacement 39c48d4b: same full-range validator; exit 0

Workflow scan: the scanner interpreted claude inside an AWS model identifier in shell source as a Claude CLI invocation. The correction moves the unchanged model literal into step environment data; the existing main-only shell export reads the quoted variable. No scanner suppression/config change or credential/model-selection change was made.

Using the unmodified Sentinel scanner pinned at 3001b71ad3fc6998649be5a18e39585479a3ef5b, the same local command before and after was:

ruby /path/to/pinned-sentinel/bin/sentinel --local . --format json --severity high

The scanner path denotes the pinned checkout, not an additional repository script. Local Ruby was 3.2.2; CI uses Docker Ruby 3.3, so this is local scanner proof rather than replacement-head CI success.

{"phase":"RED","workflows":16,"findings":[{"rule":"ai-config-injection","severity":"high","file":"test-drift.yml","line":345}],"summary":{"critical":0,"high":1,"medium":0,"low":0},"exit":1}
{"phase":"GREEN","workflows":16,"findings":[],"summary":{"critical":0,"high":0,"medium":0,"low":0},"exit":0}

The same real workflow shell blocks were executed before/after with the actual step environment. Main kept the exact model/live/profile/region and runner command; feature and PR references kept activation variables unset. All three controls were identical; no provider calls. The existing workflow suite passed 24/24, and independent review found no findings. Affected-workflow formatting, offline zizmor, structural actionlint and full-range commitlint passed. Full actionlint reports only the identical pre-existing SC2086 informational finding (line shifted 424→425).

All other 756 source paths are unchanged. The accepted full lint, explicit three-config TypeScript, composed runtime and affected-consumer test proof remain applicable, as does the fresh build with 107 unchanged build inputs. Hosted OIDC/manual daily-workflow validation and recurrence remain post-PR/post-merge acceptance work; exact-head CI is green and the explicitly authorized merge is complete; hosted acceptance remains open.

Add deterministic fault configuration, parsing, preparation, scoped controls and journal evidence. Preserve ordinary caller fixture-count identities while keeping fault registrations independent.

Apply modeled output faults across supported provider wires and preserve each wire's terminal, usage and served-tool behavior. Add local SDK, Python and boundary regression coverage.

Compare bounded native and modeled contracts through the existing drift job. Retain strict failures, explicit approved non-trigger outcomes and short-lived AWS/Vertex authentication wiring.

Document supported fault modes, fixture controls, observed native contracts and remaining fidelity limits. Update provider references, authoring guidance and Unreleased notes without release claims.

pkg-pr-new · 2026-10-09T06:05:46Z

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@508

commit: 39c48d4

…ra parity) (#509) ## Goal This PR finishes the Mastra comparison on the homepage. In the "Agent tool mocking" table, three aimock cells change to ✓: HC1 "Report unused mocks (per test; can fail the test)", which was partial; HC3 "Create fakes from recorded runs (record live MCP, replay offline)", which was Planned; and HC4 "Workflow / multi-step pipeline targets (any caller over MCP HTTP)", which changes to ✓ for both aimock and MockServer. "In-process (non-network) tools" stays ✗ for aimock by design, because aimock runs out of process. The spec is on Notion: https://app.notion.com/p/3f43aa38185281f6ab92d99209dcd92e ## What ships The MCP recorder records a live MCP server into `mcpFakes` through a pass-through proxy. You turn it on with `llmock --mcp-record /path=url` (or `--mcp-proxy-only` to proxy without writing), with `llm.record.mcp` in a config file, or in code with `MCPMock.enableRecording`. Known secrets (`AIMOCK_RECORD_SECRET_VALUES`, upstream auth from `AIMOCK_MCP_UPSTREAM_AUTH`) are redacted before anything is written, and `--watch` and `validate` understand recorded files. Recorded fakes replay with the recorded `tools/list` order, the recorded progress and log notifications, and the recorded timing (scaled by the replay speed). A new control route, `GET /__aimock/mcp/fakes/report?testId=…`, returns a per-test report of served, unconsumed, failed and unfaked MCP calls. The Vitest and Jest plugins add `fakesFor` and `fakesReport`, with a default test id taken from the running test, and a `fakesReport: "off" | "warn" | "fail"` mode that can fail a test whose MCP traffic did not match its fakes, even when the code under test swallowed the error. The pytest plugin adds `fakes_for`, `fakes_report` and `assert_fakes_report`, with the same report text as the TypeScript helper. There are new examples (`examples/mcp-record-proof`, `examples/mcp-fakes-report-proof`, `examples/mcp-workflow-targets`, `examples/mcp-record-replay-agents`) and docs for recording, the report route, the test plugins, the pytest helpers and the homepage table. ## Red-green evidence per cell Each RED ran against the published 1.44.0 tarball; each GREEN ran the same command against a tarball packed from this branch. HC1, report unused mocks (S7, S8). S7 ran `examples/mcp-fakes-report-proof/run.sh <tgz>` with real Vitest 3.2.4 and Jest 29. On base, the swallowed `MCP_FAKE_MISMATCH` test and the unused-entry test pass (wrong) and `fakesFor` is not a function (base: `== exit 1: pnpm exec vitest run`, `== exit 1: pnpm exec jest`, both from the missing `fakesFor`). On this branch the swallowed test fails with `AimockFakesReportError: ... failure MCP_FAKE_MISMATCH: tools/call get_weather on /mcp with {"city":"Portland"}`, the unused test fails naming `weather.json[2]:never-called`, and `fakesFor` passes with the default id (PR: `== exit 1: pnpm exec vitest run`, `== exit 1: pnpm exec jest`; with `AIMOCK_FAKES_REPORT=off` both runs give `== exit 0`, so the report is the only thing that fails them). The script itself always exits 0; the CI gate for HC1 is `src/__tests__/fakes-report-plugins.test.ts`, which checks real child Vitest exit codes. S8 (pytest): the new test file against base `_server.py`/`plugin.py` gives `9 failed` (`'AIMockServer' object has no attribute 'assert_fa...'`); against this branch it gives `9 passed`, and the whole package suite passes. HC3, fakes from recorded runs (S6, S10). S6 ran `examples/mcp-record-proof/proof.sh` against a real `@modelcontextprotocol/server-everything` upstream. On base it fails with `TypeError [ERR_PARSE_ARGS_UNKNOWN_OPTION]: Unknown option '--mcp-record'` and `proof.sh exit=1`. On this branch it records 13 tools with `SECRET_COUNT=0`, then replays offline with `REPLAY_EQUALS_RECORDING=true`, `REPLAY_EQUALS_LIVE_NONSECRET=true`, `GET_ENV_REDACTED=true` and `MISMATCH_CODE=MCP_FAKE_MISMATCH`, and `mcp-record-cli.test.ts` goes from 17 failed to 21/21 passed. S10 records live, then replays offline to a real Mastra agent and to the Python `mcp` client. On base: `== exit 1: Mastra agent replay`, `== exit 1: Python mcp replay`, `VERDICT=FAIL`. On this branch: `AGENT_REPORT ok=true served=5 unconsumed=0 failures=0`, each result deep-equals the recording, a changed recorded argument makes both clients fail with `MCP_FAKE_MISMATCH`, and `VERDICT=PASS`. HC4, workflow targets (S9). `examples/mcp-workflow-targets/run.sh` drives a Mastra workflow and a LangGraph graph against MCP fakes. Base, mismatch fixture: `== exit 1: vitest run (workflow-mismatch.json)`, because `fakesFor is not a function`. This branch, mismatch fixture: `== exit 1: vitest run (workflow-mismatch.json)` with `failure MCP_FAKE_MISMATCH: tools/call lookup_order on /mcp with {"orderId":"B2"}` in both files, while the Mastra run status is `success` because the step swallowed the error; with the report off it is `== exit 0`. Undeclared fixture: `== exit 1` with `MCP_FAKE_NOT_DECLARED` (report off: `== exit 0`). Good fixture: `== exit 0: vitest run (workflow.json)`, every call `answered`, report `ok: true`. ## Review The branch had one final, risk-scoped review: 24 agents, including 5 real-consumer probes. It produced 5 must-fix items, and each was fixed with a local red-green: 1. CRLF in `AIMOCK_RECORD_SECRET_VALUES` left secrets unredacted, and programmatic `enableRecording` did not read the secret and auth env vars. Both are fixed. 2. Tests now pin each secret form the sanitizer redacts, and the 8-character boundary. 3. The Vitest plugin now refuses `fakesReport` in concurrent tests with a clear error, instead of attributing calls to the wrong test. The Jest docs state the same limit. 4. The pytest report helper no longer raises `KeyError` for a `tools/call` sent with no arguments. 5. The `--watch` recording test now waits for the watcher to arm before it changes fixtures, which removes a flake. Everything else is a listed follow-up in the aimock backlog (Mastra parity, 2026-10-08), grouped as: recorder edge cases (relay and non-recordable paths, protocol-version handling, merge of repeated recordings); CLI, config and `--watch` polish; MCP handler and mount edge cases; test-plugin and report polish (including a friendly "server too old" error from the pytest helpers); type exports; and proof-script hardening, so that the example scripts gate on their own exit code. ## Release notes for the maintainer The pytest `AIMOCK_VERSION` pin (`packages/aimock-pytest/src/aimock_pytest/_version.py`, and `packages/aimock-pytest/README.md`) stays at 1.44.0 in this PR. Bump it at release together with the other version surfaces, because `fakes_for`, `fakes_report` and `assert_fakes_report` need the new server route. The homepage note, the docs, the CHANGELOG and the pytest README say "Requires 1.45.0+". If another release ships first, change that number to the release that contains this PR. The `./mcp` subpath does not yet re-export the new types; `@copilotkit/aimock` does. This is in the backlog. ## Coordination The branch is merged with main after Codex's model misbehavior PR #508, and #508's semantics are kept as they are (fault counters, admission order, match-count clearing and the early `llm.misbehavior` validation). After the merge no conflicts remained, the lockfile matches a fresh `pnpm install`, and format, lint, typecheck, build, the pytest suite, the record proof and a Docker build passed at the merge commit. The only full-suite Vitest failures were timeouts that also fail on main under load.

…s API (#511) Refs #505 ## Summary Codex CLI talks to models over the OpenAI Responses API. It routes MCP and sub-agent tools by `(namespace, name)`, and its `apply_patch` tool is a freeform custom tool. aimock could not script either shape, so a Codex session against aimock failed on its first tool call. This is PR 1 of 2. Recording is PR 2. - **Emission.** A fixture tool call can carry a `namespace`, emitted as `function_call.namespace`. A `toolCalls` entry with `type: "custom"` (or a `customToolCall` block) emits a `custom_tool_call` item with `response.custom_tool_call_input.delta` / `.done` events. Both work over HTTP SSE, non-streaming and WebSocket. Fixtures that use neither field produce byte-identical output. - **Matching.** Function and custom tools inside `namespace` tools, `custom` tools, and tools delivered in `additional_tools` and `tool_search_output` input items are now visible to `toolName` and predicates. A new `match.toolNamespace` key selects by exact namespace, alone or paired with `toolName`. `custom_tool_call` and `custom_tool_call_output` history items count as tool rounds. - **Validation and errors.** File loading validates the new shapes up front. At request time, every other wire rejects a custom call before writing any content, with a 500 `aimock_unsupported_tool_call` in that wire's own error envelope (an error frame on Realtime and Gemini Live). A malformed fixture tool call gets `aimock_invalid_fixture_tool_call` the same way, from `toolCalls` or blocks, static or factory. Fixtures that loaded and served before keep working. - **Misbehavior (#508).** Custom tool calls are never fault targets and pass through faulted output unchanged. A fixture the wire cannot serve is never faulted, so it gets the same coded error and journal entry as without a misbehavior config. Schema faults and `tool-unknown-name` resolve a Responses call by namespace and name. - **Docs and CHANGELOG.** Responses API page (shapes, event order, matching, a Codex recipe), fixtures reference, WebSocket and multi-turn pages, the write-fixtures skill, and CHANGELOG entries, including the TypeScript type changes. The README now lists the feature and links to the Responses API docs section. Spec: https://www.notion.so/3f43aa381852819c9dc0db15a4b40523 ## Design notes and decisions - Codex's MCP namespace is `mcp__<server>` with no trailing `__`. The docs follow real Codex 0.161.0 output. - An empty namespace name gets a 400, as in the OpenAI spec (`NamespaceToolParam.name` has `minLength: 1`). - `toolNamespace` matches any request tool that carries a `namespace`. Only the Responses API sets one from namespace tools. - On non-Responses wires a `namespace` is dropped and a custom call is rejected. - Legacy `toolCalls` load as on main, with warnings (legacy `type: "toolCall"`, numeric `id`, non-string `name`, `input` on a function call). The new fields are strict. When non-empty `blocks` are served, legacy `toolCalls` are ignored. - Three shapes that loaded in 1.44.0 are now load errors: an empty namespace, `type: "custom"` with `arguments`, and `type: "customToolCall"` in `toolCalls`. - `ToolCallResponse.toolCalls` becomes `FixtureToolCall[]` (a breaking type change, noted in the CHANGELOG). `AudioResponse.toolCalls` stays `ToolCall[]`. - Custom input deltas are split on code points, so no lone surrogates. - A malformed `custom_tool_call_output` becomes empty text (200, as on main). - Invalid-arguments envelopes on Ollama, Gemini and Bedrock gain an aimock `code`. Existing fields are unchanged. - Misbehavior: custom calls are never fault targets or the `tool-call-id-duplicate` destination. Output-clearing faults with `tool=<custom name>` are not applicable. `stop-length-mid-tool` always cuts a function call: custom calls before it complete, custom calls after it are dropped, and a custom-only fixture gets 501 not-applicable. Wire-format sources: the OpenAI OpenAPI spec, openai-node `responses.d.ts` (7.30/7.31), Codex `rust-v0.161.0` (`protocol/src/models.rs`, `codex-api/src/sse/responses.rs`), and captured Codex 0.161.0 requests. A trimmed responses-lite request is checked in as `src/__tests__/fixtures/codex-responses-lite-request.json`. ## Commits 1. `9c0cf717` test: pin Responses tool-call wire output. A golden of the exact HTTP, SSE and WS bytes for tool-only, content plus tools, and ordered blocks, captured before any change. 2. `4a41f44f` feat: emit namespaced function calls and custom tool calls. Fixture types, one emission path for all Responses paths, custom input deltas, coded errors on every other wire, and load-time validation. 3. `319573b7` feat: match namespace, custom and additional_tools tools. Request tool flattening, the `toolNamespace` key (router, journal, validation, aimock-pytest), history conversion, and 400s for malformed tool collections on HTTP and WS. 4. `047db367` fix(misbehavior): keep faults off custom tool calls and unservable fixtures. Also the namespace-aware schema lookup. 5. `16423c12` test: drift legs for namespaced and custom calls, streaming and non-streaming. 6. `af56cad0` docs: Responses API, fixtures, WebSocket and multi-turn pages, and the write-fixtures skill. 7. `efd33ae3` docs(changelog): the #505 entries, including the type changes that can break code compiled against 1.44.0. 8. `5e36eb2c` fix(misbehavior): judge `tool-unknown-name` within the call's own namespace. 9. `3acf0955` docs(readme): mention namespaced and custom Responses tool calls. One README feature bullet that links to the Responses API docs section. ## Verification ### Local gates Final run on the pushed head (9 commits): prettier, eslint, typecheck, build and `test:exports` pass; `pnpm test` 13318 passed, 3 skipped (run after build, as CI does); aimock-pytest 237 passed; commitlint passes on all 9 commits. The 9th commit changes only the README, so the test, build and typecheck results above still apply. For that commit, the prettier check, commitlint and the CLI test that reads the README passed. Not run: the live drift legs (no `OPENAI_API_KEY` locally). They skip with a stated reason when the model does not produce the call. ### Codex CLI 0.161.0 end to end Run on the final tree with no misbehavior faults. Model `gpt-5.5`, isolated `CODEX_HOME`, `approval_policy = "never"`, aimock with `--validate-on-load`, one aimock process per scenario. Turn 2 fixtures match on `toolCallId`. The MCP server is a small stdio server, so the namespace is `mcp__stdiodocs`. | # | Scenario | Transport | Codex exit | Observed | |---|---|---|---|---| | 1 | `custom_tool_call` `apply_patch` | HTTP | 0 | `hello.txt` written with the fixture's content. 2 requests; turn 2 carried the custom call output and matched. Final message "done: patch applied". | | 2 | `custom_tool_call` `apply_patch` | WebSocket | 0 | Same file and content. WS turn 1 and turn 2 both 200. No HTTPS fallback. | | 3 | namespaced `function_call` `mcp__stdiodocs/lookup_doc` | HTTP | 0 | The MCP server received `tools/call` `lookup_doc` with `{"id":"505"}`. The output came back and matched turn 2. Final message "done: MCP round complete". | | 4 | namespaced `function_call` `mcp__stdiodocs/lookup_doc` | WebSocket | 0 | Same MCP call received. WS turn 1 and turn 2 both 200. No HTTPS fallback. | On main before this PR (checked before the rebase), `--validate-on-load` rejects the custom fixture, and Codex fails the MCP call with `unsupported call: lookup_doc`. Over WS, Codex's empty warm-up `response.create` gets a 404 (same on main), and each run spent about 15 s before turn 1. ### Red/green evidence Core feature, on a real server, main vs this branch (measured before the rebase onto #508): - Custom call fixture on Responses: 500 `"arguments" must be a string` → 200 with `output_item.added`, input deltas, `.done`, `output_item.done`. - `toolName` on a tool offered only inside a namespace: 404 → 200, `namespace` on every item, over HTTP, SSE and WS. - `toolNamespace` + `toolName` pair with the wrong namespace registered first: 404 → 200 from the right fixture. - History `custom_tool_call` plus output with `hasToolResult: true`: 404 → 200. - The new test files on main: 111 failed, 21 passed (the passes are the byte-identity golden and unchanged-behaviour guards). Review found the defects below in this branch. Each one was reproduced on a real server before the fix and checked again after it, with a test that fails before and passes after. - Custom input deltas split surrogate pairs: 4/5 → 5/5 chunking cases clean. - Factory `toolCalls` with non-object entries: uncoded 500 on Responses, 200 nameless call on Chat → coded `aimock_invalid_fixture_tool_call`, same as static. - Array `toolCalls` entry: Chat 200 `function: {}`, Responses uncoded 500 → coded 500 on every wire. Array block in the loader: wrong message → "must be an object". - Mistyped custom `type` with no string `arguments` on Responses: uncoded 500 → coded error on HTTP and WS. - Duplicate-fixture key joined with `|`: false shadow warning → no warning; the true duplicate still warns. - Non-chat custom-call load warning: text false for 16 of 18 endpoints → text true for all 18. - Load warning read ignored legacy `toolCalls` when `blocks` were served → warns only on what is served. - `FunctionFixtureBlock` not exported: TS2305 in a consumer → compiles. - Ollama `/api/generate` misreported unknown blocks as tool calls (400) → same 500 as `/api/chat`. - `/api/generate` block errors journaled with a null body → body and fixture journaled; response bytes unchanged. - `/api/generate` with a tool call plus an unknown block: 400 → the same 500 as `/api/chat`, in both orders. - Misbehavior planner threw on an unservable block fixture: Realtime and Gemini Live unjournaled, Gemini Interactions null-body entry → skipped and journaled like a run with no config. - Invalid JSON arguments on wires that parse them: 54 off/on mismatches (faults served 200 or journaled `applied: true`) → 0. - `tool-call-id-duplicate` could land on a custom call: 22 violations → 0. - Output-clearing faults with `tool=<custom name>`: 16/16 cleared the custom call → 0/16; reported as not applicable. - Schema faults on a namespaced call used another namespace's schema (served valid arguments) → the call's own schema (arguments violate it). - `tool-unknown-name` checked the replacement name across all namespaces: wrong 501 and a skipped `_v2` name → 200, renamed within the call's namespace. - Docs: claims about `tools` validation, check order, object arguments and `/api/generate` were probed against a real server and corrected to match. ## Out of scope / follow-ups Review raised more than this PR fixes. What is left falls into three groups: behaviour that is the same on main, work for another subject or PR 2, and minor polish in this feature area. None of it blocks Codex use. The full list is below. <details> <summary>A. Same behaviour on main</summary> - WebSocket Responses does not check the shape of `input`. A null item or object input throws a TypeError, sent as `server_error` and not journaled. - An array `function_call_output` is not matched by `toolResultContains`. - An `additional_tools` developer item becomes an empty system message. - A `null` entry in `toolCalls` makes `normalizeResponse` throw a TypeError at file load, not a validation error. - A text, unknown-type or non-object entry in `blocks` throws a plain Error: uncoded 500 and no journal error. Since this PR only tool-call blocks get the coded error. - On Gemini Live and Realtime the same malformed (non-tool-call) block errors are journaled as 200. - The Gemini Live history rollback does not cover malformed-block errors, so an unanswered user turn stays in history; it also leaves the fixture match counter advanced. - A comment says `resolveFixtureBlocks` requires string `arguments`, but it stringifies objects. - Function-call `arguments` of null, a number or a boolean pass load validation, because `JSON.parse` accepts them. - A function call with no `arguments` fails load as "not valid JSON: undefined", while the runtime treats it as `{}`. The same message is given for a mistyped `type`, without naming it. - Invalid function `toolCalls` arguments on the Responses API return an uncoded 500 with journal error null. - Gemini Interactions and Anthropic streaming send malformed tool arguments as they are, while non-streaming rejects them. - Gemini Interactions streams when the request omits `stream`. A request rejected for invalid tool arguments still uses up an interaction id. - Streamed Responses events carry no `sequence_number`. - Text, function-argument and reasoning deltas can split a UTF-16 surrogate pair. This PR fixed it only for custom `input` deltas. - An `item_reference` with no `type` (or `type: null`) is not recognised, so the next tool output merges into the previous turn. - Ollama `/api/generate` rejects a text-only `blocks` fixture with 400; for `{content, blocks:[text]}` it returns `content` while `/api/chat` returns the block text; a valid tool fixture adds a misbehavior summary `{applied:false}`. - Realtime: a programmatic fixture with object `arguments` streams no argument deltas; a function call with no `arguments` crashes mid-stream and is journaled as 200. - Object `toolCalls` arguments make the `tool-args-invalid-json` fault crash with an uncoded 500. - In the duplicate-fixture key an absent field and `""` collide (this PR corrected only the comment). `describeMatch` reports the wrong location. - Errors from factory clean-up are labelled "factory threw". - `FixtureFileBlock` is not exported from the package entry point. - Skill and docs: the Reasoning row of the provider matrix disagrees with the fixtures doc; the endpoint list omits `openai-live`; the "HTTP SSE Text Response" example does not send `stream: true`. - Tests: a leftover `console.log`, weak assertions and console noise in older tests, and a stale comment in the Responses drift file. </details> <details> <summary>B. Another subject or PR 2</summary> - aimock-pytest pins server 1.44.0 by default, which ignores `toolNamespace`. Bump it at the next aimock release. - WebSocket Responses keeps no `previous_response_id` state. A follow-up turn that sends only new items cannot match on earlier content, and an outputs-only input makes placeholder tool calls with an empty name in the journal. Same on main with plain function tools. Storing responses by id fixes both. - Codex's empty warm-up `response.create` over WebSocket gets a 404 (on main). - Record mode drops custom tool calls and `namespace`, so a recorded Codex session replays as "unsupported call" (PR 2). - Recording a non-streaming Responses body loses the `output` array (PR 2). - Codex sub-agent tasks arrive as `agent_message` input items, which aimock does not convert. - The `openai` dev dependency (4.104.0) has no types for custom tool calls or `namespace`, and its `ResponseStream` ignores `output_item.done` and `response.incomplete`. Bump it. </details> <details> <summary>C. Minor follow-ups in this feature area</summary> Behaviour and validation: - Any `/api/v3/` request is labelled "BytePlus ModelArk" in error text, even when BytePlus is not configured. - A non-array Gemini audio `toolCalls` is dropped with no error; make it a coded error. - Inside a `namespace` tool, non-object and other-type entries are dropped silently, and tools with no string `name` are accepted. - A falsy top-level `tools` counts as absent, but `input[i].tools: null` gets 400. - In history items, a non-string `namespace` is dropped and a non-string custom `input` becomes "" with no warning. - A `namespace` tool with no `description` or an empty `tools` array is accepted; the OpenAI spec rejects both. - A stray `input` is a warning on a `toolCalls` function entry but an error on a `toolCall` block. - The `toolNamespace` load warning says only Responses requests carry namespaces, but a Chat Completions tool with a `namespace` also matches. - `UnsupportedToolCallError` does not give the `toolCalls[i]` index and can name an empty tool. - A factory entry with a mistyped `type` and object `arguments` is served as a function call, while the same static fixture gets a coded 500. A factory that returns a non-object is spread with `{ ...raw }`. - A fixture tool-call error on `/api/generate` uses the OpenAI envelope while `/api/chat` uses Ollama's; a custom call gets 400 on generate but 500 on chat. - aimock emits `status` on `custom_tool_call` items; openai-node's type has no such field. Verify against the real API. - Bedrock streaming routes use the non-streaming wire label in error text. - `FixtureToolCallError`, `ResponsesInputItem` and `ResponsesToolDef` are not exported. Types and names: - `FixtureFileToolCall` is one interface with optional fields, not a union. - `ResponsesToolDef` has a catch-all member that stops narrowing and forces casts. - Normalized fixture types can still hold null or numeric names and object arguments; the `resolveFixtureBlocks` overload types are loose. - `invalidToolArgumentsEnvelope` now serves three error types, so the name is misleading. - Realtime `hasToolCalls` reads the raw `toolCalls`, not the resolved calls, and has a redundant alias. - `isPlainObject` in src/helpers.ts is another private object guard; the repo already has several (`isObject` in jsonrpc.ts and misbehavior.ts, `isPlainObject` in mcp-fakes.ts). Consolidating them is a separate cleanup. Misbehavior: - When a fixture cannot be served and misbehavior is skipped, no summary or log is written, even when a header asked for a fault. - `servesFixture` treats any error as "cannot serve", which can hide a helper bug. - `tool=` cannot tell apart two tools with the same name in different namespaces, and the journal target omits the namespace. - Blocks rebuilt for a fault drop extra fields. - `validateFixtureMisbehavior` accepts a fixture fault that can never fire on wires that reject invalid-JSON object arguments. - The schema lookup for a non-namespaced call on a non-Responses wire, whose Chat request tools carry a `namespace`, is not namespace-aware. - The docs do not say that `stop-length-mid-tool` drops custom calls after the cut. Drift tests: - `isModelNotFound` treats a 400 saying "does not exist" as a retired model, so a real shape rejection can show as skipped. - The non-streaming Responses leg calls `JSON.parse` with no guard; the model-listing fetch is not tagged as infra; `response.in_progress` is dropped from the SDK expectations. Unit tests: - The Gemini audio legacy-compat test never checks that the audio part is present. - A fixture-error-parity predicate looks for "PATCH" but the input is `{a:1}`, so it never matches. - The custom-input chunking test does not create the mock with `logLevel: "silent"`. - The other-wires test has no rows for Azure streaming, the compatibility-prefix paths, or the BytePlus route. - The byte-identity test's 3 s WebSocket deadline can flake and misses frames after `response.completed`. - The emission truncation test pins exactly one delta based on same-tick timing. - The invalid-object-arguments misbehavior test table has gaps; streaming pass-through is not tested. - Namespaced schema-fault tests cover only missing-required; no wrong-type, extra-property, enum, WS or `tool_search_output` case. - Mutant M07 (key order) survives; the fixture-loader test only checks narrowing. </details> ## Base and related PRs This branch is rebased onto main after #508 (model misbehavior), and it integrates with the misbehavior planner as described above. #509 is still open. If it lands first, this branch will be rebased onto it.

…ecording (#517) ## Why #509 (MCP fakes recording) changed the published TypeScript surface of `@copilotkit/aimock@1.44.0`. aimock ships no breaking changes, type-only ones included, so this PR restores each break additively. The items use the labels from the 1.44.0 compatibility audit: - **S1.** The `declare module "./fixture-loader.js"` augmentation in `src/mcp-fakes.ts` was emitted by tsdown as a literal `sideEffect();` statement in `dist/mcp-fakes.d.ts` and `.d.cts`. Every consumer that type-checks with `skipLibCheck: false` failed on every entry point (TS1036, TS2304). The augmentation is gone. A malformed recorded key now fails with the 1.44.0 rule `mcp-fakes/bad-block:h`, and the message names its `record-replay/fa-*` rule. - **B8.** `McpFakeAddOrigin.kind` had gained `"record"`, which breaks a custom `Mountable.addMcpFakes` that switches exhaustively on it. The public type is back to `"file" | "code" | "control-api"`. The recorder uses an internal origin type and a private `MCPMock` method. - **B9.** `AimockHandle` in `/vitest` and `/jest` had gained required `fakesFor()` and `fakesReport()`, which breaks code that constructs or stubs a handle. `AimockHandle` is restored exactly. `useAimock()` now returns a new `AimockFakesHandle` that extends it, and both types are exported from `/vitest` and `/jest`. - **B11.** `Mountable` had gained `setReplaySpeed?(speed: number): void`, so a 1.44.0 mount with its own `setReplaySpeed` member of another type no longer assigned. `Mountable` is back to its exact 1.44.0 shape. - **R12.** The runtime half of B11: a custom mount with a `setReplaySpeed` member was called at start, and a non-function member threw a `TypeError` at `start()`. The server and `LLMock` mount loops now give the replay speed only to aimock's own `MCPMock`. - **R13.** A bad recorded key reported `record-replay/fa-*` as the error rule instead of the 1.44.0 `mcp-fakes/bad-block:h`, and a bad recorded key together with another error reported a different first rule. The recorded keys are now checked where 1.44.0 rejected them as unknown keys, so the first rule matches 1.44.0. A new gate, `src/__tests__/package-types-compat.test.ts`, runs `npm pack` on the build, unpacks it into a temporary consumer, and type-checks every `exports` entry point (ESM and CJS) under `strict` and `skipLibCheck: false`. The consumer holds the 1.44.0 usage patterns for B8, B9 and B11. ## The hardening fixes These fix defects in the MCP recording feature that the review of the Mastra-parity work found. All of it is new since 1.44.0. - **Start-time validation and credential redaction.** `llm.record.mcp` and the record destination are checked in a pre-pass at start, before anything is mounted. The start fails with an error that names the mount and the destination for: `null`, `false`, `0` or `""` as `llm.record.mcp`; mount keys the server can never route (a trailing `/`, `//`, a query, a fragment, whitespace, a dot segment, an escaped character), on both `llm.record.mcp` and `--mcp-record` / `--mcp-proxy-only`; an upstream that is not an http(s) URL; an empty or non-string `fixturePath`; a broken symbolic link, `EACCES` or a non-writable directory as the destination; and `maxRecordBufferBytes` that is not greater than 0 and at most 256 MiB. Credentials in mount keys are redacted in these errors with the record sanitizer's URL rule. - **Upstream death mid-body fails loudly on every relay path.** Before, a truncated upstream answer was relayed as a clean-looking 200, a `SyntaxError` in the client, or a hang until the client timed out, with nothing logged. Now the buffered, SSE, pass-through and post-overflow paths all fail: a 502 before headers, a terminated stream after them, and a JSON-RPC error event first for an SSE call that never got its response. Each failure logs one `MCP-RECORD:` line and journals `recordSkipped: "upstream-aborted"`. A client that disconnects is not reported as an upstream failure. - **Config warnings.** `aimock --config` defaulted to the silent log level, which hid the `MCP-RECORD:` warnings, so a run that recorded nothing printed nothing. A config whose `llm.record.mcp` has at least one mount now defaults `logLevel` to `"warn"`. An explicit `llm.logLevel` still wins, and an empty `llm.record.mcp` stays silent. ## Red/green evidence Each fix was reproduced on the real surface (the built CLIs, a packed tarball, real MCP clients `@modelcontextprotocol/sdk` 1.31.0 and `@modelcontextprotocol/client` 2.2.0) before the change and re-run after it. - **S1, B8, B9.** RED: the compat gate's consumer against the base build failed with `mcp-fakes.d.ts(13,1): error TS1036: Statements are not allowed in ambient contexts` and `TS2304: Cannot find name 'sideEffect'` (both `.d.ts` and `.d.cts`), `TS2345: Argument of type '"record"' is not assignable to parameter of type 'never'`, and `TS2739: ... missing the following properties from type 'AimockHandle': fakesFor, fakesReport`. GREEN: the gate passes 2/2. In the packed `dist/`, `sideEffect` and `declare module` match 0 of 86 `.d.ts` and 86 `.d.cts` files. - **B11.** RED: a strict consumer with a `Mountable` whose `setReplaySpeed` is a string fails with `TS2322 ... Types of property 'setReplaySpeed' are incompatible`; npm 1.44.0 compiles it. GREEN: it compiles, and the `interface Mountable` block in `dist/types.d.ts` and `.d.cts` is byte-identical to npm 1.44.0. - **R12.** RED: `LLMock({ replaySpeed: 2 })` with such a mount threw `TypeError: handler.setReplaySpeed is not a function` at `start()`, before and after a late mount. GREEN: it starts and the member is never called, the same as 1.44.0. - **R13.** RED: a malformed recorded key gave `"rule":"record-replay/fa-list"`; a bad recorded key with `scope: 5` gave `bad-block:b` where 1.44.0 gives `bad-block:h`. GREEN: `"rule":"mcp-fakes/bad-block:h"` with `[record-replay/fa-list]` in the message. A probe of 722 inputs that 1.44.0 rejects, each loaded both through `addMcpFakes` and from a file, went from 238 rule mismatches to 0. A separate class of 66 rows, with a recorded key that is valid now, is listed under the follow-ups below. - **Start-time validation.** RED: `llm.record.mcp` of `null`, `false`, `0` and `""`, the mount keys `/mcp/`, `/mcp?x=1`, `/a b` and `//`, an empty `fixturePath`, a dangling symlink destination and `maxRecordBufferBytes` of 0, -5 and 314572800 all started (exit 0). A credentialed URL as a mount key printed `http://user:tok3nSECRET@host/mcp` in the error. GREEN: each exits 1 with a message that names the mount; the credentialed key prints `http://host/mcp (credentials removed)`, and the full stderr has 0 secret matches. Valid configs still start. - **Upstream death mid-body.** RED (killable upstream): SSE record path, both clients `TIMEOUT 4000ms`; pass-through chunked JSON, a raw 200 with a truncated body and `SyntaxError: Unterminated string in JSON`; pass-through content-length, a hang; buffered JSON, `Unexpected content type: null`; nothing logged. GREEN: SSE gives `McpError -32603 ... the upstream response failed` at once; the pass-through and post-overflow paths give `TypeError terminated` at once; buffered JSON gives a 502 with `{"error":"aimock MCP recorder: the upstream response failed"}`. Each logs `MCP-RECORD: the upstream response failed mid-body (upstream-aborted) ...` and journals `upstream-aborted`. A client abort produced no aimock output. - **Config warnings.** RED: `aimock --config` with an MCP record mount and a request with no testId printed only the listening line. GREEN: it prints `MCP-RECORD: forwarded, not recorded: a request on mount /mcp has no testId or context ...`. `mcp: {}` stays silent, like a config with no `mcp`. - The happy path still works: `examples/mcp-record-proof/proof.sh` exits 0, recording server-everything (13 tools, 0 secrets) and replaying it offline at speed 1 and 100, identical to the recording and to the live run. ## Not breaking Every runtime change touches only features added since 1.44.0: MCP recording (`llm.record.mcp`, `--mcp-record`, `--mcp-proxy-only`), the recorded `mcpFakes` keys, and the replay-speed hook. The type changes return the published surface to 1.44.0 and add only new names (`AimockFakesHandle`). ## Coordination Another session is fixing the #501, #508, #511 and #513 breaks separately and will reuse this compat gate. The gate's evidence consumer still shows their errors (`c-others.ts`, `c508.ts`) and none from #509. Release PR #516 is held until both land. ## Known follow-ups - 66 probe rows remain where a recorded key that is now valid, together with an unrelated error, gives a different first rule than 1.44.0 (for example `timing` plus `scope: 5` gives `bad-block:b`, not `bad-block:h`). Both versions reject the input; only the rule code differs. This waits on decision D4. - Programmatic `wireMcpRecording` can leave mounts half-wired if a later step fails. - `@modelcontextprotocol/client` 2.2.0 maps the 502 to `CLIENT_HTTP_NOT_IMPLEMENTED` (client side). - The pass-through SSE and SSE-after-overflow paths terminate the stream but do not inject a JSON-RPC error event. - Some test-strength mutants beyond what the new tests cover, and a few pre-existing log-wording nits. - `AimockRecordConfig` requires `providers`, but the runtime and docs accept an `llm.record` with only `mcp`. - The aimock-cli docs say the `"warn"` default applies when `llm.record.mcp` is set; the code applies it only when it has at least one mount. ## Test plan At the head of this branch, each exit code captured on its own: `prettier --check .`, `eslint .`, `pnpm typecheck` (three tsconfigs), `pnpm build`, `package-types-compat.test.ts`, and the recorder, record-CLI, aimock-cli and MCP fakes test files. The full suite's only failures are the pre-existing vitest worker `Timeout calling "onTaskUpdate"` error (it reproduces on the base) and a `validate-cli` timeout under load that passes alone (96/96); neither file is touched here.

…strict-args behavior opt-in (#518) ## Summary With default options, this branch has zero breaking changes compared with the published `@copilotkit/aimock` 1.44.0. The rule is: 1. An input that 1.44.0 accepted behaves identically. 2. An input that 1.44.0 rejected may now be accepted. 3. New behavior is opt-in. The changes from #501, #508, #511 and #513 that altered 1.44.0 behavior now sit behind explicit opt-ins. The 1.44.0 types (`ToolCall`, `FixtureToolCall`, block unions, `CollapseResult`, `resolveFixtureBlocks`, WebSocket handler arity, `FixtureLoadRuleRegistry`) are restored. The new carriers (`customToolCalls`, `responsesBlocks`, `MisbehaviorConfigError`, `StartFromConfigOverrides`) are additive. ## Opt-ins | Feature | Programmatic | CLI flag (`llmock` / `aimock`) | `aimock --config` key | |---|---|---|---| | Extended Responses tool handling (#511/#513): custom tool calls, namespaces. The fixture keys `match.toolNamespace`, `customToolCalls` and `responsesBlocks` are honored **only** under `"extended"`; in `"legacy"` they are ignored, as in 1.44.0 | `responsesTools: "legacy" \| "extended"` (default `"legacy"`; an invalid value warns and runs `"legacy"`); aimock-pytest: `AIMockServer(responses_tools="extended")` | `--responses-tools legacy\|extended`; `aimock validate --responses-tools legacy\|extended` | `llm.responsesTools` (an invalid value warns and runs `"legacy"`) | | Strict invalid tool arguments (#501): 500 `aimock_invalid_tool_arguments` on object wires, authored text on string wires | `strictToolArguments: true` | `--strict-tool-arguments` | `llm.strictToolArguments: true` | | Model misbehavior (#508) | `enableMisbehavior: true` (also passed through the plugins' `useAimock`) | `--misbehavior` | `llm.enableMisbehavior: true` (`llm.misbehavior` is read only when misbehavior is enabled) | | MCP recording from config | n/a (`MCPMock.enableRecording()` is unchanged) | n/a (`llmock --mcp-record` / `--mcp-proxy-only` are unchanged) | `llm.enableMcpRecording: true` (`llm.record.mcp` is read only with this opt-in; without it the key is ignored with one warning, as in 1.44.0) | On the `aimock` bin, a flag overrides the matching config key. If a boolean flag is absent, the config key applies. Without its opt-in, each setting keeps the 1.44.0 behavior. For example, invalid fixture tool arguments are served as `{}` with a warning, and a `misbehavior` key in a fixture or config is ignored (the config case logs one warning). ## Proof All comparisons use the packed build of this branch against the npm 1.44.0 tarball. | Proof | Result | |---|---| | Strict consumer matrix: TS 5.0.4 / 5.4.5 / 5.9.3 × bundler / node16-esm / node16-cjs × 7 entry points (root, mcp, a2a, vector, agui, vitest, jest), `skipLibCheck: false` | **126/126 cells pass** (exit 0, 0 errors) for both 1.44.0 and this branch | | Type proofs b01–b12, s01, b07, b10-ws-handler-arity: TS 5.0.4 / 5.4.5 / 5.9.3 × nodenext / node16-esm / node16-cjs / bundler / node10, strict | 0 src errors in every cell, **except b12** (see Known open item). node10 has the same 4 TS1192 lib errors as 1.44.0 | | `src/__tests__/package-types-compat.test.ts` (strict gate) | 2/2 pass | | Runtime A/B, default options: http-ab (all wires × fixtures, stream and non-stream) | 704 same / 0 diff | | ws-ab / record-ab / record-ns-ab2 | 28/0, 60/0, 6/0 | | #501 ab501, 1.44.0 vs default (wire bytes, journal, warn count) | 260/0 | | #501 ab501, pre-fix main vs `strictToolArguments: true` | 260/0 on bytes and journal (with strict, Ollama, Interactions and Live also log the 1.44.0 warning) | | #508 enabled-ab, main vs `enableMisbehavior: true` | 39/0 | | Codex CLI 0.161.0 against the packed `dist/cli.js` with `--responses-tools extended` | **4/4**: custom `apply_patch` over HTTP and over WS, namespaced MCP over HTTP and over WS. Contrast without the flag: 404 "No fixture matched", as in 1.44.0 | There are 3 documented deltas from 1.44.0 with default options. All 3 are allowed by the rule: 1. **`GET /__aimock/misbehavior`**: 1.44.0 returns 404, this branch returns 200. This is a new additive route (rule 2). 2. **`aimock --config` with a valid or invalid `llm.misbehavior` and no opt-in**: the exit code (0) and the served responses are identical. The only difference is one extra stderr warning: `Ignoring llm.misbehavior because misbehavior is not enabled. ...` 3. **Fixture with invalid JSON tool arguments loaded through `validate` / `addFixturesFromJSON`**: 1.44.0 already rejected this input, and this branch rejects it the same way. The message now ends with a pointer to `misbehavior: tool-args-invalid-json`. ### Red / green Every RED was observed on the real failure surface (packed builds driven over HTTP/WS, the real CLI, and `tsc` consumer projects) before the fix. Every GREEN was observed afterwards on the same probe. | Track | RED (before the fix) | GREEN (after the fix) | |---|---|---| | #501 invalid tool arguments (ab501: 10 fixtures, 22 HTTP routes in stream and non-stream modes, 3 WS wires) | 1.44.0 vs pre-fix main: **119 diffs**, 7 on each of the 17 affected wire/mode cells. Controls identical | 1.44.0 vs fix (default): **260/260 identical** on bytes, journal and warn count. Audit #501 cells: 30 → 0 (HTTP), 1 → 0 (WS) | | #508 misbehavior types and WS arity (B7, B10-WS consumer matrix, 15 cells) | Consumer projects fail to compile against main (registry and 4-parameter WS handler breaks) | Same as 1.44.0 in every cell: 0 errors, except the 4 node10 TS1192 lib errors that 1.44.0 also has | | #508 `aimock --config` with `llm.misbehavior: "empty-response"` (fixture `weather` → `sunny`) | 1.44.0 `content="sunny"`, branch before the fix `content=""` | `content="sunny"` plus the warning. With `llm.enableMisbehavior: true` or `--misbehavior`: `content=""`. Invalid value with the opt-in: exit 1 with a rule-prefixed message | | #511/#513 types (b01–b06, b10) | **14 src errors per cell, 210 in total** across 15 cells (TS2339 `arguments`, `"function" \| "custom"`, `CollapseResult.toolCalls`, `resolveFixtureBlocks` arity, ...) | **0 src errors** in all 15 cells | | #511/#513 runtime (namespace / custom fixtures F08–F13, F27–F31, Q01–Q23, loader-ab, record-ns) | Up to 500 on 21/21 routes per fixture, 200 → 400 on Q01–Q06, loader rejects inputs that 1.44.0 accepted, 3/6 recordings differ | Identical to 1.44.0 by default. With `responsesTools: "extended"`, records `namespace` and `customToolCalls` | ## Known open item: B12 A consumer object that already has its own **non-fresh** `misbehavior` key of a different type (for example `misbehavior: string` on a value assigned to `Fixture`, or `misbehavior: { enabled: true }` on a value assigned to `MockServerOptions`) no longer assigns to those types: 2 × TS2322 per type-proof cell. Adding any optional key has this effect, so it cannot be avoided without renaming the key. It is the only type proof that does not compile, and `main` fails it too. This needs a maintainer decision: accept it, or rename the `misbehavior` key. ## Quality gates (head of this branch) - `prettier --check .`, `eslint .` and `tsc --noEmit` (`tsconfig.json`, `tsconfig.test.json`, `scripts/tsconfig.json`) all pass. The `tsdown` build passes. - The 65 touched test files, including the strict gate: 3695 passed, 2 skipped (env-gated). - commitlint over the branch: 0 problems, 0 warnings. All commits are signed. ## Review fixes (F1, F2, F3, C1, C2, validation) - **F1**: `match.toolNamespace` is honored only with `responsesTools: "extended"`. In the default legacy mode it is ignored, as in 1.44.0. - **F2**: `customToolCalls` and `responsesBlocks` are served only with `"extended"`. In legacy mode they are dropped on every wire (HTTP, SSE, Responses WS, Realtime, Gemini Live), as in 1.44.0. - **F3**: an invalid `responsesTools` (option, constructor or `llm.responsesTools`) logs one warning and runs `"legacy"` instead of failing startup. An invalid `--responses-tools` value still exits 1, because the flag is new. - **C1**: tests for the `llmock --misbehavior` and `--responses-tools` pass-through. - **C2**: a non-boolean `llm.strictToolArguments` or `llm.enableMisbehavior` logs one warning and the option stays off. - **Validation**: the validating doors (`validateFixtures`, `addFixturesFromJSON()`, `POST /__aimock/fixtures`, `--validate-on-load`, `aimock validate`) check the three keys only in `"extended"`. New opt-in: **`aimock validate --responses-tools extended`** (default `legacy` = 1.44.0 parity; an invalid value is a usage error). - **aimock-pytest**: new opt-in `AIMockServer(responses_tools=None)`, which passes `--responses-tools <value>` when set. Unset keeps the 1.44.0 default. The plugin's `toolNamespace` test opts in to `"extended"`. | Repro | RED (PR head before the fixes) | GREEN (608a484) | |---|---|---| | F1/F2 runtime, default vs 1.44.0 (32 cases + 4 supplementary) | 3/32 identical, 29 differ | **32/32** and **4/4** identical | | F1/F2 runtime, extended vs PR head extended | n/a | **32/32** identical (new behavior kept) | | Validating doors, default vs 1.44.0 (12 inputs x 6 doors) | 20/72 identical, 52 differ | **72/72** identical | | Validating doors, extended vs PR head | n/a | **48/48** identical | | F3/C2: `aimock -c` with invalid `responsesTools` / non-boolean flags, constructor with `responsesTools: "bogus"` | exit 1 / `TypeError`; no warning for non-boolean flags | starts, one `[aimock] Ignoring ...` warning each, legacy output identical to 1.44.0 | | Python plugin suite (CI commands, `AIMOCK_CLI_PATH=dist/cli.js`) | 1 failed (`test_add_fixture_routes_tool_namespace_into_match`: "No fixture matched"), 247 passed | **248 passed**. Removing the opt-in makes the test fail again | Gates at 608a484: full vitest (12 shards) 14,012 passed / 3 skipped; `package-types-compat.test.ts` 2/2; typecheck, eslint, prettier and build pass; commitlint clean; all commits signed. Refs #505 #501 #508 #509 #511 #513 ## CI fix: Python Tests (fe5db7c) All 16 `test (3.10..3.13, 20..26)` jobs failed with `188 failed, 60 passed`. All 188 were in the new `test_misbehavior_{openai,anthropic,controls}.py` proofs. The pytest plugin started the CLI without `--misbehavior`, so faults were ignored, for example `'{"city":"Paris","units":"metric"}' != '{"city":"Paris",'`, and `/__aimock/misbehavior` answered `409 Conflict`. No test outside the misbehavior modules failed, so this is not a default-behavior regression. Fix: `AIMockServer(enable_misbehavior=False)` appends `--misbehavior` when true. The default stays off, as in 1.44.0. The three misbehavior modules override `aimock` with a server that opts in. Local proof (Python 3.12, Node 26, `AIMOCK_CLI_PATH=dist/cli.js`, `cd packages/aimock-pytest && pytest tests/ -v`): ``` RED (cbcca44): 188 failed, 60 passed, 103 warnings in 71.36s (0:01:11) GREEN (fe5db7c): 248 passed, 103 warnings in 65.26s (0:01:05) ``` - Bedrock stream default-mode test parses event-stream frames instead of regex (was flaky on random CRC bytes) ## Review round 2 fixes - **R2-F1/F2/F3**: in legacy mode, the fixture listing (`GET /__aimock/fixtures`) and the journal match 1.44.0 byte for byte. - **Public export parity**: `isContentWithToolCallsResponse`, `collapseOpenAISSE` and `collapseStreamingResponse` return the 1.44.0 output again (93/93 export probe calls identical to 1.44.0). - **O7**: when `llm.responsesTools` is invalid, the warning names the mode that actually runs (`Using "extended" from --responses-tools.`); with no flag the text is unchanged. - **record.mcp opt-in**: `llm.record.mcp` is read only with the new `llm.enableMcpRecording: true`; without it the key is ignored with one warning, as 1.44.0 ignored it. ## Review round 3 fixes - **R3-F1**: the stream builders reproduce the 1.44.0 output when `toolCalls` is not an array. - **R3-F2**: the `responsesTools` mode is per server, so servers that share fixtures no longer affect each other. ## Review round 4 fixes - Default mode keeps caller fixture identity (addFixture/on/addFixtures/prependFixture) as 1.44.0 did; the per-add copy happens only with enableMisbehavior - Extended servers count and journal the caller's fixtures (identity matches 608a484) - POST /__aimock/fixtures on an extended server no longer leaks extended normalization into a shared caller array - Identity APIs behave the same with enableMisbehavior on (journal/counts/findByFixture name the caller's fixture; same object added twice still gives two fault sources) - Docs: responsesTools mode is fixed at server start

Includes #501, #504, #506, #508, #509, #510, #511, #512, #513, #514, #515, #517, #518. ## [1.45.0] - 2026-10-10 Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior that would change those inputs is opt-in. ### Added - Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image), `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and `MCPMock.enableRecording()` / `disableRecording()` (#509). - Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`, `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509). - Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`, `durationMs`) need 1.45.0 or later (#509). - Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509). - Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and `AimockTestIdCollisionError` (#509). - aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511). - Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM (`examples/mcp-workflow-targets`), plus record and report proofs (#509). - Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs, length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires, with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true` (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`, or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw the new `MisbehaviorConfigError` (#508). - Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace` applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0. - Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types: `CustomToolCallMessage`, `ResponsesToolsMode` (#511). - Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`, `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools: "extended"` is set (#513). - Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501). - Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift checks (#508). ### Fixed - MCP recording checks its config at start, redacts credentials in errors, fails loudly when the upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock --config` with record mounts to `logLevel: "warn"` (#517). - Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made each new behavior opt-in (#518). - Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck: false` (#517). - Align Cohere streaming length-stop faults with captured provider framing (#508). - Retain drift evidence across retries and for oversized records (#510, #512). - Classify `claude-haiku-5-5` in the drift model registry (#515). - Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight, and the copy-button fallback (#504, #506, #514).

Includes #501, #504, #506, #508, #509, #510, #511, #512, #513, #514, #515, #517, #518. ## [1.45.0] - 2026-10-10 Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior that would change those inputs is opt-in. ### Added - Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image), `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and `MCPMock.enableRecording()` / `disableRecording()` (#509). - Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`, `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509). - Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`, `durationMs`) need 1.45.0 or later (#509). - Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509). - Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and `AimockTestIdCollisionError` (#509). - aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511). - Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM (`examples/mcp-workflow-targets`), plus record and report proofs (#509). - Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs, length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires, with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true` (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`, or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw the new `MisbehaviorConfigError` (#508). - Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace` applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0. - Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types: `CustomToolCallMessage`, `ResponsesToolsMode` (#511). - Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`, `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools: "extended"` is set (#513). - Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501). - Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift checks (#508). ### Fixed - MCP recording checks its config at start, redacts credentials in errors, fails loudly when the upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock --config` with record mounts to `logLevel: "warn"` (#517). - Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made each new behavior opt-in (#518). - Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck: false` (#517). - Align Cohere streaming length-stop faults with captured provider framing (#508). - Retain drift evidence across retries and for oversized records (#510, #512). - Classify `claude-haiku-5-5` in the drift model registry (#515). - Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight, and the copy-button fallback (#504, #506, #514). - Invalid opt-in values never stop startup: an invalid `responsesTools` warns once and runs `"legacy"`, and a non-boolean `llm.strictToolArguments`, `llm.enableMisbehavior` or `llm.enableMcpRecording` warns once and stays off (#518).

Release npm/plugin **1.45.0** and **aimock-pytest 0.7.0** from `main` at c9c19d1 (#518 merged). Every fixture, config and request that 1.44.0 accepted behaves the same by default; new behavior is opt-in (#518). This branch was rebuilt as one commit on top of the #518 merge, replacing the earlier release commit bdfaeef. > **Prepared, not merged.** `publish-release.yml` runs on every push to `main`. Merging this PR publishes: npm `@copilotkit/aimock@1.45.0`, the `v1.45.0` tag, a force-move of the `v1` major tag, the GitHub Release, the Docker images (`ghcr.io/copilotkit/aimock` and `ghcr.io/copilotkit/llmock`, `:1.45.0` and `:latest`), `aimock-pytest` 0.7.0 on PyPI and the Slack announcement. The docs site deploys from `main:/docs` on merge. Merge only when the release is authorized. Do not dispatch `publish-release.yml` from this branch. ## Versions | Surface | Before | After | |---|---|---| | `package.json` `version` | 1.44.0 | 1.45.0 | | `.claude-plugin/plugin.json` `version` | 1.44.0 | 1.45.0 | | `.claude-plugin/marketplace.json` `source.version` | ^1.44.0 | ^1.45.0 | | `charts/aimock/Chart.yaml` `appVersion` (chart `version` stays 0.1.0) | 1.44.0 | 1.45.0 | | `docs/index.html` CLI banner | aimock v1.44.0 | aimock v1.45.0 | | `packages/aimock-pytest/pyproject.toml` `version` | 0.6.0 | 0.7.0 | | `packages/aimock-pytest/src/aimock_pytest/_version.py` `AIMOCK_VERSION` | 1.44.0 | 1.45.0 | | `packages/aimock-pytest/README.md` `--aimock-version` default | 1.44.0 | 1.45.0 | | `skills/write-fixtures/SKILL.md` `toolNamespace` pin note | "newer than its default 1.44.0 pin" | "aimock 1.45.0+, its default pin from 0.7.0" | "Unreleased" wording now names 1.45.0 in `README.md` (misbehavior note), `docs/cohere/index.html` (length-stop fidelity) and `docs/model-misbehavior/index.html` (two places). The pytest pin bump matters: the new `AIMockServer(enable_misbehavior=..., responses_tools=...)` options pass `--misbehavior` / `--responses-tools`, which the 1.44.0 CLI does not accept. With the pin at 1.45.0, default-download users get a server that supports them. **1.45.0 is a minor.** With #518, default behavior and the published types match 1.44.0. The remaining type additions (the optional `misbehavior` key, `ChatCompletionRequest.customTools`, `useAimock` returning `AimockFakesHandle`) were accepted by the maintainer as additive optional typing (minor). ## CHANGELOG ## [1.45.0] - 2026-10-10 Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior that would change those inputs is opt-in. ### Added - Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image), `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and `MCPMock.enableRecording()` / `disableRecording()` (#509). - Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`, `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509). - Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`, `durationMs`) need 1.45.0 or later (#509). - Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509). - Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and `AimockTestIdCollisionError` (#509). - aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511). - Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM (`examples/mcp-workflow-targets`), plus record and report proofs (#509). - Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs, length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires, with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true` (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`, or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw the new `MisbehaviorConfigError` (#508). - Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace` applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0. - Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types: `CustomToolCallMessage`, `ResponsesToolsMode` (#511). - Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`, `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools: "extended"` is set (#513). - Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501). - Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift checks (#508). ### Fixed - MCP recording checks its config at start, redacts credentials in errors, fails loudly when the upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock --config` with record mounts to `logLevel: "warn"` (#517). - Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made each new behavior opt-in (#518). - Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck: false` (#517). - Align Cohere streaming length-stop faults with captured provider framing (#508). - Retain drift evidence across retries and for oversized records (#510, #512). - Classify `claude-haiku-5-5` in the drift model registry (#515). - Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight, and the copy-button fallback (#504, #506, #514). - Invalid opt-in values never stop startup: an invalid `responsesTools` warns once and runs `"legacy"`, and a non-boolean `llm.strictToolArguments`, `llm.enableMisbehavior` or `llm.enableMcpRecording` warns once and stays off (#518). ## Release checklist - [x] Opt-in names for #501, #508, #511, #513 match the merged compat PR #518 (`strictToolArguments` / `--strict-tool-arguments` / `llm.strictToolArguments`; `enableMisbehavior` / `--misbehavior` / `aimock validate --misbehavior` / `llm.enableMisbehavior`; `responsesTools` / `--responses-tools` / `aimock validate --responses-tools` / `llm.responsesTools`; `llm.enableMcpRecording`; pytest `enable_misbehavior` / `responses_tools`), checked against the source on this branch. - [x] Pytest pin `AIMOCK_VERSION` = `"1.45.0"` and the README `--aimock-version` default = 1.45.0. - [x] aimock-pytest `version` 0.6.0 -> 0.7.0. - [x] Homepage "Requires 1.45.0+" (MCP scenario fakes card) and the other "1.45.0 or later" docs notes match this release. - [x] Version surfaces bumped (table above). `git grep -F 1.44.0`: every remaining hit is a history comparison ("as in 1.44.0", "up to 1.44.0", the 1.44.0 CHANGELOG section) or recorded `aimockVersion` provenance in fixtures/examples. - [x] "Unreleased" wording removed from README and docs. - [x] `src/__tests__/package-types-compat.test.ts` on the release head: 2/2 pass. - [ ] 1.44.0 A/B probe re-run on the release head: not re-run here; this commit changes only version strings, docs text and the CHANGELOG, and #518 carries the A/B proof on the same code. ## Local gates (release head) - `pnpm install --frozen-lockfile`, `format:check`, `lint`, `typecheck`, `build`, `test:exports` (publint + attw): pass - Full vitest, 12 shards: 14,069 passed, 3 skipped (env-gated), 0 failed - aimock-pytest, installed as CI does (hash-pinned hatchling, `--no-build-isolation`), Python 3.12, `AIMOCK_CLI_PATH=dist/cli.js`: 248 passed - `npm pack --dry-run`: `copilotkit-aimock-1.45.0.tgz`, 766 files (1.44.0: 722), dist/, fixtures/, skills/, .claude-plugin/, README, CHANGELOG, LICENSE; no src or tests - commitlint: 0 problems, 0 warnings. Commit is SSH-signed.

…ider side (#523) ## Problem Two live drift legs on main fail intermittently with `expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ]`: - `openai-responses.drift.ts`: `K9:provider:exhaustion-contract` - `ws-responses.drift.ts`: `K9:native:exhaustion-contract` This is not drift in aimock. Live o4-mini picks one of three reasoning-summary shapes at random for the same K9 request (o4-mini, effort high, summary auto, `max_output_tokens: 32`). ## Measurements (2026-10-10, exact K9 request, no changes between runs) | Surface | full-summary (9 events) | no-summary (5) | unclosed-summary (7) | |---|---|---|---| | HTTP raw probe, 12 runs | 8 | 2 | 2 | | HTTP drift test, 10 runs | 8 | 2 | 0 | | WS drift test, 10 runs | 5 | 3 | 2 | - **no-summary:** `created, in_progress, output_item.added, output_item.done, incomplete`. The reasoning item has `summary: []`. - **unclosed-summary:** the summary part opens and streams deltas, but `reasoning_summary_text.done` and `reasoning_summary_part.done` never arrive. The final item has `summary: []`. - Every run ended `response.incomplete` / `max_output_tokens` with `output_tokens: 0` on `o4-mini-2025-04-16`. ## Change This follows the principle already applied to Gemini K5/K9: accept real upstream variants on the live/native side only. - `OPENAI_K9_SUMMARY_SHAPES` in `helpers.ts` names exactly the three observed shapes, with the measured frequencies in a comment. Any other summary run still fails. The check is not "any prefix". - Provider/native K9 grading accepts those three shapes, checks each one's payload, and still requires: - the `response.incomplete` terminal with reason `max_output_tokens` - `output_tokens === 0` - the requested model, or a dated snapshot of it, matching across `created`, `in_progress` and the terminal. - Localhost grading is unchanged and must still emit the full summary. The localhost-vs-live comparison now uses only the event order outside the summary run, plus status and reason. - Retained live captures of each shape (HTTP SSE and WS events) live in `src/__tests__/drift/fixtures/` and replay without a credential. Mutants must fail: - an 8-event out-of-order summary sequence - a wrong terminal status - nonzero `output_tokens` - a different model - localhost grading fed the 5-event or 7-event shape. - The live test bodies move into `compareExhaustion` / `compareWSExhaustion` so the replays use the same comparison path as the live tests, including the WS SDK/real/mock shape triangulation. No `src/` changes, so there is no public API or type impact. ## Local red-green **RED**: the replays run through the unfixed check (`env -u OPENAI_API_KEY npx vitest run --config vitest.config.drift.ts src/__tests__/drift/openai-responses.drift.ts src/__tests__/drift/ws-responses.drift.ts -t "variant replay"`): ``` ✓ WS retained full-summary native events agree with actual localhost × WS retained no-summary native events agree with actual localhost 1. [critical] expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ] × WS retained unclosed-summary native events agree with actual localhost 1. [critical] expected [ 'response.created', …(6) ] to deeply equal [ 'response.created', …(8) ] × WS retained full-summary native events reject ... nonzero output tokens (AssertionError: expected [Function] to throw an error) ✓ HTTP retained full-summary provider stream agrees with actual localhost × HTTP retained no-summary provider stream agrees with actual localhost 1. [critical] expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ] × HTTP retained unclosed-summary provider stream agrees with actual localhost 1. [critical] expected [ 'response.created', …(6) ] to deeply equal [ 'response.created', …(8) ] × HTTP retained full-summary provider stream rejects ... nonzero output tokens (AssertionError: expected [Function] to throw an error) Tests 6 failed | 12 passed | 23 skipped (41) ``` **GREEN**: the same command after the fix: `Tests 18 passed | 23 skipped (41)`. All three shapes pass on both surfaces. Every mutant is rejected, and localhost still rejects the 5-event and 7-event shapes. **Mutation check of the new assertions.** Each mutant was applied and then reverted: - dropping the `output_tokens` check fails 6 tests - letting localhost accept the variants fails 4 tests - dropping the model check fails 6 tests. **Live GREEN**: two consecutive runs of both full drift files with `OPENAI_API_KEY` (`npx vitest run --config vitest.config.drift.ts src/__tests__/drift/openai-responses.drift.ts src/__tests__/drift/ws-responses.drift.ts`): ``` run 1: Tests 39 passed | 2 skipped (41) HTTP K9 live: 9 events, WS K9 live: 9 events run 2: Tests 39 passed | 2 skipped (41) HTTP K9 live: 7 events (unclosed-summary), WS K9 live: 9 events ``` ## Gates `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, `pnpm build`: all pass. `pnpm test`: 377 files, 14073 passed. ## Review round 1 Each finding was checked before any change. Every change has a mutation red-green. - **A1 (capture replay mode):** no workflow sets `OPENAI_RESPONSES_EXHAUSTION_CAPTURE_DIR`. A grep of `.github/` returns nothing; the only references are `openai-responses.drift.ts:92/300-310/367`. The `k9-20261008T170312Z` capture is not in the repo or in Actions artifacts, and PR #508 says no artifact host holds it. It can't be replayed, so CI never uses that path. No change. - **A2 (`isOpenAIModelSnapshot`):** it already behaves correctly. `o4-mini` and `o4-mini-2025-04-16` pass. `o4-mini-high`, `o4-mini-foo`, `o4-mini-2025-04` and `o4-mini2025-04-16` are all rejected. No change. - **A3 (which `output_tokens` field):** the assertion reads `terminal.response.usage.output_tokens`, where `terminal = events.at(-1)` narrowed to `response.incomplete`. However, the HTTP mutant's `replaceAll` also rewrote the zero-valued `tool_usage` fields in created and in_progress, so it didn't isolate the terminal. The mutant now sets only the terminal's usage to 5 and asserts that everything before the terminal is byte-identical. Removing the assertion fails 6 tests (HTTP and WS). - **A4 (terminal-only model):** added on both surfaces. Only the terminal gets `o4-mini-2099-01-01`, a valid snapshot form that differs from created/in_progress. Deleting the consistency loop fails 6 tests. - **A5 (unmatched shape label):** an unmatched live order was already rejected. It fell back to the full-summary order and failed `toEqual` with "deeply equal", as the 8-event test showed. The fallback label is gone: an unmatched order now throws `live K9 event order matches none of the accepted summary shapes (full-summary, no-summary, unclosed-summary): …`. Restoring the fallback fails the two 8-event tests. **Live, after the review fixes:** two consecutive runs of both full drift files: ``` run 1: Tests 39 passed | 2 skipped (41) HTTP K9 live: 9 events, WS K9 live: 9 events run 2: Tests 39 passed | 2 skipped (41) HTTP K9 live: 5 events (no-summary), WS K9 live: 9 events ``` **Gates:** typecheck, lint, format:check and build pass. `pnpm test`: 377 files, 14073 passed.

Misbehavior is opt-in after the CopilotKit#508 compat fix, so the two CopilotKit#501 interplay tests must enable it. Without it the fixture misbehavior key is ignored and the tool-args-invalid-json case serves {}.

jpr5 added 7 commits October 8, 2026 22:44

feat: add deterministic model misbehavior controls and fixture contracts

8af5635

Add deterministic fault configuration, parsing, preparation, scoped controls and journal evidence. Preserve ordinary caller fixture-count identities while keeping fault registrations independent.

feat: render model faults across providers with SDK coverage

930fb5e

Apply modeled output faults across supported provider wires and preserve each wire's terminal, usage and served-tool behavior. Add local SDK, Python and boundary regression coverage.

test: add bounded recurring model misbehavior comparisons

587b19f

Compare bounded native and modeled contracts through the existing drift job. Retain strict failures, explicit approved non-trigger outcomes and short-lived AWS/Vertex authentication wiring.

docs: describe model fault support and native fidelity limits

ab827a2

Document supported fault modes, fixture controls, observed native contracts and remaining fidelity limits. Update provider references, authoring guidance and Unreleased notes without release claims.

Merge current main into model misbehavior work

75ee03f

docs: clarify model fault fixture support and fidelity

ebf9298

fix(scripts): watch MockServer model fault capabilities

68719f0

fix(ci): keep Bedrock model data outside shell commands

39c48d4

jpr5 force-pushed the feat/misbehavior-pr1-engine branch from 559ec2e to 39c48d4 Compare October 9, 2026 06:18

jpr5 marked this pull request as ready for review October 9, 2026 06:23

jpr5 enabled auto-merge October 9, 2026 06:24

jpr5 merged commit 83556d0 into main Oct 9, 2026
35 checks passed

jpr5 deleted the feat/misbehavior-pr1-engine branch October 9, 2026 06:24

This was referenced Oct 9, 2026

feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509

Merged

feat: namespaced function calls and custom tool calls on the Responses API #511

Merged

This was referenced Oct 9, 2026

chore: release v1.45.0 #516

Merged

fix: Mastra-parity follow-ups — restore 1.44.0 types and harden MCP recording #517

Merged

jpr5 mentioned this pull request Oct 10, 2026

fix: restore 1.44.0 default behavior; make new Responses/misbehavior/strict-args behavior opt-in #518

Merged

jpr5 mentioned this pull request Oct 11, 2026

test(drift): accept OpenAI's three live K9 summary shapes on the provider side #523

Merged

Navigation Menu

Repository navigation

feat: add deterministic model misbehavior faults - #508

feat: add deterministic model misbehavior faults#508
jpr5 merged 8 commits into
mainfrom
feat/misbehavior-pr1-engine

jpr5 commented Oct 9, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Oct 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jpr5 commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Provider faults and controls

Native fidelity and modeled exceptions

Documentation and consumer changes

Validation and real RED → GREEN evidence

Final local verification and pending acceptance

Essential retained RED/GREEN output

Final upstream and C-S40 consumer evidence

First-CI corrections and current status

Uh oh!

pkg-pr-new Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jpr5 commented Oct 9, 2026 •

edited

Loading

pkg-pr-new Bot commented Oct 9, 2026 •

edited

Loading