Repository navigation
chore: release v1.45.0 - #516
Merged
Merged
Conversation
commit: |
jpr5
marked this pull request as draft
October 9, 2026 22:17
jpr5
added a commit
that referenced
this pull request
Oct 10, 2026
…ecording (#517) ## Why #509 (MCP fakes recording) changed the published TypeScript surface of `@copilotkit/aimock@1.44.0`. aimock ships no breaking changes, type-only ones included, so this PR restores each break additively. The items use the labels from the 1.44.0 compatibility audit: - **S1.** The `declare module "./fixture-loader.js"` augmentation in `src/mcp-fakes.ts` was emitted by tsdown as a literal `sideEffect();` statement in `dist/mcp-fakes.d.ts` and `.d.cts`. Every consumer that type-checks with `skipLibCheck: false` failed on every entry point (TS1036, TS2304). The augmentation is gone. A malformed recorded key now fails with the 1.44.0 rule `mcp-fakes/bad-block:h`, and the message names its `record-replay/fa-*` rule. - **B8.** `McpFakeAddOrigin.kind` had gained `"record"`, which breaks a custom `Mountable.addMcpFakes` that switches exhaustively on it. The public type is back to `"file" | "code" | "control-api"`. The recorder uses an internal origin type and a private `MCPMock` method. - **B9.** `AimockHandle` in `/vitest` and `/jest` had gained required `fakesFor()` and `fakesReport()`, which breaks code that constructs or stubs a handle. `AimockHandle` is restored exactly. `useAimock()` now returns a new `AimockFakesHandle` that extends it, and both types are exported from `/vitest` and `/jest`. - **B11.** `Mountable` had gained `setReplaySpeed?(speed: number): void`, so a 1.44.0 mount with its own `setReplaySpeed` member of another type no longer assigned. `Mountable` is back to its exact 1.44.0 shape. - **R12.** The runtime half of B11: a custom mount with a `setReplaySpeed` member was called at start, and a non-function member threw a `TypeError` at `start()`. The server and `LLMock` mount loops now give the replay speed only to aimock's own `MCPMock`. - **R13.** A bad recorded key reported `record-replay/fa-*` as the error rule instead of the 1.44.0 `mcp-fakes/bad-block:h`, and a bad recorded key together with another error reported a different first rule. The recorded keys are now checked where 1.44.0 rejected them as unknown keys, so the first rule matches 1.44.0. A new gate, `src/__tests__/package-types-compat.test.ts`, runs `npm pack` on the build, unpacks it into a temporary consumer, and type-checks every `exports` entry point (ESM and CJS) under `strict` and `skipLibCheck: false`. The consumer holds the 1.44.0 usage patterns for B8, B9 and B11. ## The hardening fixes These fix defects in the MCP recording feature that the review of the Mastra-parity work found. All of it is new since 1.44.0. - **Start-time validation and credential redaction.** `llm.record.mcp` and the record destination are checked in a pre-pass at start, before anything is mounted. The start fails with an error that names the mount and the destination for: `null`, `false`, `0` or `""` as `llm.record.mcp`; mount keys the server can never route (a trailing `/`, `//`, a query, a fragment, whitespace, a dot segment, an escaped character), on both `llm.record.mcp` and `--mcp-record` / `--mcp-proxy-only`; an upstream that is not an http(s) URL; an empty or non-string `fixturePath`; a broken symbolic link, `EACCES` or a non-writable directory as the destination; and `maxRecordBufferBytes` that is not greater than 0 and at most 256 MiB. Credentials in mount keys are redacted in these errors with the record sanitizer's URL rule. - **Upstream death mid-body fails loudly on every relay path.** Before, a truncated upstream answer was relayed as a clean-looking 200, a `SyntaxError` in the client, or a hang until the client timed out, with nothing logged. Now the buffered, SSE, pass-through and post-overflow paths all fail: a 502 before headers, a terminated stream after them, and a JSON-RPC error event first for an SSE call that never got its response. Each failure logs one `MCP-RECORD:` line and journals `recordSkipped: "upstream-aborted"`. A client that disconnects is not reported as an upstream failure. - **Config warnings.** `aimock --config` defaulted to the silent log level, which hid the `MCP-RECORD:` warnings, so a run that recorded nothing printed nothing. A config whose `llm.record.mcp` has at least one mount now defaults `logLevel` to `"warn"`. An explicit `llm.logLevel` still wins, and an empty `llm.record.mcp` stays silent. ## Red/green evidence Each fix was reproduced on the real surface (the built CLIs, a packed tarball, real MCP clients `@modelcontextprotocol/sdk` 1.31.0 and `@modelcontextprotocol/client` 2.2.0) before the change and re-run after it. - **S1, B8, B9.** RED: the compat gate's consumer against the base build failed with `mcp-fakes.d.ts(13,1): error TS1036: Statements are not allowed in ambient contexts` and `TS2304: Cannot find name 'sideEffect'` (both `.d.ts` and `.d.cts`), `TS2345: Argument of type '"record"' is not assignable to parameter of type 'never'`, and `TS2739: ... missing the following properties from type 'AimockHandle': fakesFor, fakesReport`. GREEN: the gate passes 2/2. In the packed `dist/`, `sideEffect` and `declare module` match 0 of 86 `.d.ts` and 86 `.d.cts` files. - **B11.** RED: a strict consumer with a `Mountable` whose `setReplaySpeed` is a string fails with `TS2322 ... Types of property 'setReplaySpeed' are incompatible`; npm 1.44.0 compiles it. GREEN: it compiles, and the `interface Mountable` block in `dist/types.d.ts` and `.d.cts` is byte-identical to npm 1.44.0. - **R12.** RED: `LLMock({ replaySpeed: 2 })` with such a mount threw `TypeError: handler.setReplaySpeed is not a function` at `start()`, before and after a late mount. GREEN: it starts and the member is never called, the same as 1.44.0. - **R13.** RED: a malformed recorded key gave `"rule":"record-replay/fa-list"`; a bad recorded key with `scope: 5` gave `bad-block:b` where 1.44.0 gives `bad-block:h`. GREEN: `"rule":"mcp-fakes/bad-block:h"` with `[record-replay/fa-list]` in the message. A probe of 722 inputs that 1.44.0 rejects, each loaded both through `addMcpFakes` and from a file, went from 238 rule mismatches to 0. A separate class of 66 rows, with a recorded key that is valid now, is listed under the follow-ups below. - **Start-time validation.** RED: `llm.record.mcp` of `null`, `false`, `0` and `""`, the mount keys `/mcp/`, `/mcp?x=1`, `/a b` and `//`, an empty `fixturePath`, a dangling symlink destination and `maxRecordBufferBytes` of 0, -5 and 314572800 all started (exit 0). A credentialed URL as a mount key printed `http://user:tok3nSECRET@host/mcp` in the error. GREEN: each exits 1 with a message that names the mount; the credentialed key prints `http://host/mcp (credentials removed)`, and the full stderr has 0 secret matches. Valid configs still start. - **Upstream death mid-body.** RED (killable upstream): SSE record path, both clients `TIMEOUT 4000ms`; pass-through chunked JSON, a raw 200 with a truncated body and `SyntaxError: Unterminated string in JSON`; pass-through content-length, a hang; buffered JSON, `Unexpected content type: null`; nothing logged. GREEN: SSE gives `McpError -32603 ... the upstream response failed` at once; the pass-through and post-overflow paths give `TypeError terminated` at once; buffered JSON gives a 502 with `{"error":"aimock MCP recorder: the upstream response failed"}`. Each logs `MCP-RECORD: the upstream response failed mid-body (upstream-aborted) ...` and journals `upstream-aborted`. A client abort produced no aimock output. - **Config warnings.** RED: `aimock --config` with an MCP record mount and a request with no testId printed only the listening line. GREEN: it prints `MCP-RECORD: forwarded, not recorded: a request on mount /mcp has no testId or context ...`. `mcp: {}` stays silent, like a config with no `mcp`. - The happy path still works: `examples/mcp-record-proof/proof.sh` exits 0, recording server-everything (13 tools, 0 secrets) and replaying it offline at speed 1 and 100, identical to the recording and to the live run. ## Not breaking Every runtime change touches only features added since 1.44.0: MCP recording (`llm.record.mcp`, `--mcp-record`, `--mcp-proxy-only`), the recorded `mcpFakes` keys, and the replay-speed hook. The type changes return the published surface to 1.44.0 and add only new names (`AimockFakesHandle`). ## Coordination Another session is fixing the #501, #508, #511 and #513 breaks separately and will reuse this compat gate. The gate's evidence consumer still shows their errors (`c-others.ts`, `c508.ts`) and none from #509. Release PR #516 is held until both land. ## Known follow-ups - 66 probe rows remain where a recorded key that is now valid, together with an unrelated error, gives a different first rule than 1.44.0 (for example `timing` plus `scope: 5` gives `bad-block:b`, not `bad-block:h`). Both versions reject the input; only the rule code differs. This waits on decision D4. - Programmatic `wireMcpRecording` can leave mounts half-wired if a later step fails. - `@modelcontextprotocol/client` 2.2.0 maps the 502 to `CLIENT_HTTP_NOT_IMPLEMENTED` (client side). - The pass-through SSE and SSE-after-overflow paths terminate the stream but do not inject a JSON-RPC error event. - Some test-strength mutants beyond what the new tests cover, and a few pre-existing log-wording nits. - `AimockRecordConfig` requires `providers`, but the runtime and docs accept an `llm.record` with only `mcp`. - The aimock-cli docs say the `"warn"` default applies when `llm.record.mcp` is set; the code applies it only when it has at least one mount. ## Test plan At the head of this branch, each exit code captured on its own: `prettier --check .`, `eslint .`, `pnpm typecheck` (three tsconfigs), `pnpm build`, `package-types-compat.test.ts`, and the recorder, record-CLI, aimock-cli and MCP fakes test files. The full suite's only failures are the pre-existing vitest worker `Timeout calling "onTaskUpdate"` error (it reproduces on the base) and a `validate-cli` timeout under load that passes alone (96/96); neither file is touched here.
jpr5
force-pushed
the
chore/release-v1.45.0
branch
from
October 10, 2026 14:25
bdfaeef to
c19e178
Compare
Includes #501, #504, #506, #508, #509, #510, #511, #512, #513, #514, #515, #517, #518. ## [1.45.0] - 2026-10-10 Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior that would change those inputs is opt-in. ### Added - Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image), `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and `MCPMock.enableRecording()` / `disableRecording()` (#509). - Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`, `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509). - Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`, `durationMs`) need 1.45.0 or later (#509). - Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509). - Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and `AimockTestIdCollisionError` (#509). - aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511). - Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM (`examples/mcp-workflow-targets`), plus record and report proofs (#509). - Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs, length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires, with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true` (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`, or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw the new `MisbehaviorConfigError` (#508). - Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace` applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0. - Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types: `CustomToolCallMessage`, `ResponsesToolsMode` (#511). - Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`, `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools: "extended"` is set (#513). - Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501). - Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift checks (#508). ### Fixed - MCP recording checks its config at start, redacts credentials in errors, fails loudly when the upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock --config` with record mounts to `logLevel: "warn"` (#517). - Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made each new behavior opt-in (#518). - Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck: false` (#517). - Align Cohere streaming length-stop faults with captured provider framing (#508). - Retain drift evidence across retries and for oversized records (#510, #512). - Classify `claude-haiku-5-5` in the drift model registry (#515). - Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight, and the copy-button fallback (#504, #506, #514). - Invalid opt-in values never stop startup: an invalid `responsesTools` warns once and runs `"legacy"`, and a non-boolean `llm.strictToolArguments`, `llm.enableMisbehavior` or `llm.enableMcpRecording` warns once and stays off (#518).
jpr5
force-pushed
the
chore/release-v1.45.0
branch
from
October 10, 2026 14:35
c19e178 to
51c1447
Compare
jpr5
marked this pull request as ready for review
October 10, 2026 15:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release npm/plugin 1.45.0 and aimock-pytest 0.7.0 from
mainat c9c19d1 (#518 merged). Every fixture, config and request that 1.44.0 accepted behaves the same by default; new behavior is opt-in (#518). This branch was rebuilt as one commit on top of the #518 merge, replacing the earlier release commit bdfaeef.Versions
package.jsonversion.claude-plugin/plugin.jsonversion.claude-plugin/marketplace.jsonsource.versioncharts/aimock/Chart.yamlappVersion(chartversionstays 0.1.0)docs/index.htmlCLI bannerpackages/aimock-pytest/pyproject.tomlversionpackages/aimock-pytest/src/aimock_pytest/_version.pyAIMOCK_VERSIONpackages/aimock-pytest/README.md--aimock-versiondefaultskills/write-fixtures/SKILL.mdtoolNamespacepin note"Unreleased" wording now names 1.45.0 in
README.md(misbehavior note),docs/cohere/index.html(length-stop fidelity) anddocs/model-misbehavior/index.html(two places).The pytest pin bump matters: the new
AIMockServer(enable_misbehavior=..., responses_tools=...)options pass--misbehavior/--responses-tools, which the 1.44.0 CLI does not accept. With the pin at 1.45.0, default-download users get a server that supports them.1.45.0 is a minor. With #518, default behavior and the published types match 1.44.0. The remaining type additions (the optional
misbehaviorkey,ChatCompletionRequest.customTools,useAimockreturningAimockFakesHandle) were accepted by the maintainer as additive optional typing (minor).CHANGELOG
[1.45.0] - 2026-10-10
Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior that would change those inputs is opt-in.
Added
mcpFakesfiles and replay them offline:--mcp-record <mount>=<url>and--mcp-proxy-only <mount>=<url>(llmockbin and Docker image),llm.record.mcpwith the opt-inllm.enableMcpRecording: trueforaimock --config(without the opt-in,llm.record.mcpis ignored with a warning, as 1.44.0 ignored it), andMCPMock.enableRecording()/disableRecording()(feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509).AIMOCK_RECORD_SECRET_VALUES,AIMOCK_MCP_UPSTREAM_AUTH); a file is not written while a secret remains (feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509).--replay-speed). The recordedmcpFakeskeys (list,recorded,timing,notifications,durationMs) need 1.45.0 or later (feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509).GET /__aimock/mcp/fakes/report(feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509).fakesFor()andfakesReport()on a newAimockFakesHandlereturned byuseAimock(), and opt-inuseAimock({ fakesReport: "warn" | "fail" })(default"off"); Jest optiontestIdRoot. New exportsassertFakesReport,AimockFakesReportErrorandAimockTestIdCollisionError(feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509).fakes_for,fakes_reportandassert_fakes_report(feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509), andAIMockServeropt-insenable_misbehaviorandresponses_tools(feat: add deterministic model misbehavior faults #508, feat: namespaced function calls and custom tool calls on the Responses API #511).examples/mcp-workflow-targets), plus record and report proofs (feat: MCP recording, per-test fake reports and workflow targets (Mastra parity) #509)./__aimock/misbehaviorruntime controls, journal evidence and theaimock_misbehavior_totalmetric. Off unless enabled withenableMisbehavior: true(programmatic, includinguseAimock),llmock --misbehavior/aimock validate --misbehavior, oraimock --misbehavior, or the config opt-inllm.enableMisbehavior: true(anllm.misbehaviorkey alone never enables it, as in 1.44.0). Invalid misbehavior settings throw the newMisbehaviorConfigError(feat: add deterministic model misbehavior faults #508).match.toolNamespace(feat: namespaced function calls and custom tool calls on the Responses API #511). Like the fixture fields below,match.toolNamespaceapplies only withresponsesTools: "extended"; in the default mode it is ignored, as in 1.44.0.responsesTools: "extended"(CLI--responses-tools extended, also onaimock validate; configllm.responsesTools), tools insidenamespacetools,customtools andadditional_tools/tool_search_outputtools are visible totoolName,toolNamespaceand predicates, custom tool rounds count forhasToolResult,toolCallIdandturnIndex, and fixtures can answer with namespaced calls andcustomToolCalls/responsesBlocks. The default"legacy"keeps 1.44.0 behavior. New types:CustomToolCallMessage,ResponsesToolsMode(feat: namespaced function calls and custom tool calls on the Responses API #511).customToolCalls,responsesBlocks,CollapseResult.customToolCalls) in streamed recordings whenresponsesTools: "extended"is set (feat(record): keep Responses namespace and custom tool calls in recordings #513).strictToolArguments(CLI--strict-tool-arguments, configllm.strictToolArguments), malformed tool-call argument strings are sent unchanged on string wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (fix: preserve malformed tool arguments across provider wires #501).Fixed
aimock --configwith record mounts tologLevel: "warn"(fix: Mastra-parity follow-ups — restore 1.44.0 types and harden MCP recording #517).llm.record.mcp, fixture identity and the exported stream helpers, and made each new behavior opt-in (fix: restore 1.44.0 default behavior; make new Responses/misbehavior/strict-args behavior opt-in #518).AimockHandle,MountableandMcpFakeAddOriginkeep their 1.44.0 shape, and every entry point type-checks withskipLibCheck: false(fix: Mastra-parity follow-ups — restore 1.44.0 types and harden MCP recording #517).claude-haiku-5-5in the drift model registry (chore(drift): classify claude-haiku-5-5 in includeFamilies #515).responsesToolswarns once and runs"legacy", and a non-booleanllm.strictToolArguments,llm.enableMisbehaviororllm.enableMcpRecordingwarns once and stays off (fix: restore 1.44.0 default behavior; make new Responses/misbehavior/strict-args behavior opt-in #518).Release checklist
strictToolArguments/--strict-tool-arguments/llm.strictToolArguments;enableMisbehavior/--misbehavior/aimock validate --misbehavior/llm.enableMisbehavior;responsesTools/--responses-tools/aimock validate --responses-tools/llm.responsesTools;llm.enableMcpRecording; pytestenable_misbehavior/responses_tools), checked against the source on this branch.AIMOCK_VERSION="1.45.0"and the README--aimock-versiondefault = 1.45.0.version0.6.0 -> 0.7.0.git grep -F 1.44.0: every remaining hit is a history comparison ("as in 1.44.0", "up to 1.44.0", the 1.44.0 CHANGELOG section) or recordedaimockVersionprovenance in fixtures/examples.src/__tests__/package-types-compat.test.tson the release head: 2/2 pass.Local gates (release head)
pnpm install --frozen-lockfile,format:check,lint,typecheck,build,test:exports(publint + attw): pass--no-build-isolation), Python 3.12,AIMOCK_CLI_PATH=dist/cli.js: 248 passednpm pack --dry-run:copilotkit-aimock-1.45.0.tgz, 766 files (1.44.0: 722), dist/, fixtures/, skills/, .claude-plugin/, README, CHANGELOG, LICENSE; no src or tests