Skip to content

feat(record): keep Responses namespace and custom tool calls in recordings - #513

Merged
jpr5 merged 4 commits into
mainfrom
feat/505-pr2-recording
Oct 9, 2026
Merged

feat(record): keep Responses namespace and custom tool calls in recordings#513
jpr5 merged 4 commits into
mainfrom
feat/505-pr2-recording

Conversation

@jpr5

@jpr5 jpr5 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Refs #505

Stacked on #511 (base feat/505-responses-namespace-custom-tools). Merge this after #511, and release the two together.

#509 has merged to main. #511 absorbed main with a merge commit, and this branch absorbed #511 (with #509) with a merge commit. Neither branch was rebased.

Summary

  • Record mode now keeps namespace on Responses function_call items. Before, the recorder dropped it, so a replayed MCP call came back without its namespace and Codex could not route it.
  • Record mode now keeps custom_tool_call items, with their full input. Before, the recorder dropped them, so a turn with only a custom call (for example apply_patch) was saved as {"content":""}.
  • Interleaved turns record custom calls as customToolCall blocks, so the order of text, function calls and custom calls survives the recording.
  • With this, a Codex session recorded through aimock replays correctly: apply_patch writes its file and the MCP tool is called.

What changed

src/stream-collapse.ts

  • Function calls and custom calls share one accumulator, keyed by output_index. The ordered-block list gains a custom-call kind.
  • namespace comes from response.output_item.added, or from response.output_item.done if added did not have it. An empty string is ignored.
  • response.custom_tool_call_input.delta appends to the input. response.custom_tool_call_input.done and output_item.done supply the input only when no delta arrived.
  • These handlers run before the generic response.* handler.
  • At the end, a custom call becomes { type: "custom", name, input, id?, namespace? } with no arguments, and its block becomes customToolCall.

src/recorder.ts

  • sanitizeRecordedToolCall sanitizes each kind separately. A function call gets arguments ?? "{}". A custom call gets input ?? "" and never gets arguments.

Type change (TypeScript only): CollapseResult.toolCalls widens from ToolCall[] to FixtureToolCall[], because collapseOpenAISSE can now return a custom call. Code that reads .arguments from a collapser result needs to narrow on type first. The existing collapse tests read .arguments in many places. They now go through a small helper, fnArgs in src/__tests__/helpers/tool-calls.ts, which fails the test if it gets a custom call. That edit is mechanical, and the test logic does not change.

Tests: a new file, src/__tests__/responses-record-namespace-custom.test.ts, has 8 collapse unit tests and one HTTP test that goes record, then disk, then validate, then replay.

Docs and CHANGELOG

Verification

Record run, before and after. Recorder A is a real aimock in record mode. Its upstream is a second aimock that serves the #511 Responses stream. A third instance then replays only A's fixtures.

  • Before (at feat: namespaced function calls and custom tool calls on the Responses API #511's head): the custom-only turn was saved as {"content":""}. The namespaced call was saved without namespace. Mixed and interleaved turns lost the custom call. Replay differed from upstream in 4 of 4 cases.
  • After: every fixture keeps namespace and the full custom input, in toolCalls and in blocks. Replay matches upstream in 4 of 4 cases.

Record, then replay, with Codex CLI 0.161.0. Codex records a session through aimock, and a second aimock replays only the recorded fixtures.

Case Before After
Custom apply_patch --validate-on-load rejects the fixture (empty content). With a plain load, Codex prints an empty message and hello.txt is not created. Validation passes. On replay, apply_patch writes hello.txt and Codex ends on "done: patch applied". Exit 0.
Namespaced MCP tool Codex logs unsupported call: lookup_doc and the MCP tool is not called. The fixture keeps namespace: "mcp__stdiodocs". On replay, the call reaches the MCP server, which handles it, and Codex ends on "done: MCP round complete". Exit 0.

In both cases the replay ends on the recorded final-text turn, so Codex does not loop.

Local gates: format, lint, typecheck, build, test:exports and commitlint pass. aimock-pytest: 248 passed. The full vitest suite ran 3 times, and all tests passed each time: 13601 passed, 3 skipped, 0 failed. Each run still exited 1 on one vitest-internal error, Timeout calling "onTaskUpdate", which no test is attributed to. It comes from machine load, and the base branch (#511 at b7eaf3f) shows the same error under the same load.

Commits

Out of scope and follow-ups

  • Malformed upstream streams in the collapser. Three cases can drop data without a warning: a function-call delta and a custom-call delta on the same output_index; a custom-call delta with no output_item.added before it (recorded with an empty name, which the loader then rejects); and custom-call events with no numeric output_index. A well-formed stream does not trigger any of them.
  • Tests. The review found two gaps in the new test coverage, and the new test file leaks a resource in its cleanup. Both go in a follow-up.
  • Non-streaming Responses recording. A non-streaming Responses call is still saved as an "unknown format" error fixture. That already happens on main and is not changed here. WebSocket Responses has no record path either.

jpr5 added 4 commits October 9, 2026 11:33
@jpr5
feat(record): keep Responses namespace and custom tool calls in recor…
3e1b94c 
…dings

Recording a streamed OpenAI Responses turn dropped `namespace` from
function calls and dropped `custom_tool_call` items entirely, so a
custom-call-only turn was saved as empty content and the replay could
not drive a client such as Codex CLI.

collapseOpenAISSE now accumulates function and custom calls in one
output_index space, captures `namespace` from output_item.added (filled
from output_item.done when absent), and collects custom input from
custom_tool_call_input.delta, adopting the full input from the .done
events only when no delta arrived. Finalize projects each kind to its
own fixture shape: a function call keeps normalized JSON arguments, a
custom call keeps its free-text input and never gets arguments.
Interleaved turns get customToolCall blocks. The recorder sanitizes per
kind.

CollapseResult.toolCalls widens to FixtureToolCall[]; existing collapse
tests narrow through a small fnArgs helper before reading arguments.

Refs #505
@jpr5
docs(record-replay): document recording of Responses namespaced and c…
65a7e4d 
…ustom calls

Refs #505
@jpr5
docs(record): describe custom tool call blocks in stream-collapse com…
f87b039 
…ments
@jpr5
chore: merge #505 branch (with #509) into recording branch
8644e38
Base automatically changed from feat/505-responses-namespace-custom-tools to main October 9, 2026 21:30
@jpr5 jpr5 closed this Oct 9, 2026
@jpr5 jpr5 reopened this Oct 9, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 9, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@513

commit: 8644e38

@jpr5
jpr5 merged commit 34891cd into main Oct 9, 2026
31 of 35 checks passed
@jpr5
jpr5 deleted the feat/505-pr2-recording branch October 9, 2026 21:54
This was referenced Oct 9, 2026
Merged
Merged
jpr5 added a commit that referenced this pull request Oct 10, 2026
@jpr5
fix: Mastra-parity follow-ups — restore 1.44.0 types and harden MCP r…
f7f8470 
…ecording (#517)

## Why

#509 (MCP fakes recording) changed the published TypeScript surface of
`@copilotkit/aimock@1.44.0`. aimock ships no breaking changes, type-only
ones included, so this PR restores each break additively. The items use
the labels from the 1.44.0 compatibility audit:

- **S1.** The `declare module "./fixture-loader.js"` augmentation in
`src/mcp-fakes.ts` was emitted by tsdown as a literal `sideEffect();`
statement in `dist/mcp-fakes.d.ts` and `.d.cts`. Every consumer that
type-checks with `skipLibCheck: false` failed on every entry point
(TS1036, TS2304). The augmentation is gone. A malformed recorded key now
fails with the 1.44.0 rule `mcp-fakes/bad-block:h`, and the message
names its `record-replay/fa-*` rule.
- **B8.** `McpFakeAddOrigin.kind` had gained `"record"`, which breaks a
custom `Mountable.addMcpFakes` that switches exhaustively on it. The
public type is back to `"file" | "code" | "control-api"`. The recorder
uses an internal origin type and a private `MCPMock` method.
- **B9.** `AimockHandle` in `/vitest` and `/jest` had gained required
`fakesFor()` and `fakesReport()`, which breaks code that constructs or
stubs a handle. `AimockHandle` is restored exactly. `useAimock()` now
returns a new `AimockFakesHandle` that extends it, and both types are
exported from `/vitest` and `/jest`.
- **B11.** `Mountable` had gained `setReplaySpeed?(speed: number):
void`, so a 1.44.0 mount with its own `setReplaySpeed` member of another
type no longer assigned. `Mountable` is back to its exact 1.44.0 shape.
- **R12.** The runtime half of B11: a custom mount with a
`setReplaySpeed` member was called at start, and a non-function member
threw a `TypeError` at `start()`. The server and `LLMock` mount loops
now give the replay speed only to aimock's own `MCPMock`.
- **R13.** A bad recorded key reported `record-replay/fa-*` as the error
rule instead of the 1.44.0 `mcp-fakes/bad-block:h`, and a bad recorded
key together with another error reported a different first rule. The
recorded keys are now checked where 1.44.0 rejected them as unknown
keys, so the first rule matches 1.44.0.

A new gate, `src/__tests__/package-types-compat.test.ts`, runs `npm
pack` on the build, unpacks it into a temporary consumer, and
type-checks every `exports` entry point (ESM and CJS) under `strict` and
`skipLibCheck: false`. The consumer holds the 1.44.0 usage patterns for
B8, B9 and B11.

## The hardening fixes

These fix defects in the MCP recording feature that the review of the
Mastra-parity work found. All of it is new since 1.44.0.

- **Start-time validation and credential redaction.** `llm.record.mcp`
and the record destination are checked in a pre-pass at start, before
anything is mounted. The start fails with an error that names the mount
and the destination for: `null`, `false`, `0` or `""` as
`llm.record.mcp`; mount keys the server can never route (a trailing `/`,
`//`, a query, a fragment, whitespace, a dot segment, an escaped
character), on both `llm.record.mcp` and `--mcp-record` /
`--mcp-proxy-only`; an upstream that is not an http(s) URL; an empty or
non-string `fixturePath`; a broken symbolic link, `EACCES` or a
non-writable directory as the destination; and `maxRecordBufferBytes`
that is not greater than 0 and at most 256 MiB. Credentials in mount
keys are redacted in these errors with the record sanitizer's URL rule.
- **Upstream death mid-body fails loudly on every relay path.** Before,
a truncated upstream answer was relayed as a clean-looking 200, a
`SyntaxError` in the client, or a hang until the client timed out, with
nothing logged. Now the buffered, SSE, pass-through and post-overflow
paths all fail: a 502 before headers, a terminated stream after them,
and a JSON-RPC error event first for an SSE call that never got its
response. Each failure logs one `MCP-RECORD:` line and journals
`recordSkipped: "upstream-aborted"`. A client that disconnects is not
reported as an upstream failure.
- **Config warnings.** `aimock --config` defaulted to the silent log
level, which hid the `MCP-RECORD:` warnings, so a run that recorded
nothing printed nothing. A config whose `llm.record.mcp` has at least
one mount now defaults `logLevel` to `"warn"`. An explicit
`llm.logLevel` still wins, and an empty `llm.record.mcp` stays silent.

## Red/green evidence

Each fix was reproduced on the real surface (the built CLIs, a packed
tarball, real MCP clients `@modelcontextprotocol/sdk` 1.31.0 and
`@modelcontextprotocol/client` 2.2.0) before the change and re-run after
it.

- **S1, B8, B9.** RED: the compat gate's consumer against the base build
failed with `mcp-fakes.d.ts(13,1): error TS1036: Statements are not
allowed in ambient contexts` and `TS2304: Cannot find name 'sideEffect'`
(both `.d.ts` and `.d.cts`), `TS2345: Argument of type '"record"' is not
assignable to parameter of type 'never'`, and `TS2739: ... missing the
following properties from type 'AimockHandle': fakesFor, fakesReport`.
GREEN: the gate passes 2/2. In the packed `dist/`, `sideEffect` and
`declare module` match 0 of 86 `.d.ts` and 86 `.d.cts` files.
- **B11.** RED: a strict consumer with a `Mountable` whose
`setReplaySpeed` is a string fails with `TS2322 ... Types of property
'setReplaySpeed' are incompatible`; npm 1.44.0 compiles it. GREEN: it
compiles, and the `interface Mountable` block in `dist/types.d.ts` and
`.d.cts` is byte-identical to npm 1.44.0.
- **R12.** RED: `LLMock({ replaySpeed: 2 })` with such a mount threw
`TypeError: handler.setReplaySpeed is not a function` at `start()`,
before and after a late mount. GREEN: it starts and the member is never
called, the same as 1.44.0.
- **R13.** RED: a malformed recorded key gave
`"rule":"record-replay/fa-list"`; a bad recorded key with `scope: 5`
gave `bad-block:b` where 1.44.0 gives `bad-block:h`. GREEN:
`"rule":"mcp-fakes/bad-block:h"` with `[record-replay/fa-list]` in the
message. A probe of 722 inputs that 1.44.0 rejects, each loaded both
through `addMcpFakes` and from a file, went from 238 rule mismatches to
0. A separate class of 66 rows, with a recorded key that is valid now,
is listed under the follow-ups below.
- **Start-time validation.** RED: `llm.record.mcp` of `null`, `false`,
`0` and `""`, the mount keys `/mcp/`, `/mcp?x=1`, `/a b` and `//`, an
empty `fixturePath`, a dangling symlink destination and
`maxRecordBufferBytes` of 0, -5 and 314572800 all started (exit 0). A
credentialed URL as a mount key printed
`http://user:tok3nSECRET@host/mcp` in the error. GREEN: each exits 1
with a message that names the mount; the credentialed key prints
`http://host/mcp (credentials removed)`, and the full stderr has 0
secret matches. Valid configs still start.
- **Upstream death mid-body.** RED (killable upstream): SSE record path,
both clients `TIMEOUT 4000ms`; pass-through chunked JSON, a raw 200 with
a truncated body and `SyntaxError: Unterminated string in JSON`;
pass-through content-length, a hang; buffered JSON, `Unexpected content
type: null`; nothing logged. GREEN: SSE gives `McpError -32603 ... the
upstream response failed` at once; the pass-through and post-overflow
paths give `TypeError terminated` at once; buffered JSON gives a 502
with `{"error":"aimock MCP recorder: the upstream response failed"}`.
Each logs `MCP-RECORD: the upstream response failed mid-body
(upstream-aborted) ...` and journals `upstream-aborted`. A client abort
produced no aimock output.
- **Config warnings.** RED: `aimock --config` with an MCP record mount
and a request with no testId printed only the listening line. GREEN: it
prints `MCP-RECORD: forwarded, not recorded: a request on mount /mcp has
no testId or context ...`. `mcp: {}` stays silent, like a config with no
`mcp`.
- The happy path still works: `examples/mcp-record-proof/proof.sh` exits
0, recording server-everything (13 tools, 0 secrets) and replaying it
offline at speed 1 and 100, identical to the recording and to the live
run.

## Not breaking

Every runtime change touches only features added since 1.44.0: MCP
recording (`llm.record.mcp`, `--mcp-record`, `--mcp-proxy-only`), the
recorded `mcpFakes` keys, and the replay-speed hook. The type changes
return the published surface to 1.44.0 and add only new names
(`AimockFakesHandle`).

## Coordination

Another session is fixing the #501, #508, #511 and #513 breaks
separately and will reuse this compat gate. The gate's evidence consumer
still shows their errors (`c-others.ts`, `c508.ts`) and none from #509.
Release PR #516 is held until both land.

## Known follow-ups

- 66 probe rows remain where a recorded key that is now valid, together
with an unrelated error, gives a different first rule than 1.44.0 (for
example `timing` plus `scope: 5` gives `bad-block:b`, not
`bad-block:h`). Both versions reject the input; only the rule code
differs. This waits on decision D4.
- Programmatic `wireMcpRecording` can leave mounts half-wired if a later
step fails.
- `@modelcontextprotocol/client` 2.2.0 maps the 502 to
`CLIENT_HTTP_NOT_IMPLEMENTED` (client side).
- The pass-through SSE and SSE-after-overflow paths terminate the stream
but do not inject a JSON-RPC error event.
- Some test-strength mutants beyond what the new tests cover, and a few
pre-existing log-wording nits.
- `AimockRecordConfig` requires `providers`, but the runtime and docs
accept an `llm.record` with only `mcp`.
- The aimock-cli docs say the `"warn"` default applies when
`llm.record.mcp` is set; the code applies it only when it has at least
one mount.

## Test plan

At the head of this branch, each exit code captured on its own:
`prettier --check .`, `eslint .`, `pnpm typecheck` (three tsconfigs),
`pnpm build`, `package-types-compat.test.ts`, and the recorder,
record-CLI, aimock-cli and MCP fakes test files. The full suite's only
failures are the pre-existing vitest worker `Timeout calling
"onTaskUpdate"` error (it reproduces on the base) and a `validate-cli`
timeout under load that passes alone (96/96); neither file is touched
here.
@jpr5 jpr5 mentioned this pull request Oct 10, 2026
Merged
jpr5 added a commit that referenced this pull request Oct 10, 2026
@jpr5
fix: restore 1.44.0 default behavior; make new Responses/misbehavior/…
c9c19d1 
…strict-args behavior opt-in (#518)

## Summary

With default options, this branch has zero breaking changes compared
with the published `@copilotkit/aimock` 1.44.0. The rule is:

1. An input that 1.44.0 accepted behaves identically.
2. An input that 1.44.0 rejected may now be accepted.
3. New behavior is opt-in.

The changes from #501, #508, #511 and #513 that altered 1.44.0 behavior
now sit behind explicit opt-ins. The 1.44.0 types (`ToolCall`,
`FixtureToolCall`, block unions, `CollapseResult`,
`resolveFixtureBlocks`, WebSocket handler arity,
`FixtureLoadRuleRegistry`) are restored. The new carriers
(`customToolCalls`, `responsesBlocks`, `MisbehaviorConfigError`,
`StartFromConfigOverrides`) are additive.

## Opt-ins

| Feature | Programmatic | CLI flag (`llmock` / `aimock`) | `aimock
--config` key |
|---|---|---|---|
| Extended Responses tool handling (#511/#513): custom tool calls,
namespaces. The fixture keys `match.toolNamespace`, `customToolCalls`
and `responsesBlocks` are honored **only** under `"extended"`; in
`"legacy"` they are ignored, as in 1.44.0 | `responsesTools: "legacy" \|
"extended"` (default `"legacy"`; an invalid value warns and runs
`"legacy"`); aimock-pytest: `AIMockServer(responses_tools="extended")` |
`--responses-tools legacy\|extended`; `aimock validate --responses-tools
legacy\|extended` | `llm.responsesTools` (an invalid value warns and
runs `"legacy"`) |
| Strict invalid tool arguments (#501): 500
`aimock_invalid_tool_arguments` on object wires, authored text on string
wires | `strictToolArguments: true` | `--strict-tool-arguments` |
`llm.strictToolArguments: true` |
| Model misbehavior (#508) | `enableMisbehavior: true` (also passed
through the plugins' `useAimock`) | `--misbehavior` |
`llm.enableMisbehavior: true` (`llm.misbehavior` is read only when
misbehavior is enabled) |
| MCP recording from config | n/a (`MCPMock.enableRecording()` is
unchanged) | n/a (`llmock --mcp-record` / `--mcp-proxy-only` are
unchanged) | `llm.enableMcpRecording: true` (`llm.record.mcp` is read
only with this opt-in; without it the key is ignored with one warning,
as in 1.44.0) |

On the `aimock` bin, a flag overrides the matching config key. If a
boolean flag is absent, the config key applies. Without its opt-in, each
setting keeps the 1.44.0 behavior. For example, invalid fixture tool
arguments are served as `{}` with a warning, and a `misbehavior` key in
a fixture or config is ignored (the config case logs one warning).

## Proof

All comparisons use the packed build of this branch against the npm
1.44.0 tarball.

| Proof | Result |
|---|---|
| Strict consumer matrix: TS 5.0.4 / 5.4.5 / 5.9.3 × bundler /
node16-esm / node16-cjs × 7 entry points (root, mcp, a2a, vector, agui,
vitest, jest), `skipLibCheck: false` | **126/126 cells pass** (exit 0, 0
errors) for both 1.44.0 and this branch |
| Type proofs b01–b12, s01, b07, b10-ws-handler-arity: TS 5.0.4 / 5.4.5
/ 5.9.3 × nodenext / node16-esm / node16-cjs / bundler / node10, strict
| 0 src errors in every cell, **except b12** (see Known open item).
node10 has the same 4 TS1192 lib errors as 1.44.0 |
| `src/__tests__/package-types-compat.test.ts` (strict gate) | 2/2 pass
|
| Runtime A/B, default options: http-ab (all wires × fixtures, stream
and non-stream) | 704 same / 0 diff |
| ws-ab / record-ab / record-ns-ab2 | 28/0, 60/0, 6/0 |
| #501 ab501, 1.44.0 vs default (wire bytes, journal, warn count) |
260/0 |
| #501 ab501, pre-fix main vs `strictToolArguments: true` | 260/0 on
bytes and journal (with strict, Ollama, Interactions and Live also log
the 1.44.0 warning) |
| #508 enabled-ab, main vs `enableMisbehavior: true` | 39/0 |
| Codex CLI 0.161.0 against the packed `dist/cli.js` with
`--responses-tools extended` | **4/4**: custom `apply_patch` over HTTP
and over WS, namespaced MCP over HTTP and over WS. Contrast without the
flag: 404 "No fixture matched", as in 1.44.0 |

There are 3 documented deltas from 1.44.0 with default options. All 3
are allowed by the rule:

1. **`GET /__aimock/misbehavior`**: 1.44.0 returns 404, this branch
returns 200. This is a new additive route (rule 2).
2. **`aimock --config` with a valid or invalid `llm.misbehavior` and no
opt-in**: the exit code (0) and the served responses are identical. The
only difference is one extra stderr warning: `Ignoring llm.misbehavior
because misbehavior is not enabled. ...`
3. **Fixture with invalid JSON tool arguments loaded through `validate`
/ `addFixturesFromJSON`**: 1.44.0 already rejected this input, and this
branch rejects it the same way. The message now ends with a pointer to
`misbehavior: tool-args-invalid-json`.

### Red / green

Every RED was observed on the real failure surface (packed builds driven
over HTTP/WS, the real CLI, and `tsc` consumer projects) before the fix.
Every GREEN was observed afterwards on the same probe.

| Track | RED (before the fix) | GREEN (after the fix) |
|---|---|---|
| #501 invalid tool arguments (ab501: 10 fixtures, 22 HTTP routes in
stream and non-stream modes, 3 WS wires) | 1.44.0 vs pre-fix main: **119
diffs**, 7 on each of the 17 affected wire/mode cells. Controls
identical | 1.44.0 vs fix (default): **260/260 identical** on bytes,
journal and warn count. Audit #501 cells: 30 → 0 (HTTP), 1 → 0 (WS) |
| #508 misbehavior types and WS arity (B7, B10-WS consumer matrix, 15
cells) | Consumer projects fail to compile against main (registry and
4-parameter WS handler breaks) | Same as 1.44.0 in every cell: 0 errors,
except the 4 node10 TS1192 lib errors that 1.44.0 also has |
| #508 `aimock --config` with `llm.misbehavior: "empty-response"`
(fixture `weather` → `sunny`) | 1.44.0 `content="sunny"`, branch before
the fix `content=""` | `content="sunny"` plus the warning. With
`llm.enableMisbehavior: true` or `--misbehavior`: `content=""`. Invalid
value with the opt-in: exit 1 with a rule-prefixed message |
| #511/#513 types (b01–b06, b10) | **14 src errors per cell, 210 in
total** across 15 cells (TS2339 `arguments`, `"function" \| "custom"`,
`CollapseResult.toolCalls`, `resolveFixtureBlocks` arity, ...) | **0 src
errors** in all 15 cells |
| #511/#513 runtime (namespace / custom fixtures F08–F13, F27–F31,
Q01–Q23, loader-ab, record-ns) | Up to 500 on 21/21 routes per fixture,
200 → 400 on Q01–Q06, loader rejects inputs that 1.44.0 accepted, 3/6
recordings differ | Identical to 1.44.0 by default. With
`responsesTools: "extended"`, records `namespace` and `customToolCalls`
|

## Known open item: B12

A consumer object that already has its own **non-fresh** `misbehavior`
key of a different type (for example `misbehavior: string` on a value
assigned to `Fixture`, or `misbehavior: { enabled: true }` on a value
assigned to `MockServerOptions`) no longer assigns to those types: 2 ×
TS2322 per type-proof cell. Adding any optional key has this effect, so
it cannot be avoided without renaming the key. It is the only type proof
that does not compile, and `main` fails it too. This needs a maintainer
decision: accept it, or rename the `misbehavior` key.

## Quality gates (head of this branch)

- `prettier --check .`, `eslint .` and `tsc --noEmit` (`tsconfig.json`,
`tsconfig.test.json`, `scripts/tsconfig.json`) all pass. The `tsdown`
build passes.
- The 65 touched test files, including the strict gate: 3695 passed, 2
skipped (env-gated).
- commitlint over the branch: 0 problems, 0 warnings. All commits are
signed.

## Review fixes (F1, F2, F3, C1, C2, validation)

- **F1**: `match.toolNamespace` is honored only with `responsesTools:
"extended"`. In the default legacy mode it is ignored, as in 1.44.0.
- **F2**: `customToolCalls` and `responsesBlocks` are served only with
`"extended"`. In legacy mode they are dropped on every wire (HTTP, SSE,
Responses WS, Realtime, Gemini Live), as in 1.44.0.
- **F3**: an invalid `responsesTools` (option, constructor or
`llm.responsesTools`) logs one warning and runs `"legacy"` instead of
failing startup. An invalid `--responses-tools` value still exits 1,
because the flag is new.
- **C1**: tests for the `llmock --misbehavior` and `--responses-tools`
pass-through.
- **C2**: a non-boolean `llm.strictToolArguments` or
`llm.enableMisbehavior` logs one warning and the option stays off.
- **Validation**: the validating doors (`validateFixtures`,
`addFixturesFromJSON()`, `POST /__aimock/fixtures`,
`--validate-on-load`, `aimock validate`) check the three keys only in
`"extended"`. New opt-in: **`aimock validate --responses-tools
extended`** (default `legacy` = 1.44.0 parity; an invalid value is a
usage error).
- **aimock-pytest**: new opt-in `AIMockServer(responses_tools=None)`,
which passes `--responses-tools <value>` when set. Unset keeps the
1.44.0 default. The plugin's `toolNamespace` test opts in to
`"extended"`.

| Repro | RED (PR head before the fixes) | GREEN (608a484) |
|---|---|---|
| F1/F2 runtime, default vs 1.44.0 (32 cases + 4 supplementary) | 3/32
identical, 29 differ | **32/32** and **4/4** identical |
| F1/F2 runtime, extended vs PR head extended | n/a | **32/32**
identical (new behavior kept) |
| Validating doors, default vs 1.44.0 (12 inputs x 6 doors) | 20/72
identical, 52 differ | **72/72** identical |
| Validating doors, extended vs PR head | n/a | **48/48** identical |
| F3/C2: `aimock -c` with invalid `responsesTools` / non-boolean flags,
constructor with `responsesTools: "bogus"` | exit 1 / `TypeError`; no
warning for non-boolean flags | starts, one `[aimock] Ignoring ...`
warning each, legacy output identical to 1.44.0 |
| Python plugin suite (CI commands, `AIMOCK_CLI_PATH=dist/cli.js`) | 1
failed (`test_add_fixture_routes_tool_namespace_into_match`: "No fixture
matched"), 247 passed | **248 passed**. Removing the opt-in makes the
test fail again |

Gates at 608a484: full vitest (12 shards) 14,012 passed / 3 skipped;
`package-types-compat.test.ts` 2/2; typecheck, eslint, prettier and
build pass; commitlint clean; all commits signed.

Refs #505 #501 #508 #509 #511 #513


## CI fix: Python Tests (fe5db7c)

All 16 `test (3.10..3.13, 20..26)` jobs failed with `188 failed, 60
passed`. All 188 were in the new
`test_misbehavior_{openai,anthropic,controls}.py` proofs. The pytest
plugin started the CLI without
`--misbehavior`, so faults were ignored, for example
`'{"city":"Paris","units":"metric"}' != '{"city":"Paris",'`,
and `/__aimock/misbehavior` answered `409 Conflict`. No test outside the
misbehavior modules failed, so this is not a
default-behavior regression.

Fix: `AIMockServer(enable_misbehavior=False)` appends `--misbehavior`
when true. The default stays off, as in
1.44.0. The three misbehavior modules override `aimock` with a server
that opts in.

Local proof (Python 3.12, Node 26, `AIMOCK_CLI_PATH=dist/cli.js`, `cd
packages/aimock-pytest && pytest tests/ -v`):

```
RED   (cbcca44): 188 failed, 60 passed, 103 warnings in 71.36s (0:01:11)
GREEN (fe5db7c): 248 passed, 103 warnings in 65.26s (0:01:05)
```

- Bedrock stream default-mode test parses event-stream frames instead of
regex (was flaky on random CRC bytes)

## Review round 2 fixes

- **R2-F1/F2/F3**: in legacy mode, the fixture listing (`GET
/__aimock/fixtures`) and the journal match 1.44.0 byte for byte.
- **Public export parity**: `isContentWithToolCallsResponse`,
`collapseOpenAISSE` and `collapseStreamingResponse` return the 1.44.0
output again (93/93 export probe calls identical to 1.44.0).
- **O7**: when `llm.responsesTools` is invalid, the warning names the
mode that actually runs (`Using "extended" from --responses-tools.`);
with no flag the text is unchanged.
- **record.mcp opt-in**: `llm.record.mcp` is read only with the new
`llm.enableMcpRecording: true`; without it the key is ignored with one
warning, as 1.44.0 ignored it.


## Review round 3 fixes

- **R3-F1**: the stream builders reproduce the 1.44.0 output when
`toolCalls` is not an array.
- **R3-F2**: the `responsesTools` mode is per server, so servers that
share fixtures no longer affect each other.



## Review round 4 fixes

- Default mode keeps caller fixture identity
(addFixture/on/addFixtures/prependFixture) as 1.44.0 did; the per-add
copy happens only with enableMisbehavior
- Extended servers count and journal the caller's fixtures (identity
matches 608a484)
- POST /__aimock/fixtures on an extended server no longer leaks extended
normalization into a shared caller array
- Identity APIs behave the same with enableMisbehavior on
(journal/counts/findByFixture name the caller's fixture; same object
added twice still gives two fault sources)
- Docs: responsesTools mode is fixed at server start
jpr5 added a commit that referenced this pull request Oct 10, 2026
@jpr5
chore: release v1.45.0
c19e178 
Includes #501, #504, #506, #508, #509, #510, #511, #512, #513, #514, #515, #517, #518.

## [1.45.0] - 2026-10-10

Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior
that would change those inputs is opt-in.

### Added

- Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record
  <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image),
  `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the
  opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and
  `MCPMock.enableRecording()` / `disableRecording()` (#509).
- Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`,
  `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509).
- Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by
  `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`,
  `durationMs`) need 1.45.0 or later (#509).
- Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509).
- Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by
  `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest
  option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and
  `AimockTestIdCollisionError` (#509).
- aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and
  `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511).
- Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM
  (`examples/mcp-workflow-targets`), plus record and report proofs (#509).
- Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs,
  length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires,
  with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the
  `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true`
  (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`,
  or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an
  `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw
  the new `MisbehaviorConfigError` (#508).
- Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and
  WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace`
  applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0.
- Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI
  `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools
  inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are
  visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for
  `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and
  `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types:
  `CustomToolCallMessage`, `ResponsesToolsMode` (#511).
- Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`,
  `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools:
  "extended"` is set (#513).
- Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config
  `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string
  wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501).
- Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift
  checks (#508).

### Fixed

- MCP recording checks its config at start, redacts credentials in errors, fails loudly when the
  upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock
  --config` with record mounts to `logLevel: "warn"` (#517).
- Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced
  and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made
  each new behavior opt-in (#518).
- Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and
  `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck:
  false` (#517).
- Align Cohere streaming length-stop faults with captured provider framing (#508).
- Retain drift evidence across retries and for oversized records (#510, #512).
- Classify `claude-haiku-5-5` in the drift model registry (#515).
- Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight,
  and the copy-button fallback (#504, #506, #514).
jpr5 added a commit that referenced this pull request Oct 10, 2026
@jpr5
chore: release v1.45.0
51c1447 
Includes #501, #504, #506, #508, #509, #510, #511, #512, #513, #514, #515, #517, #518.

## [1.45.0] - 2026-10-10

Every fixture, config and request that 1.44.0 accepted behaves the same by default. New behavior
that would change those inputs is opt-in.

### Added

- Record live MCP servers into `mcpFakes` files and replay them offline: `--mcp-record
  <mount>=<url>` and `--mcp-proxy-only <mount>=<url>` (`llmock` bin and Docker image),
  `llm.record.mcp` with the opt-in `llm.enableMcpRecording: true` for `aimock --config` (without the
  opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored it), and
  `MCPMock.enableRecording()` / `disableRecording()` (#509).
- Recordings never contain headers and redact known secrets (`AIMOCK_RECORD_SECRET_VALUES`,
  `AIMOCK_MCP_UPSTREAM_AUTH`); a file is not written while a secret remains (#509).
- Recorded MCP fakes replay list order, progress and log notifications, and call timing (scaled by
  `--replay-speed`). The recorded `mcpFakes` keys (`list`, `recorded`, `timing`, `notifications`,
  `durationMs`) need 1.45.0 or later (#509).
- Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509).
- Vitest and Jest: `fakesFor()` and `fakesReport()` on a new `AimockFakesHandle` returned by
  `useAimock()`, and opt-in `useAimock({ fakesReport: "warn" | "fail" })` (default `"off"`); Jest
  option `testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError` and
  `AimockTestIdCollisionError` (#509).
- aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and `assert_fakes_report` (#509), and
  `AIMockServer` opt-ins `enable_misbehavior` and `responses_tools` (#508, #511).
- Workflow-target examples: a Mastra Workflow and a LangGraph JS graph call MCP fakes with no LLM
  (`examples/mcp-workflow-targets`), plus record and report proofs (#509).
- Model misbehavior faults (invalid tool arguments, schema violations, unknown tools, duplicate IDs,
  length stops, empty responses, refusals, content filtering, reasoning-only) across provider wires,
  with seeded rates, `/__aimock/misbehavior` runtime controls, journal evidence and the
  `aimock_misbehavior_total` metric. Off unless enabled with `enableMisbehavior: true`
  (programmatic, including `useAimock`), `llmock --misbehavior` / `aimock validate --misbehavior`,
  or `aimock --misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an
  `llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid misbehavior settings throw
  the new `MisbehaviorConfigError` (#508).
- Namespaced function calls and custom tool calls on the OpenAI Responses API, over HTTP, SSE and
  WebSocket, plus `match.toolNamespace` (#511). Like the fixture fields below, `match.toolNamespace`
  applies only with `responsesTools: "extended"`; in the default mode it is ignored, as in 1.44.0.
- Responses namespaced and custom tool calls: with `responsesTools: "extended"` (CLI
  `--responses-tools extended`, also on `aimock validate`; config `llm.responsesTools`), tools
  inside `namespace` tools, `custom` tools and `additional_tools` / `tool_search_output` tools are
  visible to `toolName`, `toolNamespace` and predicates, custom tool rounds count for
  `hasToolResult`, `toolCallId` and `turnIndex`, and fixtures can answer with namespaced calls and
  `customToolCalls` / `responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New types:
  `CustomToolCallMessage`, `ResponsesToolsMode` (#511).
- Record mode keeps Responses namespaces and custom tool calls (`customToolCalls`,
  `responsesBlocks`, `CollapseResult.customToolCalls`) in streamed recordings when `responsesTools:
  "extended"` is set (#513).
- Strict tool arguments: with `strictToolArguments` (CLI `--strict-tool-arguments`, config
  `llm.strictToolArguments`), malformed tool-call argument strings are sent unchanged on string
  wires and return an explicit error on object wires. The default keeps 1.44.0 behavior (#501).
- Vertex and Bedrock modeled-contract comparisons for the misbehavior faults in the daily drift
  checks (#508).

### Fixed

- MCP recording checks its config at start, redacts credentials in errors, fails loudly when the
  upstream dies mid-body (502 or a terminated stream, logged and journaled), and defaults `aimock
  --config` with record mounts to `logLevel: "warn"` (#517).
- Kept 1.44.0 default behavior for strict tool arguments, model misbehavior, Responses namespaced
  and custom tools, `llm.record.mcp`, fixture identity and the exported stream helpers, and made
  each new behavior opt-in (#518).
- Kept the published types compatible with 1.44.0: `AimockHandle`, `Mountable` and
  `McpFakeAddOrigin` keep their 1.44.0 shape, and every entry point type-checks with `skipLibCheck:
  false` (#517).
- Align Cohere streaming length-stop faults with captured provider framing (#508).
- Retain drift evidence across retries and for oversized records (#510, #512).
- Classify `claude-haiku-5-5` in the drift model registry (#515).
- Docs: dark-site font smoothing, consistent homepage comparison headings, the test-suite highlight,
  and the copy-button fallback (#504, #506, #514).
- Invalid opt-in values never stop startup: an invalid `responsesTools` warns once and runs
  `"legacy"`, and a non-boolean `llm.strictToolArguments`, `llm.enableMisbehavior` or
  `llm.enableMcpRecording` warns once and stays off (#518).
jpr5 added a commit that referenced this pull request Oct 10, 2026
@jpr5
chore: release v1.45.0 (#516)
bc43634 
Release npm/plugin **1.45.0** and **aimock-pytest 0.7.0** from `main` at
c9c19d1 (#518 merged). Every fixture, config and request that 1.44.0
accepted behaves the same by default; new behavior is opt-in (#518).
This branch was rebuilt as one commit on top of the #518 merge,
replacing the earlier release commit bdfaeef.

> **Prepared, not merged.** `publish-release.yml` runs on every push to
`main`. Merging this PR publishes: npm `@copilotkit/aimock@1.45.0`, the
`v1.45.0` tag, a force-move of the `v1` major tag, the GitHub Release,
the Docker images (`ghcr.io/copilotkit/aimock` and
`ghcr.io/copilotkit/llmock`, `:1.45.0` and `:latest`), `aimock-pytest`
0.7.0 on PyPI and the Slack announcement. The docs site deploys from
`main:/docs` on merge. Merge only when the release is authorized. Do not
dispatch `publish-release.yml` from this branch.

## Versions

| Surface | Before | After |
|---|---|---|
| `package.json` `version` | 1.44.0 | 1.45.0 |
| `.claude-plugin/plugin.json` `version` | 1.44.0 | 1.45.0 |
| `.claude-plugin/marketplace.json` `source.version` | ^1.44.0 | ^1.45.0
|
| `charts/aimock/Chart.yaml` `appVersion` (chart `version` stays 0.1.0)
| 1.44.0 | 1.45.0 |
| `docs/index.html` CLI banner | aimock v1.44.0 | aimock v1.45.0 |
| `packages/aimock-pytest/pyproject.toml` `version` | 0.6.0 | 0.7.0 |
| `packages/aimock-pytest/src/aimock_pytest/_version.py`
`AIMOCK_VERSION` | 1.44.0 | 1.45.0 |
| `packages/aimock-pytest/README.md` `--aimock-version` default | 1.44.0
| 1.45.0 |
| `skills/write-fixtures/SKILL.md` `toolNamespace` pin note | "newer
than its default 1.44.0 pin" | "aimock 1.45.0+, its default pin from
0.7.0" |

"Unreleased" wording now names 1.45.0 in `README.md` (misbehavior note),
`docs/cohere/index.html` (length-stop fidelity) and
`docs/model-misbehavior/index.html` (two places).

The pytest pin bump matters: the new
`AIMockServer(enable_misbehavior=..., responses_tools=...)` options pass
`--misbehavior` / `--responses-tools`, which the 1.44.0 CLI does not
accept. With the pin at 1.45.0, default-download users get a server that
supports them.

**1.45.0 is a minor.** With #518, default behavior and the published
types match 1.44.0. The remaining type additions (the optional
`misbehavior` key, `ChatCompletionRequest.customTools`, `useAimock`
returning `AimockFakesHandle`) were accepted by the maintainer as
additive optional typing (minor).

## CHANGELOG

## [1.45.0] - 2026-10-10

Every fixture, config and request that 1.44.0 accepted behaves the same
by default. New behavior that would change those inputs is opt-in.

### Added

- Record live MCP servers into `mcpFakes` files and replay them offline:
`--mcp-record <mount>=<url>` and `--mcp-proxy-only <mount>=<url>`
(`llmock` bin and Docker image), `llm.record.mcp` with the opt-in
`llm.enableMcpRecording: true` for `aimock --config` (without the
opt-in, `llm.record.mcp` is ignored with a warning, as 1.44.0 ignored
it), and `MCPMock.enableRecording()` / `disableRecording()` (#509).
- Recordings never contain headers and redact known secrets
(`AIMOCK_RECORD_SECRET_VALUES`, `AIMOCK_MCP_UPSTREAM_AUTH`); a file is
not written while a secret remains (#509).
- Recorded MCP fakes replay list order, progress and log notifications,
and call timing (scaled by `--replay-speed`). The recorded `mcpFakes`
keys (`list`, `recorded`, `timing`, `notifications`, `durationMs`) need
1.45.0 or later (#509).
- Per-test MCP fake report at `GET /__aimock/mcp/fakes/report` (#509).
- Vitest and Jest: `fakesFor()` and `fakesReport()` on a new
`AimockFakesHandle` returned by `useAimock()`, and opt-in `useAimock({
fakesReport: "warn" | "fail" })` (default `"off"`); Jest option
`testIdRoot`. New exports `assertFakesReport`, `AimockFakesReportError`
and `AimockTestIdCollisionError` (#509).
- aimock-pytest 0.7.0: `fakes_for`, `fakes_report` and
`assert_fakes_report` (#509), and `AIMockServer` opt-ins
`enable_misbehavior` and `responses_tools` (#508, #511).
- Workflow-target examples: a Mastra Workflow and a LangGraph JS graph
call MCP fakes with no LLM (`examples/mcp-workflow-targets`), plus
record and report proofs (#509).
- Model misbehavior faults (invalid tool arguments, schema violations,
unknown tools, duplicate IDs, length stops, empty responses, refusals,
content filtering, reasoning-only) across provider wires, with seeded
rates, `/__aimock/misbehavior` runtime controls, journal evidence and
the `aimock_misbehavior_total` metric. Off unless enabled with
`enableMisbehavior: true` (programmatic, including `useAimock`), `llmock
--misbehavior` / `aimock validate --misbehavior`, or `aimock
--misbehavior`, or the config opt-in `llm.enableMisbehavior: true` (an
`llm.misbehavior` key alone never enables it, as in 1.44.0). Invalid
misbehavior settings throw the new `MisbehaviorConfigError` (#508).
- Namespaced function calls and custom tool calls on the OpenAI
Responses API, over HTTP, SSE and WebSocket, plus `match.toolNamespace`
(#511). Like the fixture fields below, `match.toolNamespace` applies
only with `responsesTools: "extended"`; in the default mode it is
ignored, as in 1.44.0.
- Responses namespaced and custom tool calls: with `responsesTools:
"extended"` (CLI `--responses-tools extended`, also on `aimock
validate`; config `llm.responsesTools`), tools inside `namespace` tools,
`custom` tools and `additional_tools` / `tool_search_output` tools are
visible to `toolName`, `toolNamespace` and predicates, custom tool
rounds count for `hasToolResult`, `toolCallId` and `turnIndex`, and
fixtures can answer with namespaced calls and `customToolCalls` /
`responsesBlocks`. The default `"legacy"` keeps 1.44.0 behavior. New
types: `CustomToolCallMessage`, `ResponsesToolsMode` (#511).
- Record mode keeps Responses namespaces and custom tool calls
(`customToolCalls`, `responsesBlocks`, `CollapseResult.customToolCalls`)
in streamed recordings when `responsesTools: "extended"` is set (#513).
- Strict tool arguments: with `strictToolArguments` (CLI
`--strict-tool-arguments`, config `llm.strictToolArguments`), malformed
tool-call argument strings are sent unchanged on string wires and return
an explicit error on object wires. The default keeps 1.44.0 behavior
(#501).
- Vertex and Bedrock modeled-contract comparisons for the misbehavior
faults in the daily drift checks (#508).

### Fixed

- MCP recording checks its config at start, redacts credentials in
errors, fails loudly when the upstream dies mid-body (502 or a
terminated stream, logged and journaled), and defaults `aimock --config`
with record mounts to `logLevel: "warn"` (#517).
- Kept 1.44.0 default behavior for strict tool arguments, model
misbehavior, Responses namespaced and custom tools, `llm.record.mcp`,
fixture identity and the exported stream helpers, and made each new
behavior opt-in (#518).
- Kept the published types compatible with 1.44.0: `AimockHandle`,
`Mountable` and `McpFakeAddOrigin` keep their 1.44.0 shape, and every
entry point type-checks with `skipLibCheck: false` (#517).
- Align Cohere streaming length-stop faults with captured provider
framing (#508).
- Retain drift evidence across retries and for oversized records (#510,
#512).
- Classify `claude-haiku-5-5` in the drift model registry (#515).
- Docs: dark-site font smoothing, consistent homepage comparison
headings, the test-suite highlight, and the copy-button fallback (#504,
#506, #514).
- Invalid opt-in values never stop startup: an invalid `responsesTools`
warns once and runs `"legacy"`, and a non-boolean
`llm.strictToolArguments`, `llm.enableMisbehavior` or
`llm.enableMcpRecording` warns once and stays off (#518).

## Release checklist

- [x] Opt-in names for #501, #508, #511, #513 match the merged compat PR
#518 (`strictToolArguments` / `--strict-tool-arguments` /
`llm.strictToolArguments`; `enableMisbehavior` / `--misbehavior` /
`aimock validate --misbehavior` / `llm.enableMisbehavior`;
`responsesTools` / `--responses-tools` / `aimock validate
--responses-tools` / `llm.responsesTools`; `llm.enableMcpRecording`;
pytest `enable_misbehavior` / `responses_tools`), checked against the
source on this branch.
- [x] Pytest pin `AIMOCK_VERSION` = `"1.45.0"` and the README
`--aimock-version` default = 1.45.0.
- [x] aimock-pytest `version` 0.6.0 -> 0.7.0.
- [x] Homepage "Requires 1.45.0+" (MCP scenario fakes card) and the
other "1.45.0 or later" docs notes match this release.
- [x] Version surfaces bumped (table above). `git grep -F 1.44.0`: every
remaining hit is a history comparison ("as in 1.44.0", "up to 1.44.0",
the 1.44.0 CHANGELOG section) or recorded `aimockVersion` provenance in
fixtures/examples.
- [x] "Unreleased" wording removed from README and docs.
- [x] `src/__tests__/package-types-compat.test.ts` on the release head:
2/2 pass.
- [ ] 1.44.0 A/B probe re-run on the release head: not re-run here; this
commit changes only version strings, docs text and the CHANGELOG, and
#518 carries the A/B proof on the same code.

## Local gates (release head)

- `pnpm install --frozen-lockfile`, `format:check`, `lint`, `typecheck`,
`build`, `test:exports` (publint + attw): pass
- Full vitest, 12 shards: 14,069 passed, 3 skipped (env-gated), 0 failed
- aimock-pytest, installed as CI does (hash-pinned hatchling,
`--no-build-isolation`), Python 3.12, `AIMOCK_CLI_PATH=dist/cli.js`: 248
passed
- `npm pack --dry-run`: `copilotkit-aimock-1.45.0.tgz`, 766 files
(1.44.0: 722), dist/, fixtures/, skills/, .claude-plugin/, README,
CHANGELOG, LICENSE; no src or tests
- commitlint: 0 problems, 0 warnings. Commit is SSH-signed.
@jpr5 jpr5 mentioned this pull request Oct 10, 2026
Closed
@jpr5 jpr5 mentioned this pull request Oct 21, 2026
Merged
jpr5 added a commit that referenced this pull request Oct 21, 2026
@jpr5
docs: refresh What's New and strengthen the comparison matrix (#570)
c5a6373 
Refreshes the docs overview "What's New" cards for the last month of
work, and adds three verified rows to the comparison matrix (homepage
and the four scanned migration pages).

## What's New (`docs/docs/index.html`)

The cards were last updated in May. The 5 old cards are replaced with 8
cards (two full rows of the 4-column grid). They cover CHANGELOG 1.43.0
(2026-09-22) through `[Unreleased]` (1.46.0, #562):

| Card | Covers | Links to |
| --- | --- | --- |
| Model Catalogs & Token Counts | Native Anthropic/Gemini model list and
detail routes. The Anthropic list on shared `/v1/models` is opt-in
(`modelCatalog`). Anthropic `count_tokens` and Gemini `countTokens`
estimates (#542, #551, #568) | `/claude-messages#model-catalog` |
| Native Auth Errors | Opt-in provider-native 401 bodies (`authErrors`)
and the opt-in required-header policy (`requiredHeaders`) (#542, #566) |
`/claude-messages#authentication-errors` |
| Model Misbehavior | Opt-in misbehavior faults (#508, #518) |
`/model-misbehavior` |
| MCP Record & Replay | MCP recording/replay, scenario-scoped tool
fakes, per-test fake reports (#490, #509, #517) |
`/mcp-mock#recorded-fakes` |
| Stream Realism | Anthropic streams stop and journal on client
disconnect (#557). Opt-in `nonStreamingTiming` for OpenAI non-streaming
replies (#545, #561, #567) | `/streaming-physics#non-streaming` |
| Responses API Tools | Opt-in (`responsesTools: "extended"`) namespaced
and custom tool calls, plus message phases (#511, #513, #531) |
`/responses-api#namespaced-and-custom-tool-calls` |
| OpenAI Platform APIs | Files, Batches, Fine-tuning jobs, Vector Stores
(#445, #446, #447, #497) | `/vector-stores` |
| GPT-Live Sessions | GPT-Live mock, record and replay (#468) |
`/websocket#openai-live` |

The Model Catalogs & Token Counts, Native Auth Errors and Stream Realism
cards carry a "New in 1.46.0" label, and the Responses API Tools card
marks its message phases "(new in 1.46.0)". Every opt-in is called out
on its card: `requiredHeaders`, `authErrors`, `modelCatalog` (the
Anthropic shared list), `nonStreamingTiming`, misbehavior and extended
Responses tools. The version banner is not touched.

## Comparison matrix

### New rows

Homepage (`docs/index.html`), inserted before "Dependencies":

| Row | aimock | MSW | VidaiMock | mock-llm | piyook/llm-mock |
mokksy/ai-mocks | MockServer | WireMock |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Native model catalogs (Anthropic, Gemini) | Built-in ✓ | ✗ | OpenAI
format only | OpenAI format only | ✗ | ✗ | ✗ | ND |
| Token-count endpoints | Built-in ✓ | ✗ | Gemini only (fixed count) | ✗
| ✗ | ✗ | ✗ | ND |
| Native 401 auth errors | Opt-in ✓ | manual | ✓ | manual | partial |
manual | partial | manual |

The migration pages for VidaiMock, mock-llm, piyook and Mokksy get the
same three rows, in each page's own style: VidaiMock uses text cells,
and the other three use the ✓/✗ glyphs with text for partial or manual
values. No new legend marks are needed. The MSW, Python-mocks and
openai-responses pages have different columns, so they are unchanged.

Every aimock ✓ cell on the three new rows carries a muted "1.46.0+"
label, on the homepage and on the four migration pages. The label uses a
`.version-note` class in the muted style of the existing "Requires
1.45.0+" table note.

### Rows considered and not added

- Already on the matrix: record & replay, drift detection, MCP / A2A /
AG-UI, streaming physics, chaos testing, model misbehavior, provider
count.
- Client-disconnect handling: dropped. Most competitors' source and docs
say nothing about it, so the row would be mostly ND. A server that drops
a closed stream is normal framework behavior, not something a
competitor's docs state.
- Recorded timing replay: dropped. I could not verify the MockServer and
WireMock cells from their docs, and Streaming physics already covers
timing.

### Per-cell evidence (checked 2026-10-20)

Competitor sources are the repos listed in
`scripts/update-competitive-matrix.ts`, at the default-branch HEAD on
that date: VidaiMock `e61b8ba`, mock-llm `aeb04d6`, piyook/llm-mock
`9e7a3ce`, mokksy/ai-mocks `fc70777`, MockServer `c892f58`. The homepage
table has an HTML comment on each new row with a short evidence note.

**Native model catalogs**
- aimock: Gemini `GET /v1beta/models` list and detail are native by
default. Anthropic `/v1/models/{id}` detail is native by default. The
Anthropic list on shared `/v1/models` needs `modelCatalog: "native"`.
See `/claude-messages#model-catalog` and `/gemini#model-catalog`.
- MSW: has no LLM endpoints. It is marked ✗, the same as the Embeddings
and Image rows.
- VidaiMock: `src/server.rs` registers `GET /v1/models`,
`/v1beta/models` and `/v1beta/openai/models` on `models_handler`. (Its
Anthropic `GET /v1/models/{model_action}` route is never registered,
because the Gemini `POST /v1/models/{model_action}` route claims the
path first.) In `src/handlers.rs`, `models_handler` always returns the
OpenAI list shape:
`{"object":"list","data":[{"id","object":"model","created","owned_by":"vidai"}]}`.
So its Gemini catalog route is not in native format, and there is no
Anthropic catalog.
- mock-llm: the README's default config ships one OpenAI-format
`/v1/models` rule (`{"object":"list","data":[{"id":"gpt-5.2",...}]}`).
There is no Anthropic or Gemini catalog.
- piyook/llm-mock: the README's "Available endpoints" lists only chat
completions, `/v1/messages` and `/v1/embeddings`. `src/api/` has only
`completions` and `embeddings`.
- mokksy/ai-mocks: `MockAnthropic` exposes only `messages()` (`POST
/v1/messages`), and `MockGemini` exposes only `generateContent` and
`generateContentStream`. The repo has no models route (code search for
`v1/models` finds nothing).
- MockServer: the `httpLlmResponse` supported-provider path table
(mock-server.com `llm_response_mocking.html`) has no models route.
`docs/code/llm-mocking.md` names `GET /v1/models` only as proxied
traffic that gets no telemetry.
- WireMock: has no LLM-specific docs (`wiremock.org/docs/solutions/ai/`
is about using AI to write stubs). It is ND, the same as WireMock's
other LLM-endpoint cells.

**Token-count endpoints**
- aimock: Anthropic `POST /v1/messages/count_tokens` and Gemini
`:countTokens` estimate tokens from the translated request. See
`/claude-messages#token-counts` and `/gemini#token-counts`.
- VidaiMock: `config/providers/gemini_count_tokens.yaml` matches
`^/v1(beta)?/models/[^:]+:countTokens$`. Its template
`config/templates/gemini/count_tokens.json.j2` always returns
`"totalTokens": 9`. There is no Anthropic `count_tokens` provider
config.
- mock-llm and mokksy/ai-mocks: GitHub code search for `count_tokens`
(mock-llm) and `countTokens` (mokksy) finds nothing. Neither README or
API has a count route.
- piyook/llm-mock: `src/api/` has only `completions` and `embeddings`,
and the endpoint list has no count route.
- MockServer: `docs/code/llm-mocking.md` names Anthropic
`/v1/messages/count_tokens` only as unaccounted proxied traffic. There
is no `httpLlmResponse` for it.
- MSW ✗ and WireMock ND, for the same reasons as the catalog row.

**Native 401 auth errors**
- aimock: opt-in `policy.authErrors: "native"` returns Anthropic
`authentication_error` and Gemini `UNAUTHENTICATED` bodies. See
`/claude-messages#authentication-errors` and
`/gemini#authentication-errors`.
- VidaiMock ✓: the README lists `X-Mock-Status` and `?chaos_status=`
overrides that return "real provider error envelopes".
`config/templates/anthropic/error.json.j2` maps 401 to
`authentication_error`. `config/templates/gemini/error.json.j2` maps 401
to `UNAUTHENTICATED`.
- piyook/llm-mock partial: chaos `status` accepts any status from 400 to
599. `src/utilities/chaos.ts` types every status except 429/529 (Claude)
or 429/503 (Gemini) as `api_error` or `INTERNAL`. So a 401 is not typed
`authentication_error` or `UNAUTHENTICATED`. The README also says
"LLMock has no authentication".
- mock-llm manual: the README example is a hand-written rule that
returns an OpenAI `invalid_api_key` 401.
- mokksy/ai-mocks manual: `AbstractBuildingStep.respondsError` lets you
set any status and body.
- MockServer partial: the LLM chaos `errorStatus` accepts 401. With no
`errorKind`, `LlmErrorBodies.kindForStatus`
(`mockserver-core/src/main/java/org/mockserver/llm/LlmErrorBodies.java`)
maps every status except 429 and 529 to `SERVER_ERROR`. So a 401 gets
the provider envelope typed `api_error` or `INTERNAL`, not
`authentication_error` or `UNAUTHENTICATED`. This is the same behavior
as piyook.
- MSW and WireMock manual: both are generic HTTP stubs with any status
and body.

### Weekly scanner compatibility

No `FEATURE_RULES` entry or migration variant names the new rows, so the
Monday scan never reads or writes them. The `GET /v1/models` rule stays
row-less. Its keyword (`/v1/models`) would match any OpenAI models
endpoint, which is not what the catalog row compares.

A new test, `hand-reviewed homepage rows with no scan rule` in
`competitive-matrix-homepage.test.ts`, checks four things:
- every new label is on the homepage;
- no rule or migration variant names a new label;
- when every competitor matches every rule, the homepage keeps every
new-row cell;
- when every rule matches, each scanned migration page keeps its new
rows.

I mutation-tested it: adding `"Token-count endpoints"` as a migration
variant fails 4 of its tests.

`docs-matrix-legend.test.ts`: the homepage ND count goes from 45 to 47,
because of the two WireMock ND cells.

Dry run (`npx tsx scripts/update-competitive-matrix.ts --dry-run`, with
a GitHub token), before and after the change:

```
before: Parsed 41 capability rows, 8 competitor columns.
after:  Parsed 44 capability rows, 8 competitor columns.
both:   No homepage matrix changes detected.
        1 migration page row(s) to check by hand: mock-llm / Azure OpenAI (combined-row)   [pre-existing]
        12 row-less detection(s) to check by hand                                        [identical list]
```

The only difference is the row count. No change is proposed to the new
rows or anywhere else.

## Checks

- `prettier --check .`, `eslint .` and `tsdown` build: pass.
- The docs and competitive-matrix tests (`docs-*`,
`competitive-matrix*`, `mcp-fakes-doc-samples`): 13 files, 530 tests,
all pass.
- Full `vitest run`: 17610 passed. One test failed: a timing assertion
in `chaos-fal-single-roll.test.ts`. That file only measures chaos
latency, and this PR does not change it. It passes when run alone (4/4).
- Local Playwright CLI screenshots, at 375x667, 430x932 and 1440x900:
  - What's New, on this branch and on main.
  - The homepage matrix, on this branch and on main.
  - The four edited migration pages.
- On phones, the homepage matrix scrolls sideways inside its wrapper, as
it does on main. The new rows add no width.
- What's New now fills two full grid rows. On main, the fifth card sat
alone in a second row.
- The "On this page" panel overlaps the quick-start card at 1440 wide.
Main does the same.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpr5