test(drift): accept OpenAI's three live K9 summary shapes on the provider side

[jpr5]

Problem

Two live drift legs on main fail intermittently with expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ]:

• openai-responses.drift.ts: K9:provider:exhaustion-contract
• ws-responses.drift.ts: K9:native:exhaustion-contract

This is not drift in aimock. Live o4-mini picks one of three reasoning-summary shapes at random for the same K9 request (o4-mini, effort high, summary auto, max_output_tokens: 32).

Measurements (2026-10-10, exact K9 request, no changes between runs)

Surface	full-summary (9 events)	no-summary (5)	unclosed-summary (7)
HTTP raw probe, 12 runs	8	2	2
HTTP drift test, 10 runs	8	2	0
WS drift test, 10 runs	5	3	2

• no-summary: created, in_progress, output_item.added, output_item.done, incomplete. The reasoning item has summary: [].
• unclosed-summary: the summary part opens and streams deltas, but reasoning_summary_text.done and reasoning_summary_part.done never arrive. The final item has summary: [].
• Every run ended response.incomplete / max_output_tokens with output_tokens: 0 on o4-mini-2025-04-16.

Change

This follows the principle already applied to Gemini K5/K9: accept real upstream variants on the live/native side only.

• OPENAI_K9_SUMMARY_SHAPES in helpers.ts names exactly the three observed shapes, with the measured frequencies in a comment. Any other summary run still fails. The check is not "any prefix".
• Provider/native K9 grading accepts those three shapes, checks each one's payload, and still requires:
  • the response.incomplete terminal with reason max_output_tokens
  • output_tokens === 0
  • the requested model, or a dated snapshot of it, matching across created, in_progress and the terminal.
• Localhost grading is unchanged and must still emit the full summary. The localhost-vs-live comparison now uses only the event order outside the summary run, plus status and reason.
• Retained live captures of each shape (HTTP SSE and WS events) live in src/__tests__/drift/fixtures/ and replay without a credential. Mutants must fail:
  • an 8-event out-of-order summary sequence
  • a wrong terminal status
  • nonzero output_tokens
  • a different model
  • localhost grading fed the 5-event or 7-event shape.
• The live test bodies move into compareExhaustion / compareWSExhaustion so the replays use the same comparison path as the live tests, including the WS SDK/real/mock shape triangulation.

No src/ changes, so there is no public API or type impact.

Local red-green

RED: the replays run through the unfixed check (env -u OPENAI_API_KEY npx vitest run --config vitest.config.drift.ts src/__tests__/drift/openai-responses.drift.ts src/__tests__/drift/ws-responses.drift.ts -t "variant replay"):

✓ WS  retained full-summary native events agree with actual localhost
× WS  retained no-summary native events agree with actual localhost
  1. [critical] expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ]
× WS  retained unclosed-summary native events agree with actual localhost
  1. [critical] expected [ 'response.created', …(6) ] to deeply equal [ 'response.created', …(8) ]
× WS  retained full-summary native events reject ... nonzero output tokens   (AssertionError: expected [Function] to throw an error)
✓ HTTP retained full-summary provider stream agrees with actual localhost
× HTTP retained no-summary provider stream agrees with actual localhost
  1. [critical] expected [ 'response.created', …(4) ] to deeply equal [ 'response.created', …(8) ]
× HTTP retained unclosed-summary provider stream agrees with actual localhost
  1. [critical] expected [ 'response.created', …(6) ] to deeply equal [ 'response.created', …(8) ]
× HTTP retained full-summary provider stream rejects ... nonzero output tokens   (AssertionError: expected [Function] to throw an error)
Tests  6 failed | 12 passed | 23 skipped (41)

GREEN: the same command after the fix: Tests 18 passed | 23 skipped (41). All three shapes pass on both surfaces. Every mutant is rejected, and localhost still rejects the 5-event and 7-event shapes.

Mutation check of the new assertions. Each mutant was applied and then reverted:

• dropping the output_tokens check fails 6 tests
• letting localhost accept the variants fails 4 tests
• dropping the model check fails 6 tests.

Live GREEN: two consecutive runs of both full drift files with OPENAI_API_KEY (npx vitest run --config vitest.config.drift.ts src/__tests__/drift/openai-responses.drift.ts src/__tests__/drift/ws-responses.drift.ts):

run 1: Tests 39 passed | 2 skipped (41)   HTTP K9 live: 9 events, WS K9 live: 9 events
run 2: Tests 39 passed | 2 skipped (41)   HTTP K9 live: 7 events (unclosed-summary), WS K9 live: 9 events

Gates

pnpm typecheck, pnpm lint, pnpm format:check, pnpm build: all pass. pnpm test: 377 files, 14073 passed.

Review round 1

Each finding was checked before any change. Every change has a mutation red-green.

• A1 (capture replay mode): no workflow sets OPENAI_RESPONSES_EXHAUSTION_CAPTURE_DIR. A grep of .github/ returns nothing; the only references are openai-responses.drift.ts:92/300-310/367. The k9-20261008T170312Z capture is not in the repo or in Actions artifacts, and PR feat: add deterministic model misbehavior faults #508 says no artifact host holds it. It can't be replayed, so CI never uses that path. No change.
• A2 (isOpenAIModelSnapshot): it already behaves correctly. o4-mini and o4-mini-2025-04-16 pass. o4-mini-high, o4-mini-foo, o4-mini-2025-04 and o4-mini2025-04-16 are all rejected. No change.
• A3 (which output_tokens field): the assertion reads terminal.response.usage.output_tokens, where terminal = events.at(-1) narrowed to response.incomplete. However, the HTTP mutant's replaceAll also rewrote the zero-valued tool_usage fields in created and in_progress, so it didn't isolate the terminal. The mutant now sets only the terminal's usage to 5 and asserts that everything before the terminal is byte-identical. Removing the assertion fails 6 tests (HTTP and WS).
• A4 (terminal-only model): added on both surfaces. Only the terminal gets o4-mini-2099-01-01, a valid snapshot form that differs from created/in_progress. Deleting the consistency loop fails 6 tests.
• A5 (unmatched shape label): an unmatched live order was already rejected. It fell back to the full-summary order and failed toEqual with "deeply equal", as the 8-event test showed. The fallback label is gone: an unmatched order now throws live K9 event order matches none of the accepted summary shapes (full-summary, no-summary, unclosed-summary): …. Restoring the fallback fails the two 8-event tests.

Live, after the review fixes: two consecutive runs of both full drift files:

run 1: Tests 39 passed | 2 skipped (41)   HTTP K9 live: 9 events, WS K9 live: 9 events
run 2: Tests 39 passed | 2 skipped (41)   HTTP K9 live: 5 events (no-summary), WS K9 live: 9 events

Gates: typecheck, lint, format:check and build pass. pnpm test: 377 files, 14073 passed.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@523

commit: 3d2dbfc