Windows: vitest pool worker killed by a V8 compiler-zone OOM, invisible in the JSON report

[bompus]

Summary

Running the full suite on Windows kills a vitest pool worker with a V8 compiler-zone OOM. It has reproduced across separate runs with a byte-identical native stack. The crash is not visible in the JSON report — a dying worker is not a test failure, so it appears in no testResults entry and vitest recycles the worker and continues. A run can therefore report an internally consistent pass/fail summary while a worker was in fact killed.

This is a Zone OOM, not a JS heap OOM, so --max-old-space-size does not reach it.

The crash

#
# Fatal process out of memory: Zone
#
----- Native stack trace -----

 1: 00007FF6F95A495D node::MultiIsolatePlatform::DisposeIsolate+5341
 2: 00007FF6FAA12006 v8::base::FatalOOM+54
 3: 00007FF6FA29F5E5 v8::Function::NewInstance+245
 4: 00007FF6F9C19576 v8::SnapshotCreator::GetIsolate+56566
 5: 00007FF6FAE93F5A v8::platform::DelayedTaskQueue::Terminate+1698970
 6: 00007FF6FAE9364B v8::platform::DelayedTaskQueue::Terminate+1696651
 7: 00007FF6FAE92B0D v8::platform::DelayedTaskQueue::Terminate+1693773
 8: 00007FF6FAE925E4 v8::platform::DelayedTaskQueue::Terminate+1692452
 9: 00007FF6FAE91B71 v8::platform::DelayedTaskQueue::Terminate+1689777
10: 00007FF6FAE918CF v8::platform::DelayedTaskQueue::Terminate+1689103
11: 00007FF6FAC9F686 v8::base::DiyFp::Multiply+414934
12: 00007FF6FAC9E3A0 v8::base::DiyFp::Multiply+410096
13: 00007FF6FAA722EB v8::internal::compiler::CompilationDependencies::FieldTypeDependencyOffTheRecord+192907
14: 00007FF6F9912F56 DH_get0_engine+658470
15: 00007FF6F990E693 DH_get0_engine+639843
16: 00007FF6F990DF79 DH_get0_engine+638025
17: 00007FF6FAA8519F v8::platform::DefaultJobHandle::UpdatePriority+591
18: 00007FF6F95A081E node::GetNodeReport+90430
19: 00007FF6FA2F46D6 uv_thread_detach+150
20: 00007FF6FB55A033 v8::base::UnsignedDivisionByConstant<unsigned __int64>+2907267
21: 00007FF9B23DCCB7 BaseThreadInitThunk+23
22: 00007FF9B2E0AD6C RtlUserThreadStart+44

Frames 13 and 17 (internal::compiler::CompilationDependencies, platform::DefaultJobHandle::UpdatePriority) put this on a background optimizing-compiler job thread, which is why the exhausted arena is a compiler Zone rather than the JS heap.

The same stderr also carries, from tinypool:

Error: Worker exited unexpectedly
 ❯ ChildProcess.onUnexpectedExit node_modules/tinypool/dist/index.js:118:30
 ❯ ChildProcess.emit node:events:521:24
 ❯ ChildProcess._handle.onexit node:internal/child_process:295:12

Reproduction

Two full-suite runs on the same host, hours apart, on different commits, produced identical stacks. Both were vitest run --maxWorkers=8 --minWorkers=1 under Node, driven by automation that preserves each run's stderr.

The second occurrence is the more informative one. That run reported 4163 passed / 59 failed — a report with no structural sign of a crash — while a worker had been killed by this OOM. Of its 59 failures, 58 were timeouts and one was a known wall-clock assertion; none corresponded to the crash, because none could.

Why this is worth fixing separately from the timeout noise

I originally filed #1773 attributing this host's unstable failure counts to machine load. That attribution was not controlled and I have corrected it there. Relevant to this issue: the compiler-zone OOM recurred after antivirus exclusions were partially applied, so unlike the timeout class it is not plausibly a scanning artifact. It is a distinct failure mode that survived the intervention which removed the timeouts.

The reporting gap is the part I would most like addressed:

• The JSON report cannot be used to establish that a run was healthy. I classified the second run from vitest.json alone and concluded it was clean apart from timeouts; the fatal crash was in stderr the whole time.
• Combined with the inverse case in Windows: the suite's failure count tracks machine load, not the tree (216 of 227 failures are 5s timeouts on a green commit) #1773 — a fully green report (success: true, 0 failed, 4396 passed) that still exits 1 — the exit code and the report disagree in both directions. Automation cannot currently trust either one alone.

Environment

• Windows 11 Pro (10.0.26200), 32 logical cores, 128 GB RAM
• Node v26.8.1 (not Bun)
• vitest run --maxWorkers=8 --minWorkers=1, JSON + default reporters

Possible directions

I have not diagnosed the allocation that exhausts the zone, so this is a report rather than a fix, but a few things would each help independently:

1. Surfacing worker deaths in the JSON report, or at minimum a non-zero structured signal distinguishable from ordinary test failure, so a crashed run cannot read as healthy.
2. Whether a smaller maxWorkers or poolOptions.forks.maxForks on Windows changes reproduction — worth knowing, though note that serializing the suite did not change the timeout class on this host, so I would not assume it helps here either.
3. Whether any test compiles or evaluates unusually large generated sources; a compiler-zone OOM points at optimizing compilation, not at retained heap.

Happy to attach the complete stderr from either run, and the JSON report from the run that looked clean.

[bompus]

Follow-up: the reporting half of this is largely fixed upstream already — this repo is two majors behind on vitest

I asked in the issue for worker deaths to be surfaced in the report. Before anyone spends time on that here, it is worth knowing that vitest has already done work in exactly this area, and that this repo cannot see any of it.

package.json pins "vitest": "^2.1.9" and resolves to 2.1.9 (tinypool 1.1.1). Current vitest is 4.x.

Relevant upstream work, merged after 2.1.9:

• vitest-dev/vitest#10587 — "fix(pool): improve error message when worker exits unexpectedly", merged 2026-06-22 (76139c5). It exposes the child process's exit code and signal on an unexpected worker exit, and raises which test file(s) the worker was executing when it died. The motivating case in that PR is close to ours: a native extension causing SIGSEGV, diagnosed only after the code and signal became visible.
• The e2e fixture added with it (test/e2e/test/pool-worker-exit.test.ts) asserts that tests belonging to a crashed worker come back with state pending rather than being silently absent, and that the run still finalizes coverage.

That last point matters most for this issue. The specific complaint I raised — a killed worker leaving no trace in the JSON, so a crashed run reads as healthy — is the thing that fixture pins down, and it postdates 2.1.9 by a long way.

There is also a long-running discussion, #6285, where Error: Worker exited unexpectedly with no file attribution is reported across CI environments, which is the same symptom seen here.

What I think this means for this issue

1. The actionable item is a vitest upgrade, not a reporting change in this repo, and not a new bug report against vitest. On 2.1.9 we are asking for behaviour that already exists further up.
2. The zone OOM itself is not vitest's bug. Fatal process out of memory: Zone on a background optimizing-compiler thread is V8/Node; vitest is only the process that notices the worker died. Current vitest would tell us which file it died in and with what signal — which is the diagnostic we are missing and the reason the upgrade is worth doing before anything else.
3. I would not open a vitest issue at this point. It should be reproduced on a current vitest first; filing against 2.1.9 would rightly be closed as "please upgrade", and I do not yet have a reproduction that survives the upgrade.

I have not attempted the upgrade — 2.1.9 → 4.x is two majors and out of scope for a bug report — so I am not claiming it fixes the crash. It would not: the OOM would still happen. What it should change is that the crash becomes visible, which is the half of this issue that is actually about tooling rather than about V8.

Retaining the rest of the report as filed: the OOM reproduces, it survives partial antivirus exclusions, and on 2.1.9 it is invisible in the JSON report.

Two negative results, for completeness

Two full-suite runs on the same host on an idle box with the partial exclusions in place, on the unmerged canonical head, produced no OOM, no worker exit, and no timeouts — 4439 passed in 68.8s, and 4438 passed with one unrelated flaky daemon-launcher timeout in 60.8s. Contrast with an earlier loaded run of the same suite on the same box and same worker count: ~10 minutes, 52 timeouts, and the dead worker.

Two runs with load not varied deliberately is not a load correlation, and I am not presenting it as one. It does make it hard to argue the crash is intrinsic to running these 4,483 tests here. Credit for both runs and for spotting that I had originally missed the crash in the JSON goes to a colleague on this host.

[bompus]

Practical consequence while the pin is at 2.1.9: a green vitest.json does not mean the run was healthy

Stating this separately because it is the operational advice that falls out of the report above, and it is easy to read the issue as being only about a crash.

On vitest 2.1.9, a worker killed mid-run leaves no trace in the JSON report. It is not a test failure, so it produces no testResults entry; vitest recycles the worker and the run continues. The report that comes out the other side can be internally consistent — coherent pass and fail counts, no structural anomaly — while a process died during it.

That makes the report unusable as evidence of a healthy run in this suite. Concretely, over one morning on one machine it produced two wrong conclusions of my own:

1. I classified a run from its vitest.json and reported "58 of 59 failures are timeouts, no product failure". The counts were correct. The run had also killed a worker with a Fatal process out of memory: Zone, sitting in a stderr file I had not opened. Nothing in the JSON indicated it.
2. Separately, a run reporting 4396 passed / 0 failed / success: true exited 1. Automation trusting the exit code over the report treats a clean suite as failed; automation trusting the report over the exit code treats a crashed suite as clean. Both directions are wrong, and neither is detectable from the JSON alone.

So until the vitest pin moves, anything consuming this suite's results programmatically needs to read stderr as well as the report — at minimum grepping for Fatal process out of memory and Worker exited unexpectedly — and should not treat success: true as sufficient.

The upgrade closes this rather than mitigating it. vitest-dev/vitest#10587 surfaces the dead worker's exit code, signal, and the test file it was executing, and its e2e fixture asserts that a crashed worker's tests come back as pending instead of silently absent. That is the difference between "the report cannot be trusted" and "the report tells you what happened".

Worth noting for anyone attempting that upgrade: it is not a drop-in. On a trial against 4.1.11, --minWorkers no longer exists (CACError: Unknown option), and vitest.workspace.mts's defineWorkspace is silently ignored — the ui project then runs under the engine config and fails to parse its Svelte imports rather than being skipped. Both are fixable, but the second fails in a way that looks like a broken test rather than a configuration problem.

[bompus]

Root cause: this is #81 (Node >= 25 turboshaft WASM Zone OOM), and the project's own --liftoff-only mitigation never reaches a vitest worker

I should have found this before filing. The crash is already known here, the fix already exists in this repo, and the reason the test suite still hits it is a gap in how that fix is delivered. Also correcting a bad detection recipe I posted above.

The crash is #81

This host runs Node v26.8.1. src/bin/codegraph.ts hard-blocks Node >= 25 with a banner that names the exact failure:

Node.js 25.x has a V8 WASM JIT (turboshaft) Zone allocator bug that crashes with Fatal process out of memory: Zone when CodeGraph compiles tree-sitter grammars. CodeGraph WILL crash on this Node version mid-indexing. See #81

That is the crash in this issue. #81's stack shows the turboshaft frames explicitly (compiler::turboshaft::WasmLoweringReducer, SnapshotTable::MergePredecessors); the Windows stacks I posted above symbolize to the nearest exported symbol, which is why they surfaced as CompilationDependencies / DefaultJobHandle::UpdatePriority — background optimizing-compiler frames either way.

Why the suite still crashes when the CLI doesn't

src/extraction/wasm-runtime-flags.ts documents the fix — --liftoff-only, which forces grammars onto the Liftoff baseline and never runs turboshaft — and is explicit that it must be on Node's command line, because V8 reads it at engine init. It also records that NODE_OPTIONS, v8.setFlagsFromString, and worker execArgv do not work.

It is delivered on exactly three paths:

1. the bundled launcher (scripts/build-bundle.sh, scripts/npm-shim.js),
2. the CLI's self re-exec (relaunchWithWasmRuntimeFlagsIfNeeded, src/bin/codegraph.ts:120),
3. src/mcp/refresh-launcher.ts:66.

A vitest pool worker is none of the three. The suite imports the extraction code directly and compiles grammars in-process, so it runs on the unpatched optimizing tier. And vitest.config.mts sets CODEGRAPH_ALLOW_UNSAFE_NODE: '1', which switches off the hard block that would otherwise stop the run — the override the banner describes as "NOT recommended - you will likely OOM".

Measured, printing process.execArgv from inside a worker:

setup	worker execArgv
plain vitest run	["--conditions","development","--conditions","node"]
node --liftoff-only node_modules/vitest/vitest.mjs run	identical — does not propagate
poolOptions: { forks: { execArgv: ['--liftoff-only'] } }	["--liftoff-only","--conditions","development","--conditions","node"], exit 0

So the one-line config change is the candidate fix:

// vitest.config.mts
test: {
  poolOptions: { forks: { execArgv: ['--liftoff-only'] } },
}

I have verified that this delivers the flag and that Node accepts it. I have not yet run enough post-change full suites to claim it eliminates the crash — the OOM is intermittent, so absence over a couple of runs would not be evidence. Flagging it as the mechanically-indicated fix, not a proven one.

Worth noting the CI implication: on Node 22/23 the guard never fires and this whole failure mode is invisible, so this is a contributor-machine problem that CI cannot see.

Correcting the detection advice I posted above

In my earlier comment I suggested grepping run output for Fatal process out of memory. That false-positives, and I hit it myself: a later full run matched that string 6 times with zero actual crashes, because the Node-25+ banner quotes the string on every CLI spawn — and the banner is printed even when the override lets the run continue.

Match Native stack trace (0 in that run) or Worker exited unexpectedly instead. The rest of that comment stands: on 2.1.9 a killed worker still leaves no testResults entry, so a green vitest.json cannot by itself establish a healthy run.

What I think should happen to this issue

The reporting half — a dead worker being invisible in the JSON — is still real and is what the vitest upgrade addresses (vitest-dev/vitest#10587, as in the comment above). But the crash half is not a new bug: it is #81, on a Node version this project already refuses to run on, reached through a test-suite override that disables the refusal. If you agree, this issue is arguably a duplicate of #81 for the crash and should be narrowed to the poolOptions delivery gap plus the reporting blind spot.

Apologies for the noise in the original report — the banner naming #81 was printed on essentially every run I looked at, and I read past it.

[bompus]

Correcting my own root-cause comment: the crashing runs are on Node 24, not 25+ — plus a failed reproduction and a measured cost for --liftoff-only

Three things: a correction to the comment directly above, a reproduction attempt that failed, and a cost figure for the fix I proposed. The proposed fix does not change; its justification does.

Correction — this is not the #81 hard-block case

In my previous comment I attributed the crash to running on Node v26.8.1, i.e. inside the >= 25 hard-block range. That is where I ran the suite by hand, but it is not the runtime the crashing runs used.

Our harness invokes vitest with the Node that ships inside the installed codegraph bundle — %LOCALAPPDATA%\codegraph\current\node.exe — which is v24.16.0. Every zone OOM I have on record came from that harness, so every one of them happened on Node 24, a fully supported version where the >= 25 banner never fires and CODEGRAPH_ALLOW_UNSAFE_NODE is irrelevant.

That makes the story simpler, and worse. src/extraction/wasm-runtime-flags.ts already says so in its header:

On Node >= 22 the V8 "turboshaft" optimizing WASM compiler can exhaust its per-compilation Zone arena while compiling these grammars on a background thread [...] Reproduced on Node 22 and 24; Node 25 is already hard-blocked for the same crash.

So this is the Node 22/24 case that flag exists for, not the Node 25 case. My "this is #81" framing was wrong on the version. What stands, and is the actual defect:

• --liftoff-only is delivered by the bundled launcher, the CLI's self re-exec, and refresh-launcher.ts.
• A vitest pool worker is none of those, and gets no execArgv.
• Therefore the suite compiles tree-sitter grammars on the turboshaft tier on a supported Node, which is exactly the configuration the flag exists to prevent.

Practical consequence: a contributor on Node 22 or 24 hits this with no banner, no override, and nothing pointing at the cause. That is worse than the Node 25 path, where at least the CLI refuses and prints an explanation.

The fix is unchanged:

// vitest.config.mts
test: {
  poolOptions: { forks: { execArgv: ['--liftoff-only'] } },
}

Failed reproduction — I could not make it crash on demand

I tried to promote "mechanically indicated" to "measured" and could not. All arms interleaved, reversed halves:

workload	runtime	arms	runs	crashes
codegraph init, 446-file mixed-language corpus	Node 26.8.1	flag / no flag	8	0
full suite (4513 tests)	Node 26.8.1	flag / no flag	8	0
full suite (4513 tests)	Node 24.16.0	no flag	2	0

Zero Native stack trace and zero Worker exited unexpectedly in all 18 runs, including every no-flag arm. Since the control arm never crashed, none of this is evidence that the flag helps. I am reporting it so nobody repeats it expecting a different answer.

The one thing it is consistent with: every occurrence I have was on a loaded machine, and these runs were on a quiet one. That matches the load correlation in #1773 without establishing it.

The fix is not free

Timing from the full-suite arms above, Node 26.8.1, 4 runs per arm, interleaved:

arm	median	range
no flag	55.1s	52.6–56.7
--liftoff-only	68.6s	67.6–85.2

About 25% wall-clock on this suite. wasm-runtime-flags.ts describes the forgone optimized tier as "marginal, and for grammars rarely reached" — for a full suite run on this host it is not marginal, and if that sentence is load-bearing for anyone's decision it deserves re-measuring on your own hardware.

Caveats: this was measured on Node 26.8.1, not the Node 24 the harness uses, and on one host. Interleaved with reversed halves, so the cold-start cost is not the explanation; arm B's slowest run was its first, and excluding it the medians barely move.

So the trade is a ~25% suite slowdown against an intermittent crash I cannot reproduce on demand. I still think the flag is right — it makes the test process match every other launch path in the project, and the crash is real even if it is not summonable — but "cheap insurance" would be overselling it, and I do not want a number-free recommendation from me to be the reason someone takes it.

If you want the version question closed

The decisive experiment is on a contributor machine that is reproducing this: run the suite with and without poolOptions.forks.execArgv under whatever load makes it crash. I do not currently have a box in that state.

[danusha2345]

The decisive experiment you asked for, on a box that reproduces it: Linux x64, Node 24.15.0, __tests__/extraction.test.ts on current main (51116a2). It dies on demand, always at the same test (the 640th, 639/655 passed) — 3 of 4 no-flag runs. With poolOptions.forks.execArgv: ['--liftoff-only'] 2 of 2 runs passed, and faster (19–24s vs 34–41s: the baseline's extra seconds are the tier-up jobs that then die). Symbolized stack: Zone::Expand ← turboshaft::LoopUnrollingPhase ← Pipeline::GenerateWasmCode ← wasm::BackgroundCompileJob::Run, i.e. the turboshaft wasm tier-up from #293/#298. --no-wasm-loop-unrolling, --no-wasm-dynamic-tiering, --no-opt, --regexp-interpret-all and --no-sparkplug --no-maglev each still died at the same test, so only the full Liftoff pin works, as wasm-runtime-flags.ts predicts. Your one-line fix is up as #1883 with these numbers.

[colbymchenry]

Thanks for the detailed report, @bompus, and thanks for using CodeGraph! This is fixed on main by #2046, which builds on @danusha2345's #1883 (thank you!): vitest's pool workers now run with the same --liftoff-only V8 flag as every real launch path, so the grammar tier-up no longer kills a worker with a compiler-zone out-of-memory (the full 691-test extraction suite now passes on a Windows 11 VM). The other half of the report, vitest's JSON reporter still saying success: true after a worker dies, is upstream vitest behaviour; the process exit code is non-zero, so CI still fails correctly. The fix will be in the next release.