Skip to content

fix(mcp): make the liveness watchdog a separate process, not a worker thread (#850) - #858

Merged
colbymchenry merged 1 commit into
mainfrom
fix/850-watchdog-use-process
Jun 13, 2026
Merged

colbymchenry merged 1 commit into
mainfrom
fix/850-watchdog-use-process

Conversation

@colbymchenry

Copy link
Copy Markdown
Owner

Fixes the watchdog from #856 — which did not actually work in the real daemon. I caught this by live-testing it (not just unit tests) against a real codegraph serve --mcp.

What was wrong

The worker-thread watchdog stalled exactly when it was needed. V8 isolates in one process coordinate on global safepoints: when any thread requests a GC, every other thread must reach a safepoint first. A main thread wedged in a tight, non-allocating loop never reaches one — so the watchdog worker deadlocks on its next safepoint check before it can SIGKILL. And #850's hot loop (SourcePositionTableIterator::Advance, a non-allocating C++ table walk) is precisely that shape.

The unit/integration tests passed only because the toy test process had no heap/GC pressure. Live evidence (real daemon, CODEGRAPH_WATCHDOG_TIMEOUT_MS=2000, main thread wedged):

[CodeGraph watchdog] armed: timeoutMs=2000 checkMs=400 maxStaleChecks=5
[CodeGraph watchdog-worker] counter=1 ... counter=2 ... counter=3   ← worker goes silent at the wedge
{ "how": "HARD_TIMEOUT — still alive after 8s (watchdog did NOT fire)" }

The fix

Move the watchdog to a separate child process. It shares no isolate or heap with the parent, so the wedge can't touch it, and it kills via the kernel (SIGKILL is honoured regardless of parent thread state). The parent heartbeats a byte to the child's stdin each tick; if it stops for the timeout, the child SIGKILLs the parent; when the parent exits the pipe closes and the child exits (no orphan). Child cwd pinned to os.tmpdir() so it can't hold a dir open (Windows cleanup).

Validated live (the whole point)

Same real-daemon test, after the fix:

{ "how": "child exit: code=null signal=SIGKILL", "elapsedMs": 3390 }
[CodeGraph] Main thread unresponsive for ~2s — killing the wedged process so a fresh one can start (#850).
  • Layer 1 (uncaught exception) live: real daemon → bounded log → exit code=1. ✓
  • Layer 2 (wedge) live: real daemon → SIGKILL in ~timeout, death-rattle in the log, no orphaned watchdog. ✓
  • New regression test reproduces the worker-stalling case (a non-allocating wedge under ~40MB heap pressure) — it would fail on the old worker-thread design.

Notes

Public API, install points (src/mcp/index.ts), and the [Unreleased] CHANGELOG entry are unchanged — that entry describes the now-working behavior, and the broken worker version was never released (lived only on main). Full suite green (75 files, 1505 passed / 2 skipped). Windows path is portable by design but not yet run on the VM.

🤖 Generated with Claude Code

… thread (#850)

The worker-thread watchdog from #856 does not actually work in the real daemon —
caught by live-testing it against a real `serve --mcp`. V8 isolates in one
process coordinate on global safepoints, so a main thread wedged in a tight,
non-allocating loop (exactly #850's SourcePositionTableIterator::Advance) strands
the watchdog worker on its next safepoint check before it can SIGKILL. The worker
only "passed" the unit test because that toy process had no GC pressure; under a
real heap it stalls.

Move the watchdog to a separate child process: it shares no isolate or heap with
the parent, so the wedge can't touch it, and it kills via the kernel (SIGKILL is
honoured regardless of parent thread state). The parent heartbeats a byte to the
child's stdin each tick; if it stops for the timeout the child SIGKILLs the
parent; when the parent exits the pipe closes and the child exits too (no
orphan). Child cwd is pinned to os.tmpdir() so it can't hold a dir open (Windows).

Validated live against a real daemon: a wedged main thread is now SIGKILLed in
~timeout (it survived forever before). New regression test reproduces the exact
worker-stalling case — a non-allocating wedge under heap pressure.

Public API, install points (src/mcp/index.ts), and the CHANGELOG entry are
unchanged — that entry describes the now-working behavior, and the broken
worker-thread version was never released (lived only on main).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@colbymchenry
colbymchenry merged commit 1702dfc into main Jun 13, 2026
1 check passed
@colbymchenry
colbymchenry deleted the fix/850-watchdog-use-process branch June 13, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant