Repository navigation
feat(mcp): worker-thread liveness watchdog to self-kill a wedged main thread (#850) - #856
Merged
Merged
Conversation
… thread (#850) Belt-and-suspenders beyond the #850 handler fix. Any non-yielding sync loop on the main thread — a future V8 stack-format pathology, a runaway regex, an accidental while(true) — wedges the event loop, and nothing running ON that loop (timers, signal handlers, the PPID watchdog) can fire to recover it. Only another thread can. Install a tiny worker thread in the two long-lived in-process modes (the detached daemon, which has no PPID watchdog or stdin lifeline, and direct mode) that watches a shared-memory heartbeat the main thread bumps each event-loop turn. If the heartbeat stops advancing across enough consecutive worker checks (~CODEGRAPH_WATCHDOG_TIMEOUT_MS, default 60s), the worker SIGKILLs the process — the one signal a wedged loop can't swallow — so a fresh daemon starts on the next connection. Counting consecutive stale checks (not a wall-clock delta) makes it immune to clock jumps / laptop sleep: a healthy main thread resumes and bumps the heartbeat within one interval of waking, resetting the count; only a thread that never resumes keeps it climbing. Tuned never to fire on real work (heavy parsing runs off-thread, indexing shells out to a child), unref'd so it never delays shutdown, worker errors swallowed so they can't escalate to the now-exiting global handler, and opt-out via CODEGRAPH_NO_WATCHDOG=1. Pure reducer (stepHeartbeat) + config parsing unit-tested; the real worker is spawn-tested end-to-end (wedge -> SIGKILL, healthy -> survives, opt-out honored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 13, 2026
colbymchenry
added a commit
that referenced
this pull request
Jun 13, 2026
… thread (#858) The worker-thread watchdog from #856 didn't work in the real daemon — caught by live-testing against a real serve --mcp. V8 isolates coordinate on global safepoints, so a main thread wedged in a tight non-allocating loop (#850's SourcePositionTableIterator::Advance) strands the watchdog worker before it can SIGKILL. A separate child process shares no isolate/heap with the parent, so the wedge can't touch it; it kills via the kernel. Parent heartbeats to the child's stdin; silence past the timeout -> SIGKILL; parent exit closes the pipe -> child exits. Validated live (real daemon SIGKILLed in ~timeout); regression test covers the non-allocating-wedge-under-heap-pressure case. API/install points/CHANGELOG unchanged; the broken worker version was never released.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #855 (the #850 handler fix). That PR removed the known trigger of the 100% CPU wedge; this adds the belt-and-suspenders defense I flagged: a watchdog that can recover from any future main-thread wedge.
Why
Any synchronous, non-yielding loop on the main thread — a future V8 stack-format pathology, a runaway regex, an accidental
while (true)— blocks the event loop. From JS you can't interrupt it: timers, signal handlers, and the PPID watchdog all run on that blocked loop, so the process pins a core forever with no self-recovery. The only observer still running is another thread.How
src/mcp/liveness-watchdog.tsspawns a tiny worker thread (in the two long-lived in-process modes — the detached daemon, which notably has no PPID watchdog or stdin lifeline, and direct mode). The main thread bumps aSharedArrayBufferheartbeat each event-loop turn; the worker watches it. If it stops advancing across enough consecutive checks (~CODEGRAPH_WATCHDOG_TIMEOUT_MS, default 60s), the workerSIGKILLs the process — the one signal a wedged loop can't swallow — so a fresh daemon starts on the next connection.Safety (false positives were the main design risk)
stepHeartbeat, unit-tested.)unref'd (never delays shutdown — verified against the idle-timeout suite), worker errors swallowed (can't escalate to the now-exiting global handler), inlineevalworker (no dist-path fragility).CODEGRAPH_NO_WATCHDOG=1; tune withCODEGRAPH_WATCHDOG_TIMEOUT_MS.Tests
__tests__/liveness-watchdog.test.ts(10): pure reducer + config parsing, plus three spawn-based end-to-end cases against the built module — wedge → SIGKILL (646 ms), healthy → survives & exits on its own, opt-out honored. Full suite green (75 files, 1508 passed / 2 skipped), incl. the daemon idle-timeout/lifecycle suites.🤖 Generated with Claude Code