Skip to content

fix(resolution): a large run waits for its pool, so the main thread doesn't scan the project - #3192

Merged
colbymchenry merged 1 commit into
mainfrom
claude/spd2-first-batch
Nov 1, 2026
Merged

colbymchenry merged 1 commit into
mainfrom
claude/spd2-first-batch

Conversation

@colbymchenry

Copy link
Copy Markdown
Owner

Summary

On a large project the main thread could read every C and C++ source file in one synchronous stretch, in the first batch, while the pool booted. A run large enough to start the pool up front now waits for it, so the main thread resolves nothing. The wait is bounded, and a pool that fails leaves the run on its own thread.

Root cause

A run starts the pool and doesn't wait: the first batch resolves on the main thread until the workers are up. That batch is the prerequisites, and a qualified C++ base class reaches matchCppSupertype > cppClassWritten > cppNamespaceAliases, which reads every C and C++ file, with no yield, on the thread the 60 s liveness watchdog watches. Main's profile at 4499d00 shows it, 287 ms on rocksdb; 1.6.2's doesn't, so this never shipped. At rocksdb's rate a project the size of LLVM is 20 s or more (#1091, #1122). #3182's bound doesn't cover this path. A C-only project like Linux doesn't reach it: it has no qualified bases.

The fix

  • resolveAndPersistBatched: after starting the pool on the ref-count path (150k references or more), await poolStartSettled(pool.ready()). The event loop keeps turning, so the watchdog is fine.
  • POOL_START_WAIT_MS, 30 s: a full index's workers are up in a second or two, and a sync's detect frameworks for themselves, which can read every source file. Past it the run goes on on its own thread, exactly as it would have, and takes the pool up if it arrives.
  • ready() could hang. A worker that failed to open the index reported an error that failed the pool but never its ready(), and a worker that exited with code 0 before it was ready did nothing. Nobody waited on it before. Both now reject.

A run under the bar starts nothing up front, as before. A pool engaged mid-run isn't waited for: the loop keeps running while it boots, as before.

Validation

Windows 11. In resolver-pool-pipeline.test.ts, with each guard's mutation failing the test named:

Proof Test Mutation that fails it
The first batch goes to the pool queues a batch ahead… counts every reference through the pool not waiting
A pool that fails to start falls back resolves on its own thread when the pool fails to start letting the rejection through
A pool that never comes up can't hang the run stops waiting for a pool that never comes up…: at 29,999 ms nothing has resolved, at 30,000 it runs and finishes unbounded wait
A real worker that can't open the index fails the start fails the start of a worker that cannot open the index…, real pool, missing database failures not reaching ready()
Under the bar nothing changes starts nothing up front under the bar, and never waits for a pool engaged mid-run waiting on the adaptive path

Interleaved, 3 rounds, this against main:

Repo Wall s CPU s Peak RSS MB
rocksdb 35.00 to 34.72 118.7 to 118.3 2440 to 2356
react 42.19 to 39.67 122.8 to 121.5 1751 to 1755

Level, as it should be: this is about the stall, not the time.

Graphs, this against main: byte-identical on rocksdb, react, redis and 12 smaller repos (gin, flask, taxonomy, serilog, ripgrep, caddy, jsoup, express, excalidraw, tokio, fmt, leveldb). django differs by one edge from some runs of main, and that isn't this PR: main alone gives both results run to run (e0f776f0 3 times, cc081a2d twice in 5 runs of the same build), and this branch gave cc081a2d all 5 times. isPythonLocallyBound memoizes per function and name an answer that depends on the reference's line, so which chunk asks first decides it. 1.6.2 has the same memo.

Full suite on this commit, the first with #3182's bound and #3183 together: 875 files passed, 6 skipped, 1 failed; 10,861 passed, 2 expected fail, 87 skipped, 1 failed, 4 workers, Vitest 4.1.11. The failure is java-value-reads.test.ts: my kernel predated java.rs in #3189, which doesn't touch this. With the kernel rebuilt, all 47 Java, Kotlin, Scala and kernel test files pass (483 tests).

🤖 Generated with Claude Code

…oesn't scan the project

The first batch of a run resolved on the main thread while the workers booted. That
batch is the prerequisites, and in C++ the first qualified base class reads every
source file for the project's namespace aliases: one synchronous stretch on the
thread the liveness watchdog watches, 287 ms on rocksdb and 20 s or more on a project
the size of LLVM (#1091). 1.6.2 had no such scan on that path; it never shipped.

A run large enough to start the pool up front, 150k references or more, now waits
for it before the first batch. The wait is bounded at 30 s, and a failure ends it, so
a pool that doesn't come up leaves the run on its own thread, as it would be with no
pool. That needed one more fix: a worker that failed to open the index never failed
ready(), which nobody had waited on before. Any failure does now, and so does a worker
that exits before it's ready. A run under the bar starts nothing up front and doesn't
wait for a pool engaged mid-run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry
colbymchenry merged commit 4e96d75 into main Nov 1, 2026
@colbymchenry
colbymchenry deleted the claude/spd2-first-batch branch November 1, 2026 10:34
bompus added a commit to bompus/codegraph that referenced this pull request Nov 2, 2026
…oesn't scan the project (colbymchenry#3192)

Ported from upstream 4e96d75.

The fork already opened the pool before its first batch; this bounds that
wait at POOL_START_WAIT_MS so a pool that never comes up falls back to the
main thread, and makes ready() reject on any worker failure and on a
worker that exits before it is ready — a worker that could not open the
index used to leave the wait stuck forever. A timed-out start destroys
the pool, because its snapshot may be stale by the time it boots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant