Repository navigation
fix(resolution): a large run waits for its pool, so the main thread doesn't scan the project - #3192
Merged
Merged
Conversation
…oesn't scan the project The first batch of a run resolved on the main thread while the workers booted. That batch is the prerequisites, and in C++ the first qualified base class reads every source file for the project's namespace aliases: one synchronous stretch on the thread the liveness watchdog watches, 287 ms on rocksdb and 20 s or more on a project the size of LLVM (#1091). 1.6.2 had no such scan on that path; it never shipped. A run large enough to start the pool up front, 150k references or more, now waits for it before the first batch. The wait is bounded at 30 s, and a failure ends it, so a pool that doesn't come up leaves the run on its own thread, as it would be with no pool. That needed one more fix: a worker that failed to open the index never failed ready(), which nobody had waited on before. Any failure does now, and so does a worker that exits before it's ready. A run under the bar starts nothing up front and doesn't wait for a pool engaged mid-run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bompus
added a commit
to bompus/codegraph
that referenced
this pull request
Nov 2, 2026
…oesn't scan the project (colbymchenry#3192) Ported from upstream 4e96d75. The fork already opened the pool before its first batch; this bounds that wait at POOL_START_WAIT_MS so a pool that never comes up falls back to the main thread, and makes ready() reject on any worker failure and on a worker that exits before it is ready — a worker that could not open the index used to leave the wait stuck forever. A timed-out start destroys the pool, because its snapshot may be stale by the time it boots.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On a large project the main thread could read every C and C++ source file in one synchronous stretch, in the first batch, while the pool booted. A run large enough to start the pool up front now waits for it, so the main thread resolves nothing. The wait is bounded, and a pool that fails leaves the run on its own thread.
Root cause
A run starts the pool and doesn't wait: the first batch resolves on the main thread until the workers are up. That batch is the prerequisites, and a qualified C++ base class reaches
matchCppSupertype>cppClassWritten>cppNamespaceAliases, which reads every C and C++ file, with no yield, on the thread the 60 s liveness watchdog watches. Main's profile at 4499d00 shows it, 287 ms on rocksdb; 1.6.2's doesn't, so this never shipped. At rocksdb's rate a project the size of LLVM is 20 s or more (#1091, #1122). #3182's bound doesn't cover this path. A C-only project like Linux doesn't reach it: it has no qualified bases.The fix
resolveAndPersistBatched: after starting the pool on the ref-count path (150k references or more),await poolStartSettled(pool.ready()). The event loop keeps turning, so the watchdog is fine.POOL_START_WAIT_MS, 30 s: a full index's workers are up in a second or two, and a sync's detect frameworks for themselves, which can read every source file. Past it the run goes on on its own thread, exactly as it would have, and takes the pool up if it arrives.ready()could hang. A worker that failed to open the index reported an error that failed the pool but never itsready(), and a worker that exited with code 0 before it was ready did nothing. Nobody waited on it before. Both now reject.A run under the bar starts nothing up front, as before. A pool engaged mid-run isn't waited for: the loop keeps running while it boots, as before.
Validation
Windows 11. In
resolver-pool-pipeline.test.ts, with each guard's mutation failing the test named:queues a batch ahead…counts every reference through the poolresolves on its own thread when the pool fails to startstops waiting for a pool that never comes up…: at 29,999 ms nothing has resolved, at 30,000 it runs and finishesfails the start of a worker that cannot open the index…, real pool, missing databaseready()starts nothing up front under the bar, and never waits for a pool engaged mid-runInterleaved, 3 rounds, this against main:
Level, as it should be: this is about the stall, not the time.
Graphs, this against main: byte-identical on rocksdb, react, redis and 12 smaller repos (gin, flask, taxonomy, serilog, ripgrep, caddy, jsoup, express, excalidraw, tokio, fmt, leveldb). django differs by one edge from some runs of main, and that isn't this PR: main alone gives both results run to run (
e0f776f03 times,cc081a2dtwice in 5 runs of the same build), and this branch gavecc081a2dall 5 times.isPythonLocallyBoundmemoizes per function and name an answer that depends on the reference's line, so which chunk asks first decides it. 1.6.2 has the same memo.Full suite on this commit, the first with #3182's bound and #3183 together: 875 files passed, 6 skipped, 1 failed; 10,861 passed, 2 expected fail, 87 skipped, 1 failed, 4 workers, Vitest 4.1.11. The failure is
java-value-reads.test.ts: my kernel predatedjava.rsin #3189, which doesn't touch this. With the kernel rebuilt, all 47 Java, Kotlin, Scala and kernel test files pass (483 tests).🤖 Generated with Claude Code