[Fieldnote-Echo]

Fixes #2416

What does this PR do?

On the sequential route (50 files or fewer, or CBM_WORKERS=1; src/pipeline/pipeline.c:2684), cbm_pipeline_pass_definitions caches each raw result with its tree-sitter tree and uncompacted working arena until the run ends (pass_definitions.c:870-871). This change frees the tree and compacts the result before caching, as the parallel worker already does (pass_parallel.c:1233, :1290), and releases the thread's working arena when the pass ends (+7 lines of C). The graph is byte-identical on 7 sequential-route corpora (Python, Go, C/Rust). The cost is that the cross-file LSP pass re-parses these files, as it always has on the parallel route (pass_lsp_cross.c:1282).

Linux x86_64, 32 logical CPUs, GCC 16	main	this PR
Peak memory, 40 large .py files, 3 workers (sequential because ≤ 50 files)	962 MB	246 MB
Peak memory, 480 .py files, CBM_WORKERS=1	865 MB	137 MB
Cross-file LSP pass, same 480 files	268-275 ms	528 ms

sequential_definitions_cache_drops_trees_and_compacts fails on main, where a cached result still holds its parse tree.

Checklist

• Every commit is signed off (git commit -s) — required, CI rejects
  unsigned commits (DCO, see CONTRIBUTING.md)
• Tests pass locally (make -f Makefile.cbm test)
  (run as make -f Makefile.cbm test-par)
• Lint passes (make -f Makefile.cbm lint-ci)
  (for this PR's changes: CI's lint job passes on this head, and local cppcheck 2.22 / clang-format 23 add no findings over main)
• New behavior is covered by a test (reproduce-first for bug fixes)

Prepared with AI assistance (exploration, reproduction, verification, implementation); the spec and design decisions are mine, and I reviewed and ran everything above.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[Fieldnote-Echo]

Thx bot -- branch updated. Will try to keep this (and the other 4) I opened today up to date with main. Take your time codeowners.

[DeusData]

Thanks for the sequential-result compaction fix and for recording the reparse cost alongside the memory savings. I have read the description and follow-up; this remains pending review. No routine rebase is requested just for this acknowledgement.

[DeusData]

Thank you, @Fieldnote-Echo -- this is a valuable fix with an honest trade-off write-up. Holding the sequential cache the way the parallel extract already does (no tree, compacted), releasing the thread's working arena when the pass ends, and recording the extra cross-file LSP parse time next to the memory savings made it easy to evaluate.

We pushed two commits to your branch:

1. Merged current main. The merge is clean.
2. Adapted the change to two things main gained after your branch was cut:
   • A new reader of the cached tree. The PHP vendor-import check (php_import_kind_at_site in pass_pkgmap.c) reads each call and usage site's syntax from the tree to tell a class site from a function or constant site. The sequential route keeps no source to fall back on, so without the tree a same-named vendor use function or use const import vetoed the edges of new Thing(), Thing $t and extends Thing to the project class. PHP results now keep their tree; every other language still drops it. ei_php_vendor_function_import_keeps_same_named_class_sites covers this, and your test now expects PHP results to keep their tree.
   • A new early exit. A configured test convention whose file cannot be extracted aborts the pass (configured_failure), and that path now releases the working arena as well. sequential_definitions_configured_failure_releases_work_arena covers it.

What we checked on the sequential route (CBM_WORKERS=1, or 50 files or fewer):

• Peak memory: redis 1,317 MB -> 365 MB, a 535-file Go tree 200 MB -> 121 MB, 40 large Python files 232 MB -> 153 MB, and a 1,412-file Laravel app 614 MB -> 212 MB (164 MB without the PHP trees).
• The graph, every node and edge with its properties, is identical to main's on all four and on the PHP fixture above.
• Your test fails on main (a cached result still holds its tree), and each new test fails without its change. The parallel, pipeline, edge-import, incremental and PHP LSP suites pass (1,121 tests), and cppcheck and clang-format are clean.

Thanks again for the careful measurements -- the memory saving on small and single-worker runs is substantial.

[DeusData]

Thank you, @Fieldnote-Echo, for the sequential extraction memory fix and regression coverage! I merged current main into this branch to pick up the shared language-count CI correction. The prior Linux, macOS, and Windows jobs stopped at that inherited contract check before reaching the C tests.

The PR-specific diff is unchanged, including the PHP parse-tree exception and arena cleanup. This refresh preserves your commits and starts a fresh set of required checks.

[DeusData]

Thank you again for the sequential memory improvement. The October updates record the PHP-tree exception, early-exit cleanup, and baseline refresh as completed on our side. No further contributor work is requested; sorry for the delay in the remaining follow-up, and I’ll follow up soon.

[DeusData]

Thank you, @Fieldnote-Echo, for this change and for your patience while it waited for its benchmark. Every change to indexing memory or speed gets a graph-quality bench here before it merges, and yours came out very well. We indexed six projects with a single worker (so the sequential route you changed is the one that runs; every run confirmed it), three interleaved runs per binary, current main against your change merged onto it:

project	nodes / edges	graph (nodes, edges, CALLS set, 30-field census)	peak RSS main -> PR	wall time
mojo	2,375 / 6,144	identical	0.10 -> 0.08 GB	same
koel (PHP)	16,630 / 58,553	identical	0.61 -> 0.21 GB	-3.6%
meilisearch	18,368 / 98,373	identical	1.17 -> 0.20 GB	-4.3%
redis	41,115 / 145,572	identical	1.31 -> 0.38 GB	-6.2%
Exposed	33,103 / 270,608	identical	2.70 -> 0.29 GB	-9.0%
django	67,169 / 319,412	identical	3.50 -> 0.55 GB	-7.2%

The graph is byte-identical on every project and deterministic across runs, and peak memory drops by 20 to 89 percent at no time cost. Even koel, where your change deliberately keeps the PHP syntax trees, drops by two thirds because its TypeScript and JavaScript files are released.

Main has moved since your last run (it also touched src/pipeline/pass_definitions.c), so we merged current main into your branch (one merge commit, e1405ea; your change is untouched) for a fresh CI run. We will merge once it is green. Please pull before pushing again.

[DeusData]

Merged as ba4122d - thank you, @Fieldnote-Echo! The sequential definitions pass now drops each file's parse tree and working arena as soon as its definitions are out, which took peak memory down by 20 to 89 percent on the six projects we benchmarked, with a byte-identical graph. Thank you for the careful change and for your patience while it waited for its bench.