fix(extraction): a file over the size limit is never read (#1910)

[danusha2345]

What

The second defect from #1910 (the first, MPEG-TS detection, is #1914 — this PR is stacked on it because both touch the batch reader).

MAX_FILE_SIZE was enforced after readFile, so a committed video or blob fixture over the limit was decoded into a JS string in full — the issue measured 3.4 GB of RSS for one 400 MB .ts, 3.5 GB for ten 60 MB fixtures in one I/O batch, and a hard Invalid string length failure past ~512 MB — only to be stored as size_exceeded and discarded. Tracing the indexer on a sparse 400 MB file showed it was then read again by every pass that scans files by name: the framework detectors' readFile and the resolver's cached reader. Three full passes, 153 603 read(2) calls, 1.26 GB through the page cache for one file.

Change

Stat first, everywhere a source file is read by path:

• batch reader and extractFile — an oversize file is stored as skipped with a size stamp (codegraph:oversize:<bytes>) standing in for its content; it is never opened for reading.
• change detection (sync path, git fast path, full-scan fallback) hashes the same stamp, so the stored hash and the detected hash agree: a same-size rewrite of a file nothing is indexed from is not a change, crossing the limit in either direction is (covered by the test: over → under → over).
• framework-detection readFile and the resolver's readFileCached return null above the limit instead of decoding. The resolver hunk is byte-identical to the one in fix: bound large-repo memory and traversal work #1583, so the two merge cleanly whichever lands first.
• The limit moves to src/file-limits.ts (MAX_SOURCE_FILE_SIZE_BYTES), same file and name as fix: bound large-repo memory and traversal work #1583, so the three sites share one number.

Upgrade note: an index written before this change stored the hash of an oversize file's real bytes; change detection now hashes the stamp, so each such file shows as modified once after upgrading and the next sync stores the stamp — a one-time pass that no longer reads the file.

Files under the limit are read exactly as before; .ts files still get the ≤752-byte MPEG-TS sniff from #1914.

Verification

• strace -f on codegraph init over ok.ts + a sparse 400 MB huge.ts: reads of the big file 153 603 → 0 (one openat remains: the MPEG-TS head sniff).
• New __tests__/oversize-file-not-read.test.ts: neighbours indexed, the oversize record carries the stamp hash and no nodes, getChangedFiles() is empty right after indexing, same-size rewrite is not a change, shrinking under the limit makes it ordinary source (symbols appear after sync), growing back over it drops them again; a 400 MB sparse fixture indexes in under 5 s with < 100 MB RSS growth after a warm-up init.
• tsc clean; extraction, sync, git-index-currency, sync-rebuild-convergence, resolution, index-command, frameworks-integration, mpeg-ts-not-typescript + the new file: 9 files / 978 tests pass.

Fixes the remaining half of #1910.

🤖 Generated with Claude Code

[danusha2345]

Rebased onto the updated #1914 (10def05) and tidied (0593f9e): readSourceOrStamp lost an argument no caller passed, and the batch reader's comments are back in the order the code runs. Added an upgrade note to the description: an existing index shows each oversize file as modified once, then stores the stamp. Suites green on both arms (980 tests each).

[danusha2345]

Ported the bounded-read hardening from #1919 (3b0878e): every source read goes through readBoundedSource/readBoundedSourceSync, which re-checks the open descriptor and stops one byte past the limit, so a file that grows after the stat is still stamped as oversize. I also bounded the two parse-retry re-reads, which #1919 still does with fsp.readFile (3bcf88e on this branch). The sparse 400 MB fixture test stays.

[inth3shadows]

I tested this on Linux with Node 22, merged onto current main. Bounding the reads and the size stamp both look right. I found four problems:

1. Breaks an existing test. git-index-currency.test.ts › keeps a committed path pending when sync cannot read it fails every time on this branch and passes on main (expected 0 to be greater than 0). The test injects its read error on fs.readFileSync, but sync now reads through readBoundedSourceSync (openSync/readSync), so the error never fires. The pending-path behavior itself still holds: moving the injection to fs.openSync makes the test pass.
2. The viewer reports false drift for files between 1 and 8 MB. The stored contentHash of an oversize file is now the hash of the size stamp. readFileShape and hasDriftedOnDisk in src/ui-server/api/source.ts still hash the real bytes. For an unchanged 1.4 MB big.js, readFileShape returns drift: true ("This file changed on disk after the last index sync…"); on main it returns drift: false.
3. The MPEG-TS check can drop a valid .ts file. A TypeScript file with G at bytes 0/188/376/564 (four 188-byte lines each starting with G) and one raw NUL inside a comment is treated as video. Its export function realFn() is not indexed; on main it is. It's contrived, but a stricter check (for example more packets, or checking the PID/continuity fields) would rule it out.
4. Changelog style. The two new entries include benchmark figures ("a 400 MB fixture cost 3.4 GB of memory", "about 28 seconds"). The changelog rules in AGENTS.md exclude benchmark numbers from entries.

[danusha2345]

Thanks for the careful review. All four points are addressed:

1. git-index-currency test (5f5cd77): the read failure is now injected at fs.openSync as well as readFileSync, so it reaches the bounded reader. 13/13 pass.
2. False drift for 1–8 MB files (5f5cd77): oversizeStamp moved to src/file-limits.ts, next to a small indexedHashInput(size, content) helper. All four places that compare a file on disk with its stored hash now hash what the index stored: readFileShape, hasDriftedOnDisk and the source endpoint in ui-server/api/source.ts, plus MCP's isFileStaleOnDisk. An oversize file is compared by its stamp without being read. The new test touches an unchanged 1.4 MB big.js: readFileShape, hasDriftedOnDisk and MCP all report no drift, and appending one line makes it drift. The test fails without the change.
3. MPEG-TS false positive (03eecf6 on fix(extraction): an MPEG-TS video named .ts is not TypeScript (#1910) #1914, merged here): the sniff now needs the sync byte on 16 consecutive packets (3 KB) and at least 1/64 of the head to be control bytes, instead of 4 packets plus any NUL. Your counterexample (G at every stride plus a NUL in a comment) now stays TypeScript at 4, 16 and 20 strides, and its realFn is indexed. ffmpeg-made H.264/AAC, MPEG-2 and MP2 streams have 8–20% control bytes and are still recognised, as is a head that opens with 0xFF-stuffed PAT/PMT packets. A .ts clip shorter than 3 KB is parsed, which is cheap.
4. Changelog: both entries lost their figures.

Full suite on this branch (Linux, Node 22, native kernel): 4737 passed, 11 skipped.

[colbymchenry]

Thanks @danusha2345 — this landed in #2082 with all your commits and authorship kept, rebased onto current main. I added one commit on top: discovery was reading the first 3 KB of every .ts file on every full scan (a 10k-file tree went from ~105 ms to ~243 ms, and on a slow disk it's a random read per file), so the video check now runs on the bytes each file is already being read for. Scan time is back to main's and a full index is unchanged. Fixes #1910.