fix(extraction): never read an oversize file; an MPEG-TS video named .ts is not TypeScript (#1910) - #2082
Merged
Conversation
`.ts` mapped to TypeScript by extension alone, so MPEG transport-stream fixtures (`testdata/*.ts`) were parsed by tree-sitter: ~28 s of CPU for a 900 KB clip, minutes for a directory of them, for zero symbols. Recognise the stream from the head of the file — the 0x47 sync byte at offsets 0, 188, 376 and 564 plus a NUL byte, which every stream carries in its first packets and UTF-8 source never does — and drop it at discovery: not indexed, not parsed, not counted, not tallied as an unsupported language. The check reads under 1 KB and only for `.ts` files; the batch reader sniffs the bytes it already read, and the single-file path (sync, watcher) does the same head read, so a clip handed in by name is skipped too. The skip happens before parse dispatch, so no kernel mirror is needed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…oped sync Review found two places that decide "is this a source file" without the MPEG-TS check the scan applies: the git fast path's candidate loop in getChangedFiles and the scoped-sync path filter (watcher events). An untracked clip in a git repo was reported as `added`, skipped by sync without a record, and reported as `added` again on every status; a tracked .ts that became a clip stayed `modified` with its stale nodes. Both now treat the clip as non-source: removed when tracked, ignored otherwise. Regression tests fail without this change and pass with it, on both arms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1910) Review on #1915: the sniff took four aligned 0x47 sync bytes plus any NUL as proof of a transport stream, so TypeScript with a `G` at four 188-byte strides and one raw NUL in a comment was dropped unindexed. The sniff now needs the sync byte on sixteen consecutive packets (3 KB of head) and at least 1/64 of the head to be control bytes. Compressed payload carries 8-20% of them (checked on ffmpeg-made H.264/AAC, MPEG-2 and MP2 streams); source text carries none. A shorter clip is cheap to parse anyway. Also drops the benchmark figures from the changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The size gate ran after `readFile`, so a committed video or blob fixture was decoded in full — 3.4 GB of RSS for a 400 MB file, `Invalid string length` past ~512 MB — only to be stored as skipped. It was then read again by every pass that scans files by name: the framework detectors' `readFile` and the resolver's cached reader, three full passes on the reporter's fixtures. Stat first, everywhere a source file is read by path: the batch reader and `extractFile` store an oversize file with a size stamp in place of its content, change detection hashes the same stamp (so a same-size rewrite of a file nothing is indexed from is not a change, while crossing the limit in either direction is), and both by-name readers return null above the limit. The limit moves to `src/file-limits.ts` so the three sites share one number. strace on a sparse 400 MB `.ts`: 153 603 reads before, 0 after; indexing it next to a real file costs no measurable RSS. Stacked on #1914, which shares the batch-reader hunk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ation order `readSourceOrStamp` took a `stats` argument no caller passed and returned stats no caller read; it now returns the text or stamp change detection hashes. The batch reader's two comments are back in the order its code runs: the size gate, then the MPEG-TS head check. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1910) A file can grow between the stat that gates the size limit and the read that follows (a log, a download, a build output being written). Every source read now goes through readBoundedSource/readBoundedSourceSync: the open descriptor is re-checked, and the read stops one byte past the limit, so an oversize file is still stamped rather than decoded. The extraction batch reader, single-file extraction, framework detectors and the resolver's file cache all use it. Ported from the maintainer's hardening of this fix in #1919. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The retry passes for files whose worker crashed or timed out re-read the source with an unbounded fsp.readFile. The file can have grown since the first, bounded read; an oversize file is now skipped by the retry instead of being decoded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…here (#1910) Review on #1915: - The viewer (readFileShape, hasDriftedOnDisk, the source endpoint) and MCP's drift gate hashed a file's real bytes, but the index stores the size stamp for a file over the limit, so every unchanged 1-8 MB file read as "changed on disk". oversizeStamp moves to file-limits.ts with an indexedHashInput helper, and all four checks hash what the index stored. An oversize file is now compared without being read. - git-index-currency's "keeps a committed path pending when sync cannot read it" injected its failure only into readFileSync, which the bounded reader no longer calls, so it failed. The failure now reaches openSync too. - The changelog entry drops its benchmark figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2043's answer-freshness check hashes each contributing file whole and compares it with the stored contentHash. For a file over the index's size limit the stored hash is the size stamp, so an unchanged oversize file would read as stale. Compare it by its stamp, without reading it, like the viewer and the MCP drift gate do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… at discovery (#1910) Discovery sniffed the head of every `.ts` file on every full scan — an open + 3 KB read + close per file, before anything asked for its content. On a 10k-file TypeScript tree that took a warm-cache scan from ~105 ms to ~243 ms; on a slow disk or a network share it is a random read per file per scan, the access pattern #1231 and the SMB reports (#1014, #448) are about. The bytes are now judged where they are read anyway, so the check costs no I/O and runs only for a file that is new or changed: the batch reader (as before), `indexFile`, the sync reconcile (an untracked clip is skipped, a tracked file that became one is removed through the same removal path as a deletion), and both `getChangedFiles` paths. The scan lists `.ts` files by name again; a test pins that discovery opens none of them on the walk or the git path. Scan time is back to main's (~98 ms), and a full index is unchanged within noise. Also collapses the four #1910 changelog bullets (#1914 and #1915 each added both) to two, without the benchmark figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 29, 2026
danusha2345
pushed a commit
to danusha2345/codegraph
that referenced
this pull request
Sep 29, 2026
Resolve conflicts: - CHANGELOG.md: take main; the saved-trail entries move to docs/viewer-launch-changelog.md with the other held viewer entries (colbymchenry#2089). - __tests__/git-index-currency.test.ts: keep the Vitest 4 injection helpers, with colbymchenry#2082's note on why openSync is injected too. - package-lock.json: keep the Vitest 4 dependency set, regenerated for 1.6.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries @danusha2345's #1914 and #1915 (#1915 already contains all three of #1914's commits), rebased onto current
mainwith their commits and authorship intact, plus one maintainer commit on top. Supersedes both.What the contributor's commits do (#1910)
readFile, so a committed video or blob fixture was decoded in full only to be stored as skipped, and was then read again by every by-name pass. Reads now go through a bounded reader that re-checks the size on the open descriptor and stops one byte past the limit. An oversize file is stored and compared by a size stamp instead of its content, in indexing, sync, change detection, MCP freshness and the viewer..tsis not TypeScript. It's recognized from its head: the 0x47 sync byte at 16 consecutive 188-byte packet boundaries, and a binary head. That second condition keeps source text with aGat every stride, even one with a NUL in a comment, as TypeScript.The maintainer commit: judge the bytes where they're read, not at discovery
As written, discovery sniffed the head of every
.tsfile on every full scan: an open, a 3 KB read and a close per file before anything asked for its content. Measured with a synthetic 10,000-file TypeScript tree, warm cache, median of 7:scanDirectoryAsyncOn a slow disk or a network share that's one random read per
.tsfile per scan, the access pattern behind #1231 and the SMB reports (#1014, #448).The bytes are now judged where they're read anyway, so the check costs no I/O and runs only for a file that is new or changed:
indexFile;getChangedFilespaths.Discovery lists
.tsfiles by name again. A new test pins that it opens none of them on the walk or the git path; it fails against #1915's original code. A full index of the same tree is unchanged within noise: medians ~19.4 s on main and ~19.6 s here, interleaved runs.The same commit collapses the four #1910 changelog bullets (#1914 and #1915 each added both) to two, drops the benchmark figures, and credits @danusha2345.
Verification
main+ this + the isInitialized() accepts a schema-less codegraph.db, so one empty file in an ancestor makes a real index unreachable #1895 carry: 315 files, 5,461 passed, 0 failed..tsMPEG-TS video fixtures are parsed as TypeScript: 28s of CPU per 900KB file, and oversized ones are fully read into memory before the size check discards them #1910 test files plusgit-index-currency,sync,watcher,mcp-projectpath-lifecycleandmcp-stale-slicepass (164 passed).Fixes #1910
Co-authored-by: danusha2345 danusha2345@users.noreply.github.com
🤖 Generated with Claude Code