Benchmark progress: socket-patch scan

[Mikola Lysenko (mikolalysenko)]

[agent] Bench: progress log for the socket-patch scan benchmark suite (crates/socket-patch-bench, .github/workflows/bench.yml). The daily benchmark steward rewrites this body on every run and posts one comment per run. Machine-readable history is in history.json on the bench/ledger branch. The first run (2026-10-02) was logged in #580 and backfilled there.

Last run

• When: 2026-10-09, about 06:35–08:45 UTC
• main: f3c6313a (Fix open npm issues, Fix open npm issues #1008). That is 37 first-parent commits since ea097142, including Stop re-parsing bun lock entries for every patch in scan (#580) #1051 (bun lock re-parse fix), Fix open npm issues #1008 (npm/bun/vlt lock rework), Scope project-mode NuGet crawls to the restore's resolved packages (#427) #1183 (NuGet restore-scoped crawl), Scope the project-mode cargo crawl to the crates Cargo.lock resolves (#1204) #1205 (Cargo.lock-scoped crawl), Decide whether a hosted patch is pinned through lockfile discovery alone #1058 (hosted pins decided through lockfile discovery) and Remove scan --apply/--vendor, get --no-apply and the download/gc aliases (#966) #1031 (removes scan --apply/--vendor; the suite never used them).
• Suite source: main, 45 scenarios. Bench: benchmark NuGet as a restored project (#1183 scoped crawl) #1250 updates the nuget fixture; it was measured separately and is not in today's numbers.
• Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.80GHz, cloud sandbox. Verdicts come only from same-machine interleaved compare. The 3a run was noisy today (pnpm/hosted ranged 224–396 ms; npm/hosted's 515 ms median is about 330 ms in every compare), so treat today's absolute medians loosely.
• Validation: 45/45 scenarios are valid on today's main. Request counts are unchanged in every scenario.
• Daily A/B (ea097142 → f3c6313a):
  • Big wins: bun/hosted -53.3%, bun/rescan -61.7%, bun-isolated -52% / -56% (Stop re-parsing bun lock entries for every patch in scan (#580) #1051). vlt/hosted -51.2%, vlt/rescan -52.3%. npm/rescan -25.5%, yarn-classic/rescan -22.8%, gradle/rescan -31.1%, composer/rescan -18.9%, pdm/rescan -17.1%, nuget/rescan -12.9%, and pnpm, yarn-berry, uv and pylock rescans 10–14% faster.
  • Peak RSS +15–17% on npm-family hosted scans (confirmed): npm/hosted +16.6% (44.0 → 50.6 MiB), yarn-classic/hosted +16.5%, npm/public-proxy +15.2% and npm/latency +15.6%. npm/dry-run +13.4% and yarn-berry/hosted +11.1% are under the 15% gate. Wall time is flat. Bisected with 3 builds: Fix open npm issues #1008 is cleared (identical RSS before and after). About half the growth is in ea097142..ef48495c (13 commits; Stop re-parsing bun lock entries for every patch in scan (#580) #1051, Make every --json top-level error a {code, message} object (#704) #1027 and Fix vendored revert reading a removed dependency as drift (#1132, #1140, #1142) #1147 are candidates) and half is Decide whether a hosted patch is pinned through lockfile discovery alone #1058 (4d06019b). This is under the 30% RSS issue gate, so it's ledger only.
• Weekly A/B (61cfb9b2, which predates Fix Bun/vlt bundled copies left unpatched (#469, #471) #472, → f3c6313a). The base fails validation on yarn-berry, bun-isolated and gradle (it predates those fixtures), so those have no weekly ratio.
  • bun is back to its pre-Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 baseline: bun/hosted -8.1% (≈), bun/rescan -26.3%. That's run 1 of 2 for closing Perf regression: bun/hosted wall +110% (1169ae68, #472) #578 (evidence commented).
  • vlt: +13.9% and +17.3% vs pre-Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 (it was about +110% yesterday). This is the accepted Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 residual (Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579).
  • Confirmed, below the issue gate: hatch/hosted wall +11.1% → +19.3% [+12.1, +25.0] in round 2 (~19 ms, just under the 20% gate). maven/hosted +10.2% → +14.7% [+7.4, +18.3] (~17 ms). Neither is bisected (the build budget went to RSS). npm-family RSS is +22–25%.
  • Not confirmed: bundler/rescan +10.6% → +8.3%, golang/hosted +14.0% → +6.6%, and npm/dry-run wall +10.0% → +1.0%.
  • Faster: poetry -44% / -52%, pdm -40% / -51% (Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877), nuget/hosted -21.2%, npm/rescan -20.1%, yarn-classic/rescan -19.2%.
• A/A sanity check: clean. npm/hosted +2.4%, hatch/hosted +1.9% and maven/hosted -1.0%, all ≈.
• Full compare wall time: 21.7 min for the daily run (45 scenarios plus 4 confirmation rounds) on this runner. That is still over the ~12-minute budget.

Scoreboard

Day: wall ratio vs ea097142. Week: wall ratio vs 61cfb9b2. The median ms/pkg across */hosted is 0.1716 today (from the noisy 3a run).

scenario	median wall (ms)	ms/pkg	day	week	status
npm/hosted	514.9	0.1716	1.004	0.999	OK; RSS +16.6% day (ledger)
npm/rescan	253.6	0.0845	0.745	0.799	OK
pnpm/hosted	313.0	0.1043	0.892	1.031	OK
pnpm/rescan	202.1	0.0674	0.862	0.902	OK
yarn-classic/hosted	332.8	0.1109	0.936	1.009	OK; RSS +16.5% day (ledger)
yarn-classic/rescan	221.2	0.0737	0.772	0.808	OK
yarn-berry/hosted	312.1	0.1040	0.908	—	OK
yarn-berry/rescan	269.5	0.0898	0.866	—	OK
bun/hosted	234.0	0.0780	0.467	0.919	recovered (#578, run 1/2)
bun/rescan	187.8	0.0626	0.383	0.737	recovered (#578, run 1/2)
bun-isolated/hosted	218.9	0.0730	0.482	—	OK (faster, #1051)
bun-isolated/rescan	223.1	0.0744	0.440	—	OK (faster, #1051)
vlt/hosted	635.9	0.4239	0.488	1.139	slow; -51% day, week +13.9% (accepted #472 residual)
vlt/rescan	598.9	0.3993	0.477	1.173	slow; -52% day, week +17.3% (accepted #472 residual)
pip/hosted	138.4	0.1384	1.072	1.069	OK
pip/rescan	122.6	0.1226	1.005	0.985	OK
uv/hosted	339.3	0.8482	1.010	1.036	slow (#836)
uv/rescan	210.8	0.5271	0.865	0.901	slow (#836)
pylock/hosted	213.2	0.5331	1.044	1.032	slow (#836)
pylock/rescan	202.6	0.5064	0.877	0.881	slow (#836)
poetry/hosted	197.3	0.4932	0.978	0.557	slow (2.9x, 2 runs since #877)
poetry/rescan	138.5	0.3463	0.945	0.482	OK
pipenv/hosted	181.4	0.4535	0.971	1.069	slow (#1246)
pipenv/rescan	139.4	0.3485	0.981	0.986	OK
pdm/hosted	170.1	0.4254	1.001	0.600	slow (2.5x, 2 runs since #877)
pdm/rescan	139.1	0.3478	0.829	0.489	OK
hatch/hosted	96.3	0.0963	1.020	1.111	regressed, week +19.3% confirmed (ledger)
hatch/rescan	94.9	0.0949	0.969	1.014	OK
bundler/hosted	246.2	0.3078	1.033	1.092	OK
bundler/rescan	220.0	0.2751	0.938	1.106	OK
composer/hosted	218.7	0.2733	1.047	1.099	OK
composer/rescan	180.7	0.2259	0.811	0.864	OK
cargo/hosted	340.5	0.5675	1.057	1.038	slow (#761)
cargo/rescan	325.5	0.5425	1.000	0.976	slow (#761)
golang/hosted	141.2	0.1176	1.088	1.140	OK
golang/rescan	101.3	0.0844	0.916	0.997	OK
nuget/hosted	106.6	0.1184	1.056	0.788	OK (fixture updated in #1250)
nuget/rescan	116.1	0.1290	0.871	0.886	OK (faster, #1183; fixture updated in #1250)
maven/hosted	124.7	0.1247	1.093	1.102	regressed, week +14.7% confirmed (ledger)
maven/rescan	87.9	0.0879	1.017	0.977	OK
gradle/hosted	215.0	0.2150	0.954	—	OK
gradle/rescan	219.7	0.2197	0.689	—	OK
npm/dry-run	282.3	0.0941	0.966	1.100	OK; RSS +13.4% day
npm/public-proxy	391.1	0.1304	1.036	1.060	OK; RSS +15.2% day (ledger)
npm/latency	786.4	0.2621	1.002	1.015	OK; RSS +15.6% day (ledger)

Trends

Median <pm>/hosted wall time per run, oldest → newest (ms). Runners differ between runs (10-05, 10-07 and 10-09 ran on slower or noisier sandboxes), so read these as trends only. The poetry and pdm drop on 10-08 is #877. The bun drop on 10-09 is #1051. The 10-09 npm value is a noisy outlier.

pm	10-02	10-03	10-04	10-05	10-06	10-07	10-08	10-09
npm	198.6	204.0	198.3	296.1	233.5	439.7	222.4	514.9
pnpm	153.6	180.5	162.1	234.8	197.4	216.5	188.2	313.0
yarn-classic	179.9	195.4	166.9	227.5	181.9	303.1	199.8	332.8
yarn-berry	219.5	228.8	208.6	293.5	242.0	412.8	231.0	312.1
bun	289.7	303.6	321.5	394.6	297.0	396.5	350.7	234.0
bun-isolated	—	—	336.6	392.4	342.4	465.8	359.2	218.9
vlt	843.8	894.2	764.0	1109.7	840.0	1246.6	840.7	635.9
pip	102.2	97.0	81.8	125.8	136.1	126.3	104.5	138.4
uv	136.0	143.9	127.1	244.0	156.6	220.3	165.0	339.3
pylock	113.6	142.1	126.2	179.8	122.1	194.8	127.8	213.2
poetry	187.6	221.4	201.6	342.9	261.7	369.5	111.3	197.3
pipenv	106.2	96.5	86.5	143.9	100.2	133.8	105.3	181.4
pdm	180.4	180.5	162.4	260.3	193.6	309.5	133.2	170.1
hatch	—	—	—	—	—	86.1	78.0	96.3
bundler	136.0	180.4	141.5	205.9	165.6	204.7	149.9	246.2
composer	122.2	123.0	115.4	180.8	122.9	175.7	136.0	218.7
cargo	228.4	262.5	258.5	364.1	255.1	332.4	254.0	340.5
golang	68.5	71.4	64.3	110.3	80.5	95.1	104.8	141.2
nuget	90.9	90.1	87.2	133.3	91.3	98.5	73.4	106.6
maven	79.3	81.6	62.0	87.4	73.5	98.8	67.8	124.7
gradle	—	—	—	—	170.5	—	197.6	215.0

Open issues and PR

• Perf regression: bun/hosted wall +110% (1169ae68, #472) #578: Perf regression: bun/hosted wall +110% (1169ae6, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472). Recovered today: -53% day, ≈ vs pre-Fix Bun/vlt bundled copies left unpatched (#469, #471) #472. It closes after a 2nd run within 5%. Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009 (bun fixes) is still open.
• Slow scan: pipenv hosted 2.6x median ms/pkg (per-patch Pipfile.lock re-tokenize) #1246 (new): Slow scan: pipenv. Flagged in 3 consecutive runs. Hot spot: redirect::pipenv::entries, 61%.
• Slow scan: cargo hosted 3.8x median ms/pkg (per-patch Cargo.lock re-parse) #761: Slow scan: cargo.
• Slow scan: uv/pylock hosted 4.1x/3.0x median ms/pkg (per-patch lock re-serialize) #836: Slow scan: uv/pylock.
• Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993: closed (npm drift; Fix open npm issues #1008 merged). Today npm/hosted is ≈ day and week, and npm/dry-run's wall did not confirm.
• Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579 (closed): the vlt Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 cost is accepted. Most of it is gone today.
• Benchmark tracking: socket-patch scan #580: Benchmark tracking.
• PR Bench: benchmark NuGet as a restored project (#1183 scoped crawl) #1250 (bench/refresh): the nuget fixture gets obj/project.assets.json, so the suite times Scope project-mode NuGet crawls to the restore's resolved packages (#427) #1183's restore-scoped crawl.

Standing slow-systems list

A package manager is slow when its hosted ms/pkg is more than 2x the median across */hosted, or when any of its scenarios takes over 500 ms (npm/latency is excluded). An issue opens after 3 consecutive runs, capped at 3 new issues per run. Today's 3a run was noisy, so the same ranking was cross-checked against the daily compare head samples (median 0.143 ms/pkg). Both put the same 7 package managers over 2x.

pm	x median today (3a / compare)	consecutive runs	issue	hot spot
uv	4.9x / 3.9x	8	#836	PythonLockSession::rewrite 39%: the whole lock is re-serialized per patched dep (callgrind 10-05)
cargo	3.3x / 4.9x	8	#761	formats::cargo::CargoLock::parse inside rewrite_cargo, 79% (callgrind 10-04). #1205 scoped the crawl but not the rewrite
pylock	3.1x / 3.7x	8	#836	Same as uv (callgrind 10-05)
poetry	2.9x / 3.3x	2 since #877	none	Likely the batched-rewrite and VEX lock-extract split, as for pdm (not profiled)
pipenv	2.6x / 2.8x	3	#1246	patch::redirect::pipenv::entries 61% inclusive (properties 40%): plan() re-tokenizes Pipfile.lock once per patch (callgrind 10-09)
pdm	2.5x / 2.9x	2 since #877	none	lock_fragments::rewrite_batch 45%, vex pypi_locks::extract 26% (callgrind 10-08)
vlt	2.5x / 2.9x	8	none (#579 accepted)	Halved today. Its two scenarios are still over 500 ms (636 / 599 ms)

If poetry and pdm are still over 2x tomorrow, that's their 3rd run since #877 and each gets an issue. No scenario's peak RSS is above 2x the median (34.2 MiB). Scenarios over 500 ms: vlt/hosted and vlt/rescan (npm/hosted's 515 ms in the 3a run is noise; it was about 330 ms in every compare).

Coverage-gap backlog

New since 10-08 (from ea097142..f3c6313a):

• Covered by Bench: benchmark NuGet as a restored project (#1183 scoped crawl) #1250: NuGet restore-scoped crawl (Scope project-mode NuGet crawls to the restore's resolved packages (#427) #1183).
• Cargo.lock-scoped crawl (Scope the project-mode cargo crawl to the crates Cargo.lock resolves (#1204) #1205): already exercised. The fixture has a Cargo.lock and no vendor/, so it takes the new lookup. Uncovered: extra crates in ~/.cargo/registry that the lock doesn't name (the scope's whole point), which would need a fixture change that the base can't validate.
• Non-UTF-8 requirements files (Fix requirements files read as absent when not UTF-8 (#1120, #1119) #1152): Latin-1 with a PEP 263 coding line, or UTF-16 with a BOM. Uncovered.
• Gem lock re-sort on re-scan (Fix gem lock section order after a hosted re-scan (#1186) #1190): a superseding patch or rotated grant moves a GEM section. The rescan only covers the same grant. Uncovered.

Carried over:

• Yarn classic offline mirror (Fix yarn classic offline-mirror config resolution (#1013, #1078) #1083), multi-lockfile governance (Decide which lockfile governs installs in one table #1044) and patch-generation re-pin (Replace a superseded patch generation's wiring on re-pin and remove #1035).
• Hatch variants: [tool.hatch.envs] in pyproject only, hatch + requirements.txt, uv-installer refusal.
• Gradle variants: Kotlin DSL, multi-project builds, version catalogs, verification-metadata.xml, mavenLocal() (Full Gradle support in agent, hosted and vendored modes #646).
• Yarn 4 pnpm linker (Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496); yarn berry re-pin of __archiveUrl pins (Fix yarn berry hosted pin leaking npm auth (#404) #465).
• bun.lockb binary lockfile (Fix Bun/vlt bundled copies left unpatched (#469, #471) #472); bundled copies (redirect_*_bundled_instance_skipped, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472/Fix in-run hosted VEX attesting npm bundled copies (#325) #669).
• pnpm global store / modulesDir store (Fix pnpm global-store transitive deps passing silently (#362) #829, Fix pnpm modulesDir store being skipped (#661, #696) #698); vlt 1.3 brotli lock nodes (Fix vlt 1.3 brotli lock nodes being refused (#372) #820).
• Python env discovery: UV_PROJECT_ENVIRONMENT / PDM venvs (Fix Python env discovery ignoring uv/PDM env (#502, #525, #528) #540), .egg-info (Fix Python crawler missing .egg-info installs (#447) #452), Poetry envs.toml (Fix Poetry venv selection ignoring envs.toml (#476, #526) #527), lock-only requirements.txt (Fix lock-only requirements.txt discovery (#412, #523) #530).
• pnpm-workspace.yaml append (Fix pnpm-workspace.yaml key detection and appends (#400, #402) #414): the non-fresh-file case.
• Gemfile multi-layer .bundle/config (Fix Bundler settings resolution order (#483, #507) #532, Gem settings resolution skips Bundler's global config (~/.bundle/config / BUNDLE_USER_CONFIG), so a global cache_path or gemfile gets no warning or refusal and VEX attests an unpatched install #577), gems.locked (Fix gem lock readers ignoring gems.locked (#736) #750), Bundler 4 standalone (Fix gem crawler missing Bundler 4 standalone bundle (#796) #797).
• Suite runtime: a full compare takes 16–22 min on a 4 vCPU sandbox. Before more scenarios land, shrink something, e.g. fewer --runs for the npm family's */rescan twins, or merge bun/rescan into bun-isolated/rescan.
• deno: no hosted rewrite, so no scenario is needed.

Stale / redundant scenarios

None removed. #1031 removed scan --apply/--vendor, and no scenario used them. Merging hosted/rescan pairs needs 14+ days of history; there are 8 days so far. Candidate: bun/rescan overlaps bun-isolated/rescan on the lockfile path. Both moved together today (-62% / -56%).

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: mirror test. The bench/ledger workflow posted this comment. Daily runs add their summaries after it.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-03, main 045d7ec7, suite from main (#485 landed).

• All 39 scenarios validated on today's main.
• Daily A/B vs 1169ae68 (the previous run): no regressions across 37 comparable scenarios. yarn-berry/* had no comparable base because of Fix yarn berry hosted pin leaking npm auth (#404) #465's intentional package.json pin. A/A was clean.
• Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a; the 7-day SHA predates v5): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions remain, still unfixed. bun/* +117–125% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578) and vlt/* +139–153% (Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579), with fresh data commented on both.
• Main's bench gate failed at 203e092b and 0ac5b91a: Fix yarn berry hosted pin leaking npm auth (#404) #465 changed yarn-berry's rewritten files before Stream ZIP member hashes during vendored verification #587 updated the fixture. It is green again at 045d7ec7.
• Slow list (run 2 of 3): vlt 4.7x, poetry 4.3x, pdm 3.5x, cargo 3.4x, uv 2.8x, pylock 2.8x. pipenv dropped off (1.9x).
• Suite change: Bench: cover Bun isolated .bun store; fix rescan restore #667 adds bun-isolated/{hosted,rescan}, which covers Bun 1.3's default .bun store walked since Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496. A pre-Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496 binary fails it.
• New issues: none.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Janitor: state drift. The body still lists #579 as an open regression (vlt/hosted and vlt/rescan rows, the weekly A/B line, and the open-issues list), but a maintainer closed #579 at 2026-10-03 12:24Z as accepted: the tool is slower because it now scans code it used to skip. #578 (bun/*) is still open.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-04, main 045d7ec7 (unchanged since 10-03), suite from main + #667.

• All 41 scenarios validated on today's main.
• Daily A/B: skipped, because main didn't move. Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions. bun/* +117% / +127% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented). vlt/* +112% / +116%, now accepted: a maintainer closed Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579. A/A clean.
• Slow list, run 3 of 3: vlt 4.5x, poetry 4.5x, cargo 3.8x, pdm 3.6x, uv 2.8x, pylock 2.8x.
• New issues (callgrind hot spots): Slow scan: poetry hosted 4.5x median ms/pkg (per-patch poetry.lock re-parse) #760 poetry, Slow scan: cargo hosted 3.8x median ms/pkg (per-patch Cargo.lock re-parse) #761 cargo, Slow scan: pdm hosted 3.6x median ms/pkg (per-patch pdm.lock re-parse) #762 pdm. Each one re-parses the whole lockfile per patched dep. uv and pylock overflow the 3-issue cap and get filed next run. vlt gets no slow issue because its cost was accepted on Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579.
• Suite change: Bench: cover Bun isolated .bun store; fix rescan restore #667 gains 93b1c929. A rescan whose prep scan failed validation no longer leaves the fixture dirty, which had made compare report the other binary INVALID (seen today on bun-isolated/rescan).

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-05, main 045d7ec7 (unchanged since 10-02), suite from main + #667.

• All 41 scenarios validated on today's main.
• Daily A/B: skipped, because main didn't move. Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions, bun/hosted +107%, bun/rescan +92% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented) and vlt/hosted +123% (accepted, Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579). The other 33 comparable scenarios are unchanged. A/A clean.
• Today's runner was about 1.4x slower on every scenario. The weekly A/B is flat, so absolute medians jumped for machine reasons only.
• New issue Slow scan: uv/pylock hosted 4.1x/3.0x median ms/pkg (per-patch lock re-serialize) #836: uv and pylock, slow 4 runs in a row. PythonLockSession::rewrite re-serializes the whole lock per patched dep (callgrind: 39% / 38%, mostly toml_edit Display).
• Slow list: poetry 5.8x, vlt 5.0x, pdm 4.4x, uv 4.1x, cargo 4.1x, pylock 3.0x. pipenv is newly borderline at 2.4x (run 1, noisy).
• Suite: no changes needed. Bench: cover Bun isolated .bun store; fix rescan restore #667 is unchanged and still ready for review.
• Gap: vlt/rescan's weekly confirm round was cut short by a session timeout (its cost is already accepted).

---

Generated by Claude Code

[github-actions[bot]]

[agent] Janitor: state drift. The coverage-gaps note for hatch still says "Open PRs #700/#743/#680 are reworking Hatch". #743 (0475695f) and #680 (cd75b38e) merged on 2026-10-05. Only #700 is still open.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-06, main 9c43dfc9 (74 commits since 045d7ec7), suite from main, plus gradle/* from #925.

• All 41 scenarios validated on today's main. The 2 new gradle/* scenarios validate too.
• Daily A/B (045d7ec7 → 9c43dfc9): no regressions and no request-count changes. nuget/hosted is 26% faster (0.739 [0.686, 0.901]), likely from Unify hosted NuGet routing and XML splice anchors #597.
• Weekly A/B: the base is 2463257a (v5 prerelease: scan → vex → vendor workflow, hosted by default #277), because the 7-day SHA f6b7fb9e predates the v5 output. Only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions remain: bun/hosted +112% and bun/rescan +123% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented), vlt +105%/+133% (accepted). A/A is clean.
• New PR Bench: cover Gradle and Hatch hosted modes #925: adds gradle/hosted and gradle/rescan, covering Gradle hosted mode (Full Gradle support in agent, hosted and vendored modes #646). Measured at 0.17 ms/pkg; A/A is clean.
• Slow list: poetry 4.8x, vlt 4.1x, pdm 3.6x, cargo 3.1x, uv 2.9x, pylock 2.2x, all already tracked. pipenv is back under 2x (1.8x). No new issues.
• Next gap: a hatch scenario. Fix uv/Hatch hosted re-pin to a newer patch (#742, #650) #743/Fix Hatch takeover leaving direct-ref permission (#674) #680 have landed.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-07, main 9c43dfc9 (unchanged since 10-06, so there was no daily A/B), suite from main, plus hatch/* from #925.

• All 41 scenarios validated on today's main, and the 2 new hatch/* scenarios validate too. Request counts are unchanged everywhere.
• New: confirmed weekly npm drift. npm/hosted is +15.0% / +19.6% and npm/rescan +15.8% / +12.6% vs 2463257a (two rounds; A/A clean). Filed as Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993. No single culprit: about half lands before 045d7ec7 and half after. Callgrind shows +9.4% instructions, from new npm lock passes (Fix npm rewiring git/URL/file lock entries (#326) #345, Fix npm overrides of git deps being refused (#490) #491, Fix npm VEX attesting a patch the twin lock lacks (#798) #799, Full Gradle support in agent, hosted and vendored modes #646).
• Known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions are unchanged: bun +99% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented), vlt +128% / +143% (accepted). pip/rescan +12.7% didn't hold up in the confirmation round (+8.9%). nuget/hosted is still 19.5% faster.
• Bench: cover Gradle and Hatch hosted modes #925 updated: adds hatch/hosted and hatch/rescan (a lockless hatch.toml env rewrite, 1000 pkgs / 25 patched, ~80 ms). A/A is clean.
• Slow list: poetry 5.9x, vlt 5.4x, pdm 5.0x, cargo 3.6x, uv 3.5x, pylock 3.1x, all already tracked. pipenv is back at 2.15x. Its hot spot is redirect::pipenv::entries, 69%. No slow-system issue yet.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Janitor: drift. The body still lists #760 (poetry) and #762 (pdm) as slow, with "Draft fix in #877". #877 merged on 2026-10-07 (79c25971), and both issues are closed as completed. Please re-measure poetry/pdm hosted and rescan on main db83f014.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-08, main ea097142 (73 commits since 9c43dfc9), suite from main (#925 merged, 45 scenarios).

• All 45 scenarios validated on today's main. Request counts are unchanged everywhere. A/A is clean.
• Daily A/B: Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877 makes poetry 46–49% faster and pdm 44–46% faster. yarn-berry/rescan +14.3% didn't hold up in the confirmation round (+2.6%). No daily regressions.
• Weekly vs 2463257a: the Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993 npm drift shrank. npm/hosted is +4.5% (≈) and npm/rescan +8.9% (≈); only npm/dry-run still confirms (+10.0%). Data is on Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993, fix Fix open npm issues #1008. bun +102% / +125% is unchanged (data on Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, fix Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009). vlt is accepted.
• New, below the issue gates (ledger only): yarn-classic/rescan wall +13.3% (confirmed, ~20 ms), hatch/rescan CPU +12.6%, and pdm/hosted RSS +17.6% (from Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877; its wall is -41%). pip/rescan was flagged and not confirmed for the 2nd run in a row (+9.9%).
• Slow list: vlt 4.7x, cargo 3.5x, uv 3.4x, pdm 2.8x, pylock 2.7x, poetry 2.3x, pipenv 2.2x. poetry and pdm restart their counts after Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877. pipenv is at 2 runs, so tomorrow decides whether it gets an issue. No new issues today.
• Suite: no changes and no PR. New gaps: yarn classic offline mirror (Fix yarn classic offline-mirror config resolution (#1013, #1078) #1083), multi-lockfile governance (Decide which lockfile governs installs in one table #1044), generation re-pin (Replace a superseded patch generation's wiring on re-pin and remove #1035). A full compare takes 15.8 min, over the 12-minute budget.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-09, main f3c6313a (37 first-parent commits since ea097142), suite from main (45 scenarios).

• All 45 scenarios validate on today's main. Request counts are unchanged everywhere, and A/A is clean.
• Daily A/B: bun -53% to -62% from Stop re-parsing bun lock entries for every patch in scan (#580) #1051, vlt -51% and -52%, and many rescans 10–31% faster (npm, yarn-classic, gradle, composer, pdm, nuget). No confirmed wall regressions.
• Daily RSS: npm-family hosted peak RSS is up 15–17% (npm/hosted 44.0 → 50.6 MiB; confirmed). It's split about evenly between Decide whether a hosted patch is pinned through lockfile discovery alone #1058 and ea097142..ef48495c; Fix open npm issues #1008 is cleared. Under the 30% gate, so ledger only.
• Weekly vs 61cfb9b2 (pre-Fix Bun/vlt bundled copies left unpatched (#469, #471) #472): bun is back to baseline (run 1/2 to close Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, evidence commented). The vlt residual is +14% / +17% (accepted). Confirmed but below the gates: hatch/hosted +19.3% (~19 ms) and maven/hosted +14.7% (~17 ms). bundler/rescan, golang/hosted and npm/dry-run wall did not confirm.
• New issue Slow scan: pipenv hosted 2.6x median ms/pkg (per-patch Pipfile.lock re-tokenize) #1246: pipenv is slow for a 3rd consecutive run (2.6x the median ms/pkg). Hot spot: redirect::pipenv::entries, 61%, Pipfile.lock re-tokenized per patch.
• Slow list: uv, cargo, pylock, poetry, pipenv, pdm, vlt. poetry and pdm reach their 3rd run tomorrow.
• Suite PR Bench: benchmark NuGet as a restored project (#1183 scoped crawl) #1250: the nuget fixture gets obj/project.assets.json, so it times Scope project-mode NuGet crawls to the restore's resolved packages (#427) #1183's restore-scoped crawl. New gaps: non-UTF-8 requirements (Fix requirements files read as absent when not UTF-8 (#1120, #1119) #1152) and gem section re-sort on re-scan (Fix gem lock section order after a hosted re-scan (#1186) #1190).

---

Generated by Claude Code