Skip to content

Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993

Description

[agent] Bench: npm/hosted wall time is +15% to +20% over the last 7 days of main (CPU +12% to +14%, peak RSS +3% to +5%). npm/rescan, npm/dry-run and npm/public-proxy moved the same way. Request counts are unchanged (127 / 67 / 151).

There is no single culprit commit. The cost built up across the range, about half before and half after 045d7ec7. This is gradual drift, not one bad PR.

Same-machine interleaved compare (base 2463257a #277 → head 9c43dfc9, run 2026-10-07):

scenario base wall head wall Δ wall [95% CI] Δ CPU Δ RSS requests
npm/hosted (full run) 287.9 ms 317.5 ms +15.0% [+10.1, +17.8] +12.0% +3.1% 127
npm/hosted (confirm, 15 pairs) 271.9 ms 321.8 ms +19.6% [+10.8, +25.6] +13.7% +4.8% 127
npm/rescan (full run) 270.5 ms 314.2 ms +15.8% [+9.3, +23.5] +13.0% +6.7% 127
npm/rescan (confirm) 256.2 ms 290.4 ms +12.6% [+7.5, +19.4] +11.8% +8.2% 127
npm/dry-run 272.6 ms 297.0 ms +15.1% [+2.8, +18.4] +13.2% +3.1% 67
npm/public-proxy 272.4 ms 302.7 ms +10.1% [+4.4, +16.5] +10.1% +3.0% 151

An A/A run of the same binary at two paths on this runner was clean (npm/hosted -2.3%), so the result isn't runner noise. On 2026-10-06 the weekly ratio was already 1.100 (CI [-0.3, +15.9]); before that it held at 1.02-1.05.

Narrowing (one extra build, 045d7ec7, the main SHA measured 10-03 to 10-05):

window npm/hosted npm/rescan
2463257a..045d7ec7 (55 commits) +5.8% [-8.4, +11.6] +3.0% [-0.3, +16.4]
045d7ec7..9c43dfc9 (74 commits) +10.6% [-3.7, +21.2] +4.8% [+0.3, +13.6]

Neither half is significant alone, so bisecting further won't name one commit.

Profile (callgrind, npm/dry-run fixture, base vs head): total instructions went up 9.4%, from 948M to 1,036M Ir. The added work is in new npm lock passes during VEX discovery and the rewrite:

Candidates, in order: #799 (48085cef), #646 (0685ba8c), #345 (6e7ef748) and #491 (73b17db5). Each one re-walks the 3000-entry package-lock.json, so sharing one parsed lock walk across the VEX npm passes would likely win back most of this.

Repro

export CARGO_PROFILE_PERF_INHERITS=release CARGO_PROFILE_PERF_LTO=thin CARGO_PROFILE_PERF_STRIP=none
git worktree add /tmp/base 2463257a && (cd /tmp/base && CARGO_TARGET_DIR=/tmp/tb cargo build --locked --profile perf -p socket-patch-cli)
cargo build --locked --profile perf -p socket-patch-cli -p socket-patch-bench
target/perf/socket-patch-bench compare --base /tmp/tb/perf/socket-patch --head target/perf/socket-patch -f '^npm/hosted$' -f '^npm/rescan$'

Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.80GHz (cloud sandbox), nproc = 4.

Why this was filed: the regression is confirmed (it held in a second round), and npm/hosted has measured ≥10% over the week window on two consecutive runs (10-06 1.100, 10-07 1.150/1.196). Progress is logged in #575.


Generated by Claude Code


Backlog review — 2026-10-08

Priority: P1 → P3. A roughly 15% microbenchmark regression at subsecond absolute runtime needs performance tracking, not P1 incident priority. Keep active #1008.

Activity

  1. mikolalysenko commented on Oct 7, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Triaged: priority:p1 (npm-family). No duplicate or open PR covers it. Progress is tracked in #575. The profile points at repeated package-lock.json walks in VEX discovery (#799, #345, #491) and in the lock cache (#646). The suggested fix is to share one parsed lock walk across the npm VEX passes.


    Generated by Claude Code

  2. mikolalysenko commented on Oct 7, 2026

    @mikolalysenko
    CollaboratorAuthor

    Being fixed in draft PR #1008 (batch fix for open pm:npm issues).

  3. mikolalysenko commented on Oct 8, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Bench: run 2026-10-08. main is now ea097142 (#1044). The npm drift is smaller today and below the gate on npm/hosted, but npm/dry-run still confirms.

    Weekly same-machine interleaved compare, base 2463257a (#277) → head ea097142, on a 4 vCPU Xeon @ 2.10GHz. A/A was clean.

    scenario base wall head wall Δ wall [95% CI] Δ CPU Δ RSS verdict
    npm/hosted 205.8 ms 215.3 ms +4.5% [-3.2, +7.2] +6.0% +5.7% ≈
    npm/rescan (round 1) 225.1 ms 238.1 ms +6.8% [+2.7, +19.5] +11.3% +8.8% flagged
    npm/rescan (round 2, 30+ pairs) 190.7 ms 210.2 ms +8.9% [+5.6, +12.9] +8.8% +9.2% ≈ (not confirmed)
    npm/dry-run (round 1) 183.0 ms 209.4 ms +14.7% [+10.4, +17.2] +10.8% +5.8% flagged
    npm/dry-run (round 2, 30+ pairs) 211.3 ms 238.9 ms +10.0% [+6.1, +14.5] +9.4% +5.6% confirmed
    npm/public-proxy 201.2 ms 219.5 ms +4.4% [+1.0, +13.4] +5.9% +5.8% ≈

    The daily A/B (9c43dfc9 → ea097142) shows no change on any npm scenario: hosted -3.2%, rescan +1.4%, dry-run +2.6%. Request counts are unchanged. The overhead is still there, about +5-10% wall and CPU and about +6-9% RSS on every npm scan, but it is no longer above 10% on npm/hosted. #1008 is the open fix.

    Repro: socket-patch-bench compare --base <2463257a build> --head <ea097142 build> -f '^npm/(hosted|rescan|dry-run)$' --runs 30


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions