Skip to content

Perf regression: bun/hosted wall +110% (1169ae68, #472) #578

Closed
#1009
Closed
Perf regression: bun/hosted wall +110% (1169ae68, #472)#578
#1009
Labels
agent:claimedagent:triagedbenchsocket-patch scan benchmark suiteperf-regressionConfirmed performance regressionpm:bunBunpriority:p2

Description

@mikolalysenko
Mikola Lysenko (mikolalysenko)
opened on Oct 2, 2026

[agent] Bench: bun/hosted and bun/rescan take about 2.1x as long since #472 (1169ae6, "Fix Bun/vlt bundled copies left unpatched"). Wall time is up 105–117% and CPU time 88–96%. The request count is unchanged (127).

Same-machine interleaved compare (socket-patch-bench, PR #485 suite)

round base scenario base wall head wall Δ wall [95% CI] Δ CPU Δ RSS requests
daily, 15 pairs + 10 confirm 6e7ef748 (main −24h) bun/hosted 144.2 ms 302.7 ms +116.6% [+103.7, +129.7] +96.2% +0.0% 127
daily, 15 pairs + 10 confirm 6e7ef748 bun/rescan 132.3 ms 277.8 ms +113.2% [+101.8, +123.0] +90.0% +0.0% 127
confirm, 25 pairs + 20 6e7ef748 bun/hosted 144.6 ms 286.5 ms +105.5% [+95.7, +112.7] +87.5% 127
confirm, 25 pairs + 20 6e7ef748 bun/rescan 130.2 ms 278.0 ms +113.7% [+107.0, +117.5] +89.9% 127
bisect, 11 pairs cbf1f748 (parent of #472) bun/hosted 142.0 ms 295.5 ms +109.5% [+103.6, +117.5] +94.8% 127

An A/A check on the same runner (head against a copy of itself, 3 scenarios) found no regression, so the runner was not too noisy.

Hot spot

Callgrind on bun/hosted (head) puts 62% of all instructions in socket_patch_core::vendor::bun_lock_text::is_bundled_entry, nearly all of it inside serde_json::from_str::<Value>.

rewrite_bun_lock (crates/socket-patch-core/src/patch/redirect/mod.rs, the for dep in &npm { for entry in &entries { loop) calls is_bundled_entry(entry) before it checks the spec:

if is_bundled_entry(entry)
    && (spec == target_spec
        || spec == url_spec
        || is_prior_hosted_bun_spec(&spec, &fname, &dep.artifact_url))

That parses every entry's metadata object as JSON once per patched dependency: 60 patches × 3000 entries is 180k JSON parses, where the old loop did string compares. Proposed fix (not applied):

  • swap the operands so the cheap spec match runs first and is_bundled_entry only runs on a matching entry; or
  • compute bundled once per entry before the dep loop.

The behavior stays the same either way. bundled_matches in vendor/bun_lock.rs has the same shape (it filters on is_bundled_entry first) on the vendor path.

Repro

export CARGO_PROFILE_PERF_INHERITS=release CARGO_PROFILE_PERF_LTO=thin CARGO_PROFILE_PERF_STRIP=none
git worktree add /tmp/base cbf1f748 && (cd /tmp/base && CARGO_TARGET_DIR=/tmp/tb cargo build --locked --profile perf -p socket-patch-cli)
# on a checkout of #485 merged with main:
cargo build --locked --profile perf -p socket-patch-cli -p socket-patch-bench
target/perf/socket-patch-bench compare --base /tmp/tb/perf/socket-patch --head target/perf/socket-patch -f '^bun/'

Runner: 4 vCPU (nproc = 4), Intel(R) Xeon(R) Processor @ 2.10GHz, cloud sandbox.

Generated by Claude Code

Backlog review — 2026-10-08

Priority: P1 → P2. The Bun regression is measurable and worth fixing, but the supplied subsecond benchmark does not establish a P1 user-facing performance outage.

Activity

  1. Mikola Lysenko (mikolalysenko)
    added
    pm:bunBun
    benchsocket-patch scan benchmark suite
    perf-regressionConfirmed performance regression
    agent:triaged
    priority:p1
    on Oct 2, 2026

  2. mikolalysenko commented on Oct 2, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 2, 2026
    CollaboratorAuthor

    [agent] Triaged as priority:p1 (Bun, npm-family). Confirmed on main (6cd3754): rewrite_bun_lock (patch/redirect/mod.rs:3723) still calls is_bundled_entry before the spec compare, and vendor/bun_lock.rs:1048 filters the same way. Related to #579 (same culprit, #472) but a different hot spot and fix, so not clustered. No open PR addresses it.

    Generated by Claude Code

  3. mikolalysenko commented on Oct 3, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 3, 2026
    CollaboratorAuthor

    [agent] Bench: still regressed on main 045d7ec7 (2026-10-03 run).

    round base scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    weekly, 7 pairs + confirm 2463257a (#277) bun/hosted 142.3 ms 301.5 ms +116.6% [+108.1, +123.6] +92.4% 127
    weekly, 7 pairs + confirm 2463257a (#277) bun/rescan 127.8 ms 286.1 ms +124.7% [+112.7, +132.1] +104.3% 127
    daily, 9 pairs 1169ae68 (#472) bun/hosted 312.4 ms 314.7 ms −0.7% [−3.2, +6.4] −0.3% 127

    None of the commits since #472 touched the Bun rewrite: rewrite_bun_lock still calls is_bundled_entry(entry) before it compares the spec. The A/A check on this runner was clean. The new bun-isolated/* scenario in #667, which uses Bun's isolated .bun store, runs at about the same 320 ms, so it most likely has the same hot spot.

    Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.80GHz, cloud sandbox.

    Generated by Claude Code

  4. mikolalysenko commented on Oct 4, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 4, 2026
    CollaboratorAuthor

    [agent] Bench: still regressed on main 045d7ec7 (2026-10-04 run). main hasn't moved since the 10-03 run, so there was no daily A/B.

    round base scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    weekly, 7 pairs + 10 confirm 2463257a (#277) bun/hosted 134.4 ms 298.2 ms +116.7% [+113.3, +127.9] +105.0% 127 = 127
    weekly, 7 pairs + 10 confirm 2463257a (#277) bun/rescan 125.6 ms 284.6 ms +126.7% [+118.8, +140.0] +106.4% 127 = 127

    A/A on the same runner was clean (npm/hosted −1.0%, vlt/hosted +0.9%, poetry/hosted −0.4%). Runner: 4 vCPU Intel Xeon @ 2.10GHz.

    For reference, the matching vlt regression (#579) was closed by a maintainer as an accepted cost of scanning previously skipped copies. If the same reasoning applies to Bun's bundled-entry check, this can be closed the same way. The bench steward won't close it, since a human triaged it.

    Generated by Claude Code

  5. mikolalysenko commented on Oct 5, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 5, 2026
    CollaboratorAuthor

    [agent] Bench: still regressed on main 045d7ec7 (2026-10-05 run). main hasn't moved since 10-02, so there was no daily A/B.

    round base scenario base wall head wall Δ wall verdict
    weekly, 7 + 10 confirm pairs 2463257a (#277) bun/hosted 181.5 ms 369.9 ms +107.3% Regression
    weekly, 7 + 10 confirm pairs 2463257a (#277) bun/rescan 201.8 ms 386.1 ms +92.2% Regression

    The A/A check (npm/hosted, uv/hosted, poetry/hosted) was clean: +1.6%, +0.8% and +0.2%. Runner: 4 vCPU Intel Xeon @ 2.80GHz. The hot spot is unchanged (see earlier comments). No new data argues either for or against accepting this the way #579 was. That decision is still with a human.

    Generated by Claude Code

  6. mikolalysenko commented on Oct 6, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 6, 2026
    CollaboratorAuthor

    [agent] Bench: run 2026-10-06. The regression is still present on main 9c43dfc9.

    This run used a same-machine interleaved weekly A/B against 2463257a (#277) on a 4 vCPU Xeon @ 2.10GHz:

    scenario base wall head wall Δ wall [95% CI] Δ CPU
    bun/hosted 148.2 ms 314.1 ms +111.6% [+92.2, +121.0] +97.2%
    bun/rescan 135.9 ms 299.6 ms +122.5% [+114.7, +137.0] +103.8%

    The daily A/B against 045d7ec7 was flat: bun/hosted +0.8%, bun/rescan +5.4%. None of the 74 commits since the last run touched this regression. Requests are unchanged at 127.

    Generated by Claude Code

  7. mikolalysenko commented on Oct 7, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 7, 2026
    CollaboratorAuthor

    [agent] Bench: run 2026-10-07. Still regressed on main 9c43dfc9. main hasn't moved since the 10-06 run, so there was no daily A/B.

    Weekly same-machine compare (base 2463257a → head 9c43dfc9, 4 vCPU Xeon @ 2.80GHz):

    scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    bun/hosted 210.0 ms 414.3 ms +99.2% [+85.2, +104.3] +84.7% 127
    bun/rescan 216.0 ms 425.1 ms +98.7% [+89.2, +110.2] +83.7% 127

    A/A on this runner was clean. Nothing has changed since the last comment: the culprit is still #472, and accepting it the way #579 was is still a human call.

    Generated by Claude Code

  8. Mikola Lysenko (mikolalysenko)
    added
    agent:claimed
    on Oct 7, 2026

  9. mikolalysenko commented on Oct 7, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 7, 2026
    CollaboratorAuthor

    Claim-ID: agent/fix-bun-open-issues. Draft fix PR: #1009 (it covers all open pm:bun issues).

  10. Mikola Lysenko (mikolalysenko)
    added 4 commits that reference this issue on Oct 7, 2026

    Check bun spec match before the bundled-entry parse (#578)

    b8475ca

    Reconcile #371 with #578's bun.lock hot-loop budget

    94da1cb

    Tighten Bun user-tarball checks from #497 review (#497)

    edc8745

    Reconcile #497 with #578's bundled-check budget

    4528fde

  11. mikolalysenko commented on Oct 8, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 8, 2026
    CollaboratorAuthor

    [agent] Bench: run 2026-10-08. Still regressed on main ea097142.

    Weekly same-machine interleaved compare, base 2463257a (#277) → head ea097142, on a 4 vCPU Xeon @ 2.10GHz. A/A was clean.

    scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    bun/hosted 170.3 ms 346.6 ms +102.1% [+82.3, +115.3] +91.1% 127 (=)
    bun/rescan 137.4 ms 318.6 ms +125.5% [+117.5, +142.2] +105.9% 127 (=)

    Daily (9c43dfc9 → ea097142): bun/hosted -2.8%, bun/rescan +1.9%. Nothing that landed today moved it. #1009 is the open fix.

    Generated by Claude Code

  12. Mikola Lysenko (mikolalysenko)
    added
    priority:p2
    and removed
    priority:p1
    on Oct 8, 2026
  13. Mikola Lysenko (mikolalysenko)
    added a commit that references this issue on Oct 8, 2026

    Merge main into agent/fix-bun-open-issues

    46a0dd1

  14. mikolalysenko commented on Oct 9, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 9, 2026
    CollaboratorAuthor

    [agent] Bench: run 2026-10-09. bun is back at its pre-#472 speed on main f3c6313a. #1051 (830749f9, "Stop re-parsing bun lock entries for every patch in scan") landed in this range.

    Same-machine interleaved compare on a 4 vCPU Xeon @ 2.80GHz. A/A was clean.

    Daily, ea097142 → f3c6313a:

    scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    bun/hosted 395.7 ms 188.2 ms -53.3% [-55.2, -52.1] -49.9% 127
    bun/rescan 410.2 ms 153.4 ms -61.7% [-66.1, -60.0] -56.9% 127
    bun-isolated/hosted 457.6 ms 230.0 ms -51.8% [-56.0, -46.7] -40.6% 127
    bun-isolated/rescan 438.4 ms 188.2 ms -56.0% [-58.6, -51.6] -45.4% 127

    Weekly, against 61cfb9b2, which predates #472 (1169ae68) and so serves as the pre-regression baseline:

    scenario base wall head wall Δ wall [95% CI] Δ CPU requests
    bun/hosted 238.9 ms 244.3 ms -8.1% [-13.1, +15.3] -0.8% 127
    bun/rescan 197.5 ms 147.4 ms -26.3% [-30.8, -14.5] -19.3% 127

    That is run 1 of 2 within 5% of the pre-regression baseline. If tomorrow's run agrees, this issue can close. It stays open for now; #1009 is still open for the other bun issues.

    Generated by Claude Code

  15. Mikola Lysenko (mikolalysenko)
    closed this as completedin #1009on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:claimedagent:triagedbenchsocket-patch scan benchmark suiteperf-regressionConfirmed performance regressionpm:bunBunpriority:p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions