diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 0c6352d76..5475e95bb 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -76,7 +76,7 @@ Every subcommand accepts the same set of "global" flags via a single shared `Glo `--offline` means the same thing on every command (v3.0): never contact the network, fail loudly when a required local source is missing. On `repair`, `--offline` and `--download-only` are mutually exclusive (exit 2). `scan` and `get` need remote data for their core function (patch discovery / patch fetch), so `--offline` refuses them up front — exit 1 with an error naming the offline gate (JSON: `status: "error"`), before any crawl, client build, or network contact. This covers `scan --mode vendored` too: offline vendored staging is `vendor --offline`'s job. -The `--strict` mismatch policy applies to the in-place apply paths (apply/get/scan --mode agent/hook/go redirect). DEFAULT (v3.4): a file whose on-disk content matches neither the patch's beforeHash nor its afterHash is overwritten with the FULL verified patched content (the diff strategy self-disables on a wrong base; archive/blob writes are hash-gated to exactly afterHash; the missing blob is downloaded on demand) and surfaced as a `content_mismatch_overwritten` stderr warning + Skipped event. A file the patch adds (empty beforeHash) that already exists with other content is the same case. `--strict` turns that case into a hard error. Rollback of an added file deletes it; the content it replaced is not kept. `--force` overrides `--strict` and additionally skips missing files. Vendor staging is unaffected (it always auto-overwrites into its private stage). +The `--strict` mismatch policy applies to the in-place apply paths (apply/get/scan --mode agent/hook/go redirect). DEFAULT (v3.4): a file whose on-disk content matches neither the patch's beforeHash nor its afterHash is overwritten with the FULL verified patched content (the diff strategy self-disables on a wrong base; archive/blob writes are hash-gated to exactly afterHash; the missing blob is downloaded on demand) and surfaced as a `content_mismatch_overwritten` stderr warning + Skipped event (agent-mode `get` / `scan --json`: a `warnings[]` entry, see "Agent-mode mismatch overwrites"). A file the patch adds (empty beforeHash) that already exists with other content is the same case. `--strict` turns that case into a hard error. Rollback of an added file deletes it; the content it replaced is not kept. `--force` overrides `--strict` and additionally skips missing files. Vendor staging is unaffected (it always auto-overwrites into its private stage). ## Per-subcommand arguments @@ -114,7 +114,9 @@ Each matching package instance is spliced, including scoped, quoted and nested-p For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLockfile: true` (created with a root-only `packages:` scaffold, or appended while preserving user bytes). pnpm >=11 requires this to accept hosted URLs; it disables registry re-verification for the whole lock, while sha512 tarball integrity remains enforced. The write (edit kind `redirect_pnpm_workspace_trust`) respects `--dry-run`, skips legacy locks and Rush repos, preserves explicit user settings (an existing top-level key in any YAML spelling: quoted, `trustLockfile :`, with a trailing comment), and is disabled by `--no-trust-lockfile-config`. The key goes inside the document (before a `...` end marker); a file a line append would corrupt (a flow-style root, an indented root, several documents) is left untouched and the warning gives the manual recoveries. The vendored `overrides:` mirror in `pnpm-workspace.yaml` reads keys the same way and refuses those shapes before writing. The `redirect_pnpm_trust_lockfile` warning explains manual configuration when required and clean reinstall guidance for all pnpm versions. Existing installs and warm stores can retain upstream files; use a clean install tree and empty store, then verify installed files with `socket-patch vex`. Neither a successful install nor a local VEX export guarantees hosted SBOM recognition or changes dashboard alert actions/counts. -**npm hosted-mode `allow-remote` contract**: npm >=12 defaults `allow-remote=none` and refuses (EALLOWREMOTE) every lockfile entry whose `resolved` tarball is not served by the configured registry — exactly what a hosted redirect writes into `package-lock.json` / `npm-shrinkwrap.json`. Whenever a run leaves a ROOT npm lock carrying a granted hosted artifact URL (spliced this run, or already redirected by an earlier one — a missed config heals on re-run), the CLI ensures `allow-remote=all` in the project-root `.npmrc`: the file is created holding exactly `allow-remote=all\n` when absent, otherwise one `allow-remote=all` line is spliced in after the last non-empty top-level line (before any ini `[section]` header), in the file's own line ending, with the BOM, CRLF and trailing-newline shape preserved. The write lands in `redirect.rewrittenFiles` (edit kind `redirect_npmrc_allow_remote`), respects `--dry-run` (nothing written; the warning says what would be — including for a vendored → hosted takeover the dry run only previews), and is disabled by `--no-npm-allow-remote-config` / `SOCKET_NO_NPM_ALLOW_REMOTE_CONFIG`. The `.npmrc` grammar is npm's own `ini` parser's (cross-checked against it): lines split on any run of `\r` / `\n` (a bare `\r` ends a line), only the exact key `allow-remote` counts after ini unquoting (npm ignores `allow_remote` / `ALLOW-REMOTE` in a `.npmrc`; such a line is left alone and the real key appended), comment lines are ignored, a `[section]` header is recognized only as npm does — on the UNTRIMMED line (an indented or BOM-prefixed `[sec]` is a plain top-level key) — and ends the top-level scope, quotes and inline comments are stripped, the LAST top-level assignment wins, and the value is case-sensitive. An explicit other value (`allow-remote=none` / `root` / anything but `all`) is RESPECTED and never rewritten — the pnpm `trustLockfile: false` precedent — in the project `.npmrc` AND in every other npm config layer npm would consult: an `npm_config_allow_remote` environment variable (any spelling npm normalizes; it beats every `.npmrc`, so a project write could not take effect), and — when the project file sets nothing — the user (`npm_config_userconfig` / `~/.npmrc`), global (`npm_config_globalconfig` / `/etc/npmrc`, prefix from `npm_config_prefix`, the user/builtin config, `PREFIX` or the `node` binary's install root) and builtin (npm's own `npmrc` beside the `node` binary: `/lib/node_modules/npm/npmrc`, `\node_modules\npm\npmrc` on Windows) config files — path values `${VAR}`-expanded and `~`-expanded like npm, env names case-insensitive on Windows, where a committed project line would silently override a machine / org policy. A symlinked, non-regular or unreadable `.npmrc`, or one with bare-`\r` line endings (npm splits on them, the line splice does not), is left untouched. Every variant emits the `redirect_npm_allow_remote` warning (written / would write / already set / explicit value respected — naming the project file, the env var, or the user/global/builtin config path — / opted out / unreadable or unsupported), always with the tradeoff: `allow-remote=all` lets npm install ANY url-resolved dependency, not just Socket's patched ones, while the per-entry sha512 integrity pins stay enforced; the remedy for the non-writing variants is `allow-remote=all` in `.npmrc` or `npm ci --allow-remote=all`. npm <=11 is unaffected (11 defaults to `all`, <=10 has no such setting). **Unwind (v5.0: no ledger)**: once `rollback`, `remove` or a hosted → vendored takeover has restored the last hosted entry of the root `package-lock.json` / `npm-shrinkwrap.json` to its upstream registry entry (see "Hosted unwind coverage"), a project `.npmrc` holding exactly `allow-remote=all\n` (the file hosted mode creates) is deleted; any other `.npmrc` that still has a top-level `allow-remote=all` line is left untouched and the `npm_allow_remote_left` warning says the line may be removed if nothing else needs it (v5 keeps no record of whether hosted mode added it, so it is never removed behind the user's back). The rewrite's stage file is created with the `.npmrc`'s own permission bits (a 0600 token-bearing file is never staged world-readable). **Vendored mode is unaffected**: its `file:.socket/vendor/…` resolutions are npm `file` specs, which npm gates by `allow-file` (default `all`), never `allow-remote` — verified by the real npm 12 vendored matrix. +**npm hosted-mode `allow-remote` contract**: npm >=12 defaults `allow-remote=none` and refuses (EALLOWREMOTE) every lockfile entry whose `resolved` tarball is not served by the configured registry — exactly what a hosted redirect writes into `package-lock.json` / `npm-shrinkwrap.json`. Whenever a run leaves a ROOT npm lock carrying a granted hosted artifact URL (spliced this run, or already redirected by an earlier one — a missed config heals on re-run), the CLI ensures `allow-remote=all` in the project-root `.npmrc`: the file is created holding exactly `allow-remote=all\n` when absent, otherwise one `allow-remote=all` line is spliced in after the last non-empty top-level line (before any ini `[section]` header), in the file's own line ending, with the BOM, CRLF and trailing-newline shape preserved. The write lands in `redirect.rewrittenFiles` (edit kind `redirect_npmrc_allow_remote`), respects `--dry-run` (nothing written; the warning says what would be — including for a vendored → hosted takeover the dry run only previews), and is disabled by `--no-npm-allow-remote-config` / `SOCKET_NO_NPM_ALLOW_REMOTE_CONFIG`. The `.npmrc` grammar is npm's own `ini` parser's (cross-checked against it): lines split on any run of `\r` / `\n` (a bare `\r` ends a line), only the exact key `allow-remote` counts after ini unquoting (npm ignores `allow_remote` / `ALLOW-REMOTE` in a `.npmrc`; such a line is left alone and the real key appended), comment lines are ignored, a `[section]` header is recognized only as npm does — on the UNTRIMMED line (an indented or BOM-prefixed `[sec]` is a plain top-level key) — and ends the top-level scope, quotes and inline comments are stripped, the LAST top-level assignment wins, and the value is case-sensitive. An explicit other value (`allow-remote=none` / `root` / anything but `all`) is RESPECTED and never rewritten — the pnpm `trustLockfile: false` precedent — in the project `.npmrc` AND in every other npm config layer npm would consult: an `npm_config_allow_remote` environment variable (any spelling npm normalizes; it beats every `.npmrc`, so a project write could not take effect), and — when the project file sets nothing — the user (`npm_config_userconfig` / `~/.npmrc`), global (`npm_config_globalconfig` / `/etc/npmrc`, prefix from `npm_config_prefix`, the user/builtin config, `PREFIX` or the `node` binary's install root) and builtin (npm's own `npmrc` beside the `node` binary: `/lib/node_modules/npm/npmrc`, `\node_modules\npm\npmrc` on Windows) config files — path values `${VAR}`-expanded and `~`-expanded like npm, env names case-insensitive on Windows, where a committed project line would silently override a machine / org policy. A symlinked, non-regular or unreadable `.npmrc`, or one with bare-`\r` line endings (npm splits on them, the line splice does not), is left untouched. Every variant emits the `redirect_npm_allow_remote` warning (written / would write / already set / explicit value respected — naming the project file, the env var, or the user/global/builtin config path — / opted out / unreadable or unsupported), always with the tradeoff: `allow-remote=all` lets npm install ANY url-resolved dependency, not just Socket's patched ones, while the per-entry sha512 integrity pins stay enforced; the remedy for the non-writing variants is `allow-remote=all` in `.npmrc` or `npm ci --allow-remote=all`. npm <=11 is unaffected (11 defaults to `all`, <=10 has no such setting). **Unwind (v5.0: no ledger)**: once `rollback`, `remove` or a hosted → vendored takeover has restored the last hosted entry of the root `package-lock.json` / `npm-shrinkwrap.json` to its upstream registry entry (see "Hosted unwind coverage"), a project `.npmrc` holding exactly `allow-remote=all\n` (the file hosted mode creates) is deleted; any other `.npmrc` that still has a top-level `allow-remote=all` line is left untouched and the `npm_allow_remote_left` warning says the line may be removed if nothing else needs it (v5 keeps no record of whether hosted mode added it, so it is never removed behind the user's back). The rewrite's stage file is created with the `.npmrc`'s own permission bits (a 0600 token-bearing file is never staged world-readable). **Vendored mode is unaffected by `allow-remote`**: its `file:.socket/vendor/…` resolutions are npm `file` specs, which npm >= 11.14 gates by `allow-file` (default `all`), never `allow-remote` — verified by the real npm 12 vendored matrix. **npm vendored `allow-file` contract (#969)**: `allow-file=none` refuses every `file:` dependency, and any other non-`all` value (`root`) admits one only when the project root or a workspace declares it and resolves to that very lock node (arborist's `_isRoot`); otherwise every `npm ci` / `npm install` fails EALLOWFILE. The effective value is read from the same npm config layers as `allow-remote` (an `npm_config_allow_file` environment variable, then the project `.npmrc`, then the user / global / builtin config files). When it refuses a `package-lock.json` / `npm-shrinkwrap.json` instance of a vendored `name@version` (the set vendoring rewires), the vendored run (`vendor`, `scan` / `get --mode vendored`, dry runs and in-sync re-runs included) still vendors, leaves the setting untouched, and records the `vendor_npm_allow_file` advisory naming the setting's source, the refused lock entries and the remedy (`allow-file=all` in the project `.npmrc`, unsetting the env var, or `npm ci --allow-file=all`); `vendor --check` fails that entry `vendor_check_failed` with the same reason. Other npm-family flavors (yarn, pnpm, bun, vlt) do not read `allow-file` and are not checked. + +**npm hosted-mode `replace-registry-host` contract**: npm >=8 rewrites the origin of a lock's `resolved` URL to the configured registry when its `replace-registry-host` setting is `always`, or equals the URL's hostname (`npmjs`, the default, stands for `registry.npmjs.org`; `never` matches nothing). A hosted pin rewritten that way is fetched from `/patch/npm/…` and every `npm ci` / `npm install` fails E404 (closed: never unpatched bytes; a warm npm cache can hide it locally). Whenever a run leaves a root npm lock carrying a granted hosted artifact URL (the same trigger as the `allow-remote` contract, so an "already on hosted patches" re-run reports it too), the CLI resolves `replace-registry-host` across npm's layers — an `npm_config_replace_registry_host` environment variable (any spelling npm normalizes), then the project `.npmrc`, then the user / global / builtin config files (located exactly like the `allow-remote` layers) — and, when the effective value rewrites a pinned host, emits the `redirect_npm_replace_registry_host` warning naming the layer (env var / project file / config path) and the remedies: `replace-registry-host=npmjs` in the project `.npmrc` (or unsetting the env var), or vendored mode, whose `file:` resolutions npm never rewrites. The setting is never rewritten; the redirect itself still lands and the exit status is unchanged. `redirect_pnpm_no_lockfile` names pnpm when installer markers exist without a lock; `redirect_pnpm_entry_vendored` identifies a vendored entry instead of reporting it missing. Supported `shrinkwrap.yaml` files are writable lockfiles, not read-only markers. @@ -146,7 +148,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc **Vendored write durability (v5.0)**: every write is atomic (stage + rename), but only the durable commit points — lockfiles, `go.mod`/`go.sum`, `pom.xml`, `nuget.config`, `package.json`, `pnpm-workspace.yaml`, `.cargo/config.toml`, the Python/Ruby manifests, `.socket/vendor/state.json` and `redirect-state.json` — are fsynced on write. The content-verified artifacts under `.socket/vendor///` (patched copies, packed/rebuilt archives and sidecars, markers) are written without an fsync and made durable by one barrier (file + directory fsync, one `F_FULLFSYNC` per device on macOS) ahead of the next commit point — and, for an artifact rebuilt in place that no commit point follows, at the end of the vendored run's commit and when the command releases the apply lock — so a crash can only lose an artifact that no durable commit point names yet, which the next run redownloads. -**Vendored group commit (v5.0)**: `vendor`, `scan --mode vendored` and `get --mode vendored` capture every lockfile / manifest / config edit and every ledger save of the run in memory (reads inside the run see them) and commit them ONCE after the per-package loop — including the packages that succeeded in a run where others failed, so a completed run leaves the same files per-package commits would. Captured: every file under the project root outside `.socket/`, plus `.socket/vendor/state.json` and `.socket/vendor/redirect-state.json`; artifacts are written directly (see the durability note). A multi-file commit goes through a roll-forward journal, `.socket/vendor/.commit-journal.json` (the new bytes of every changed file, plus the bytes each replaces and their sha256; deleted once the commit completes). **Crash semantics**: before the journal is durable, nothing is committed — the lockfiles and ledgers are the pre-run ones and the run's artifacts are unreferenced orphans; after it, the next command that takes the apply lock replays the journal before reading anything (files already at their new bytes are left alone), so a locked command never observes a half-committed run. A journal that matches neither side of some file (edited by hand since the crash) is renamed to `.socket/vendor/.commit-journal.set-aside-.json` (keeping every file's pre-commit bytes) and stderr says what was done (`Warning: an interrupted vendored run's commit could not be finished as written: …`): the edited files are never written over; when they all still carry the commit's own lines the rest of the commit is finished around them, when none of them does the files the crash had already replaced are put back to their pre-commit bytes, and otherwise nothing is applied. A journal that is unreadable, names a path outside the lockfiles and ledgers, or would write through a symbolic link is set aside with nothing applied. A replay that fails on I/O keeps the journal and fails the lock acquire (`lock_io`, naming the journal). Read-only commands that take no lock (`vex`, `list`) may observe the interrupted state until then. A re-vendor under a newer uuid removes the replaced uuid's dir only after the commit (its `vendor_stale_artifact_removed` event follows the run's per-package events), and a golang takeover removes the `.socket/go-patches/` copy only after the commit that repoints `go.mod`. A commit never renames over a symbolic link: when a changed file is a symlink, the whole commit is refused before anything is written, with the top-level error `redirect_symlinked_file_unsupported` (exit 1; a `--dry-run` predicts it with a `vendor_would_refuse_symlinked_file` advisory). A commit write failure is the top-level error `vendor_commit_failed` (exit 1; the pre-run lockfiles and ledger stay — unless putting back the files already replaced failed too, in which case the journal is kept and the next locked command finishes the commit). `repair`, `vendor --revert` and `rollback` still save per entry. +**Vendored group commit (v5.0)**: `vendor`, `scan --mode vendored` and `get --mode vendored` capture every lockfile / manifest / config edit and every ledger save of the run in memory (reads inside the run see them) and commit them ONCE after the per-package loop — including the packages that succeeded in a run where others failed, so a completed run leaves the same files per-package commits would. Captured: every file under the project root outside `.socket/`, plus `.socket/vendor/state.json` and `.socket/vendor/redirect-state.json`; artifacts are written directly (see the durability note). A multi-file commit goes through a roll-forward journal, `.socket/vendor/.commit-journal.json` (the new bytes of every changed file, plus the bytes each replaces and their sha256; deleted once the commit completes). **Crash semantics**: before the journal is durable, nothing is committed — the lockfiles and ledgers are the pre-run ones and the run's artifacts are unreferenced orphans; after it, the next command that takes the apply lock replays the journal before reading anything (files already at their new bytes are left alone), so a locked command never observes a half-committed run. A journal that matches neither side of some file (edited by hand since the crash) is renamed to `.socket/vendor/.commit-journal.set-aside-.json` (keeping every file's pre-commit bytes) and stderr says what was done (`Warning: an interrupted vendored run's commit could not be finished as written: …`): the edited files are never written over; when they all still carry the commit's own lines the rest of the commit is finished around them, when none of them does the files the crash had already replaced are put back to their pre-commit bytes, and otherwise nothing is applied. A journal that is unreadable, names a path outside the lockfiles and ledgers, or would write through a symbolic link is set aside with nothing applied. A replay that fails on I/O keeps the journal and fails the lock acquire (`lock_io`, naming the journal). Read-only commands that take no lock (`vex`, `list`) may observe the interrupted state until then. A re-vendor under a newer uuid removes the replaced uuid's dir only after the commit (its `vendor_stale_artifact_removed` event follows the run's per-package events), and a golang takeover removes the `.socket/go-patches/` copy only after the commit that repoints `go.mod`. A commit never renames over a symbolic link: when a changed file is a symlink, the whole commit is refused before anything is written, with the top-level error `redirect_symlinked_file_unsupported` (exit 1; a `--dry-run` predicts it with a `vendor_would_refuse_symlinked_file` advisory) — the artifact dirs the run's loop added under `.socket/vendor/` are removed, and every package it vendored is reported as a `failed` event with that code instead of `applied` (its per-package `skipped` advisories from the same run, such as `vendor_prebuilt_downloaded`, are dropped). A commit write failure is the top-level error `vendor_commit_failed` (exit 1; the pre-run lockfiles and ledger stay, the artifact dirs the run added are removed and its vendored packages are `failed` events with that code, the same way — unless putting back the files already replaced failed too, in which case the journal is kept, the next locked command finishes the commit, and the per-package events stand). `repair`, `vendor --revert` and `rollback` still save per entry. `scan --sync` is sugar for `--mode agent --prune` — the canonical single-flag agent-mode bot invocation. `scan --json --sync` discovers, applies, and reconciles state in one pass. @@ -157,7 +159,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc * **Hosted and vendored mode (bare `scan` included) — project directories** (`run_project_dirs`). Each PATH is a directory, or a glob (`*?[`) matching directories, relative to `--cwd`; the set is sorted and deduplicated, and each directory is scanned on its own exactly as if it were `--cwd` (its own lockfiles, ledgers and `.socket/`). With more than one directory, each run is headed `== ==` on stdout (unless `--silent`), and the exit code is the worst of the runs. Usage errors (exit 2, stderr only, before any scan): a PATH that is not a directory (`` `X` is not a directory``), a glob matching no directory (`` `X` matches no directory``), an invalid glob, and `--json` with more than one directory (`--json takes one project directory (N given); run one scan per directory`), so stdout stays one document. Likewise `--vex` with more than one directory (`--vex takes one project directory (N given); run one scan per directory`): the one output path would be overwritten by each run. * **Agent mode (and a mode-less `--prune`/`--global` report) — installed-path globs** scoping DISCOVERY at the **purl level**: a package is in scope iff ANY of its crawled installed copies sits under a matching path, and a selected package is then handled with ALL its copies (scoping selects which packages are considered, never which copies). Glob semantics (shared with `rollback`'s path targets, `src/path_scope.rs`): Unix-shell globs with `require_literal_separator` — `*`/`?` never cross a `/`, `**` spans directories; a pattern matching any **ancestor** directory of the copy path also matches, so a bare `scan packages/foo` scopes the whole subtree without `/**`; relative patterns match against the copy path relativized to `--cwd`, absolute patterns against the absolute path (the ONLY way to reach paths outside the project tree, e.g. `--global` stores — a relative pattern never matches outside `--cwd`); leading `./` and trailing `/` are normalized away, matching is purely textual (no filesystem access or symlink resolution), case-sensitive except on Windows (whose filesystems are not); an unparseable or empty pattern is a usage error (exit 2). **The prune universe is never narrowed**: the path filter is applied strictly AFTER the `scanned_purls` capture (and after `--ecosystems`), so `scan PATHS --prune` prunes exactly what an unscoped `scan --prune` would — a scoped scan can never treat an out-of-scope package as uninstalled (the same fail-safe as the `--ecosystems` filter). Lockfile-only and vendor-ledger supplement records have no installed path and are EXCLUDED from a path-scoped scan, surfaced as one run-level `path_scope_excluded_supplements` warning carrying the count. A scope matching nothing is a normal empty scan — exit 0, zero packages, **no GC** (the zero-package early return fires before any GC). `PATHS` combine with `--mode agent`/`--sync`/`--prune`/`--global`. Every scan JSON shape (success, zero-package, and error alike) carries an always-present `paths` key echoing the patterns verbatim (empty array when unscoped; a hosted/vendored per-directory run is unscoped, so it is `[]`). One-sentence duality rule: **a target that selects nothing is an error on `rollback` (exit 1) and an empty scan on an agent-mode `scan` (exit 0)**. -`scan --mode vendored` swaps the in-place apply for the vendor pipeline: discover → download the selected patch records **into memory** (no manifest write) → vendor every selected dependency via the same engine as the `vendor` command (under the same lock). Vendored mode is **manifest-free (v5.0)**: `.socket/manifest.json` is never written or read by a vendored run; each ledger entry carries `detached: true` plus an embedded copy of the patch record (`record`) as its verification source, and the run's footprint is `.socket/vendor/**` only. The vendor step's scope is what discovery selected — the former "whole manifest is vendored" re-vendor on an empty discovery is retired (`repair` verifies and redownloads committed vendored state; `scan --prune` reconciles ledger entries whose dependency left the lockfile). The vendor-ledger discovery supplement (the fresh-clone rule: a ledger entry with no installed copy stays discoverable because its committed artifact IS the dependency) holds only while the lockfile still resolves through that artifact: an entry the lockfile in-use probe (the one `--prune` reverts by) proves unwired, because the dependency was upgraded or removed, is NOT discovered and so is never re-vendored. A run without a non-hosted `--prune` reports it through the run-level `vendor_ledger_entry_unwired` warning; a `--prune` run reverts it in its GC and exits 0. That GC runs even when the crawl found no packages, as its vendored half alone (the manifest prune stays skipped there). A package the ledger holds at an older patch uuid is still **re-vendored automatically** when discovery selects the newer patch (its old uuid dir is removed — `vendor_stale_artifact_removed`); same-uuid re-runs reuse the embedded record, skip the patch-view fetch, and are `already_vendored` skips. **Legacy manifest-mode entries**: when a vendored run vendors a purl that also has a `.socket/manifest.json` record (a project vendored by a pre-5.0 binary, or by standalone `vendor` from an agent-mode manifest), that manifest record is dropped in the same run — the ledger becomes the owner (migration write); an emptied manifest is left as `{"patches": {}}`, never deleted. The migration is reported through the run-level `warnings[]` (stderr in human mode), never as a run error: `vendor_manifest_record_migrated` (`N manifest records moved to the vendor ledger (vendored mode is manifest-free): `) or `vendor_manifest_migration_failed` (the manifest or the ledger could not be read or rewritten; the legacy records were left in place) — so a corrupt `.socket/manifest.json` no longer fails a vendored run (standalone `vendor`, the one manifest-driven writer, still fails closed on it). With `--prune`, GC runs **after** the vendor step (the step never reads the manifest, and running the sweep last lets it reclaim what the run itself orphaned — a migrated legacy record's blobs, a superseded uuid dir). JSON output gains a `download` sub-object — the detached download envelope `{found, downloaded, skipped, failed, detached: true, patches: [{purl, uuid, action: "downloaded" | "skipped" | "failed", …}], warnings?}` (no `applied` field — nothing is applied in place; `detached: true` is pinned and always present; a `downloaded` record whose purl the ledger already holds at another uuid carries the additive `oldUuid` — the re-vendor the vendor step then performs — and its human `[fetch]` line reads ` (replacing )`) — and a `vendor` sub-object (a full vendor Envelope). Patch blobs are held in memory (see "Patch sources stay in memory" under the vendor contract). `--dry-run` previews per-patch `would_vendor` | `would_revendor` (+`oldUuid`) | `already_vendored` — plus, additive, `would_refuse` (+`errorCode`, `error`) for npm purls the wet run's Bun preflight (see the `get --mode vendored` bullet below) would refuse — without network downloads or disk writes; the preview never flips status or exit (the human path — `scan` and `get` alike, through one shared printer — prints `[would-refuse] (): ` lines behind the `--silent` gate). Interactive mode prompts "Download and vendor N patches?" (singular for one). +`scan --mode vendored` swaps the in-place apply for the vendor pipeline: discover → download the selected patch records **into memory** (no manifest write) → vendor every selected dependency via the same engine as the `vendor` command (under the same lock). Vendored mode is **manifest-free (v5.0)**: `.socket/manifest.json` is never written or read by a vendored run; each ledger entry carries `detached: true` plus an embedded copy of the patch record (`record`) as its verification source, and the run's footprint is `.socket/vendor/**` only. The vendor step's scope is what discovery selected — the former "whole manifest is vendored" re-vendor on an empty discovery is retired (`repair` verifies and redownloads committed vendored state; `scan --prune` reconciles ledger entries whose dependency left the lockfile). The vendor-ledger discovery supplement (the fresh-clone rule: a ledger entry with no installed copy stays discoverable because its committed artifact IS the dependency) holds only while the lockfile still resolves through that artifact: an entry the lockfile in-use probe (the one `--prune` reverts by) proves unwired, because the dependency was upgraded or removed, is NOT discovered and so is never re-vendored. A run without a non-hosted `--prune` reports it through the run-level `vendor_ledger_entry_unwired` warning; a `--prune` run reverts it in its GC and exits 0. That GC runs even when the crawl found no packages, as its vendored half alone (the manifest prune stays skipped there). A package the ledger holds at an older patch uuid is still **re-vendored automatically** when discovery selects the newer patch (its old uuid dir is removed — `vendor_stale_artifact_removed`) — unless the patch service has no prebuilt artifact for the newer patch yet (or at all): an npm-family package then keeps the older patch and is a `skipped` `vendor_prebuilt_pending` / `vendor_prebuilt_unavailable` event, not a failure (#954); same-uuid re-runs reuse the embedded record, skip the patch-view fetch, and are `already_vendored` skips. **Legacy manifest-mode entries**: when a vendored run vendors a purl that also has a `.socket/manifest.json` record (a project vendored by a pre-5.0 binary, or by standalone `vendor` from an agent-mode manifest), that manifest record is dropped in the same run — the ledger becomes the owner (migration write); an emptied manifest is left as `{"patches": {}}`, never deleted. The migration is reported through the run-level `warnings[]` (stderr in human mode), never as a run error: `vendor_manifest_record_migrated` (`N manifest records moved to the vendor ledger (vendored mode is manifest-free): `) or `vendor_manifest_migration_failed` (the manifest or the ledger could not be read or rewritten; the legacy records were left in place) — so a corrupt `.socket/manifest.json` no longer fails a vendored run (standalone `vendor`, the one manifest-driven writer, still fails closed on it). With `--prune`, GC runs **after** the vendor step (the step never reads the manifest, and running the sweep last lets it reclaim what the run itself orphaned — a migrated legacy record's blobs, a superseded uuid dir). JSON output gains a `download` sub-object — the detached download envelope `{found, downloaded, skipped, failed, detached: true, patches: [{purl, uuid, action: "downloaded" | "skipped" | "failed", …}], warnings?}` (no `applied` field — nothing is applied in place; `detached: true` is pinned and always present; a `downloaded` record whose purl the ledger already holds at another uuid carries the additive `oldUuid` — the re-vendor the vendor step then performs — and its human `[fetch]` line reads ` (replacing )`) — and a `vendor` sub-object (a full vendor Envelope). Patch blobs are held in memory (see "Patch sources stay in memory" under the vendor contract). `--dry-run` previews per-patch `would_vendor` | `would_revendor` (+`oldUuid`) | `already_vendored` — plus, additive, `would_refuse` (+`errorCode`, `error`) for npm purls the wet run's Bun preflight (see the `get --mode vendored` bullet below) would refuse — without network downloads or disk writes; the preview never flips status or exit (the human path — `scan` and `get` alike, through one shared printer — prints `[would-refuse] (): ` lines behind the `--silent` gate). Interactive mode prompts "Download and vendor N patches?" (singular for one). **Vendored entries and the rest of the CLI.** Because nothing is in the manifest, vendored patches are invisible to `apply` (nothing to apply in place) but fully visible to `list` (listed from the ledger, labeled `Mode: vendored (recorded in .socket/vendor/state.json)` in human mode, exit 0 on a vendored-only project), `vex` (attested from the embedded records while a lockfile still wires the artifact — see "Manifest-less VEX"), `repair` (health-checked and rebuilt from the ledger), and `scan --prune` (lockfile-driven reconcile). They are exempt from standalone `vendor`'s manifest reconcile (`reconcile_dropped` never touches `detached` entries) and exit via `remove ` (which reverts them), `vendor --revert`, or `rollback`, whose vendored leg reverts every in-scope ledger entry (unscoped and identifier-scoped runs; path-scoped runs reach them only when an installed copy matches). @@ -165,7 +167,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc **Attribution gate (v5.0).** A hosted run never leaves wiring that lockfile discovery calls contested: before the rewrite writes any lockfile, the same discovery `vex`, `list`, `rollback`, `remove` and `vendor` read runs over the project as the rewrite would leave it. A candidate whose pin discovery reads but cannot attribute to one package version (a requirements `-r` include resolving the same version from the registry beside a rewired `Pipfile.lock`, #567; a Maven pin in a ``, #260) is left out of the rewrite and reported in `redirect.skipped[]` as `redirect_unattributable` (nothing written for it; exit code unchanged). Unchanged: a pin in a file discovery does not read (a pre-2.6 bundler `Gemfile`, locked by the next `bundle install`) keeps the rewriter's verdict, and so does a deliberate partial redirect the run already reports (a bundled or `bun patch`-ed copy left on the registry, a dep withheld from the vlt rewrite while a sibling lock takes it), and so does a vendored→hosted takeover: its vendored wiring is reverted in the run's staged overlay before the rewrite plans, and it is redirected when the rewriters pin it (otherwise it is retracted and stays vendored), in a `--dry-run` preview the same. A lockless NuGet / Cargo pin (an exclusive Socket source mapping without `packages.lock.json`, a Cargo registry pin without `Cargo.lock`) is still written, with a `redirect_pin_lockless` warning: no lockfile records its version, so `vex` cannot attest it and `rollback` / `remove` / `vendor` refuse it until the lockfile exists (whether such pins should be written at all is an open decision). The rollout's recorded view uses the same discovery: a uuid a file merely mentions (a stale `package.json` field, an inactive `pdm.lock`, a comment) is not a pin and does not count as already patched. -The rewriter reads a fixed set of candidate files from the project root: the npm-family locks (`package-lock.json`, `npm-shrinkwrap.json`, `pnpm-lock.yaml`, `shrinkwrap.yaml`, `yarn.lock`, plus `.yarnrc.yml` for the berry cache-config gate, `bun.lock` / `bun.lockb`, and `vlt-lock.json` with `vlt.json` and `node_modules/.vlt-lock.json` read only), `requirements.txt` / `uv.lock` / `Pipfile.lock` (pipfile-spec 6; see the Pipenv section below) / `poetry.lock` (every Poetry lock generation from 1.0 on — the 0.12 `[metadata.hashes]` layout is refused because that installer ignores URL sources; a Poetry < 1.4 writer additionally gets `redirect_poetry_stale_install_risk`, see `docs/testing/poetry-compatibility.md`) / `pdm.lock` (PDM lock formats `2` and `4.3`–`4.5.1`; the identity-losing `3.1` / `4.0`–`4.2` formats and unknown future formats are refused with `redirect_pdm_refused`, and a lock-format-`2` writer additionally gets `redirect_pdm_legacy_sync_required`, see `docs/testing/pdm-compatibility.md`; when `uv.lock` or `poetry.lock` sits beside it they drive and `pdm.lock` is left alone), `Cargo.toml` / `Cargo.lock` / `.cargo/config.toml` (plus the legacy extensionless `.cargo/config` — cargo reads that spelling in preference when both exist, so the managed `[registries.…]` block is written into whichever one is present; **cargo also reads every workspace-member manifest** — the `[workspace] members` globs minus `exclude` — and every in-root path-dependency manifest, recursively, reached without crossing a symbolic link and never under `.socket/`, and pins the crate in each one that declares it, so those `/Cargo.toml` files can appear in `rewrittenFiles`. A crate is redirected only when every declaration pins and every other `Cargo.lock` package depending on it is a planned member: one a registry or git crate — or a path package outside the root or behind a link — also depends on is refused `redirect_cargo_transitive_dependents` (a pin reaches only the declarations it sits on), a crate no manifest declares keeps `redirect_cargo_toml_dep_not_found` with a transitive-only detail naming `--mode vendored`, a crate every declaration of which requires another version (no requirement accepts the patched version) is refused `redirect_cargo_toml_dep_unrewritable`, and so is a requirement that also matches another locked version of the crate — each a transactional skip, never recorded or attested. With NO `Cargo.lock` there is no resolved graph to ask, so the dependents question is answered from the manifests instead: a crate declared beside any other dependency — anything but a path dependency on a manifest this run also pins, or a `workspace = true` inheritor of a table it scans — or beside a workspace member this run did not read (a `members` glob, or a member outside the project or behind a symbolic link, which member discovery drops) is refused `redirect_cargo_lockless_dependents`, whose detail names the remedies (commit a lockfile, or `--mode vendored`); a project whose only dependency is the patched crate has nothing that could pull it in and still redirects. All-CRLF manifests, locks and configs are rewritten with CRLF kept (mixed endings keep refusing where the grammar does not match), and `remove` / rollback match the recorded fragments across a later CRLF↔LF checkout conversion), `composer.lock`, `nuget.config` / `packages.lock.json`, `Gemfile` / `Gemfile.lock`, `pom.xml` (+ `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` for maven Trusted Checksums merge, and, for a Gradle build, every settings, build, `buildSrc`, included-build, applied and plugin-source script, version catalog and lock file the script graph reaches, plus `gradle/verification-metadata.xml`, `gradle/wrapper/gradle-wrapper.properties` and the owned `.socket/gradle/` files), and the sbt build files (`socket-patch.sbt`, `socket-patch-vendor.sbt`, `build.sbt`, `project/build.properties`, `.sbtopts`, `.jvmopts`; `build.sbt.lock` and the Mill / scala-cli build files `build.mill`, `build.mill.yaml`, `build.sc`, `.mill-version`, `project.scala` for their presence only) — read, never edited; `socket-patch.sbt` is the only sbt file hosted mode writes (see **Hosted sbt** below). **npm-family flavor coverage**: package-lock / npm-shrinkwrap, pnpm (root OR any nested `*/pnpm-lock.yaml`), yarn classic (a yarn 2+ install migrates a v1 `yarn.lock` and drops its pins, so a run whose v1 lock carries a hosted pin warns `redirect_yarn_classic_berry_migration_risk` — the hosted twin of the vendored `yarn_classic_berry_migration_risk` — unless the root `package.json`, read as advisory input, declares `"packageManager": "yarn@1…"`), **yarn berry** (the pin yarn writes for a root `resolutions` entry: the root `package.json` — edited only beside a berry `yarn.lock` — gains one `"@npm:": ""` selector per locked range (`redirect_yarn_berry_resolution` edits), and only that `yarn.lock` entry is re-keyed `"@"` with the same `resolution:` + `yarnBerry10c0` checksum (`redirect_yarn_berry_entry`), moved to yarn's key order; never an `npm:` locator, whose fetcher sends npm registry auth to the patch host, nor a tarball locator under an `npm:` key, which hardened mode rejects (YN0078). An older release's `npm:::__archiveUrl=` pin is still recognized and is re-pinned on the next run; rollback rebuilds the key from the selectors and drops them. Refused, nothing written: a user-authored `resolutions` entry for the package `redirect_yarn_berry_resolutions_conflict`, no root manifest `redirect_yarn_berry_manifest_missing`, a builtin `patch:` entry wrapping the same descriptor `redirect_yarn_berry_shared_descriptor`, an artifact URL yarn cannot fetch as a tarball `redirect_yarn_berry_artifact_url_unsupported`; cacheKey `10c0` and `.yarnrc.yml compressionLevel 0` gated by `redirect_yarn_berry_cache_unsupported`), and **bun** (text `bun.lock` lockfileVersion 0, 1 or 2 — 0 is the `--save-text-lockfile` opt-in lock of Bun 1.1.39–1.1.45, 1 the 1.2–1.3 default, 2 the 1.4+ default; all three emit one `packages` grammar, so the registry 4-tuple → URL 3-tuple rewrite is version-independent and the lock's own version line is kept. Any other or missing version, or a `packages` section outside bun's single-line grammar, is refused `redirect_bun_lock_unsupported` — the detail is the shared version gate's text (a newer version: update socket-patch, re-locking would reproduce it; no integer: re-lock with Bun ≥ 1.2), identical to the vendored refusal. A version-0 lock holding `workspace:` packages is refused `redirect_bun_workspace_unsupported` (its 2-tuple workspace grammar cannot keep the hosted tuple through a frozen install); the remedy is to delete `bun.lock` and re-run `bun install` with Bun ≥ 1.2, which writes lockfileVersion 1 (accepted). A plain in-place `bun install` bumps the version only when a workspace depends on another workspace (e.g. root → member — the shape the matrix measured); otherwise Bun 1.2.0 keeps version 0 and Bun 1.2.23+ fail to resolve, so the in-place bump is not the documented remedy. Bun lock version, grammar and workspace compatibility are checked before a vendored takeover, including during dry-run: these refusals preserve the existing lock, artifact and vendor ledger. Version-1 and version-2 workspace locks are rewritten, nested versions included. A granted dep with no rewritable entry warns `redirect_bun_entry_not_found`, a grant without a sha512 `redirect_bun_missing_sha512`; a CRLF lock keeps `\r\n` on the rewritten line, and a hosted URL left by an earlier grant of the same `name@version` is re-pinned in place. **Digest-less re-saves (Bun 1.1.39–1.3.9)**: every text-lock Bun below 1.3.10 re-saves a URL tuple WITHOUT its `sha512` whenever the lock is re-saved for another reason (`bun add`, `bun install` after a package.json or workspace change), leaving the 2-tuple `["name@", {meta}]` — the spec Bun installs from is intact. The CLI treats that spelling as its own wiring: a repeat hosted run counts the dep as redirected (no `redirect_bun_entry_not_found`) and HEALS the line back to the 3-tuple with the current `sha512`, recording the heal as a further `redirect_bun_lock_package` edit whose `original` is the 2-tuple (a stale URL is re-pinned from either spelling); `rollback`, scoped `rollback ` / `remove ` and the vendored takeover accept the digest-less spelling of a recorded `new` line (same key, spec and meta, only the trailing `"sha512-…"` missing) and restore the recorded original over it, so the chain always unwinds to the pristine registry line. Anything else — another uuid/token, another version, a re-laid meta object — is still drift. **Native `bun.lockb`**: when no text `bun.lock` exists, binary format versions 1, 2 and 3 are read and rewritten directly. Socket Patch does not invoke Bun or convert the project to a text lockfile. Exact matching package records are rewritten to hosted tarballs with the granted integrity, preserving dependency resolution IDs, workspace/dependency topology and unrelated package metadata; binary pointers and the package metadata hash are updated. Per-package `redirect_bun_lockb_package` snapshots support scoped rollback, repeat runs, superseding grants and hosted ↔ vendored takeover. A regular binary lock is discoverable even with no Bun runtime or `node_modules`; a dry run previews the same binary edits without writing them. A malformed, unreadable, unsupported or unverified binary structure is `redirect_bun_lockb_invalid` (exit 0, `redirected: 0`), and it refuses the npm rewrite before any takeover or sibling npm-family lock mutation. A symlinked binary write target is `redirect_symlinked_file_unsupported` (exit 1, including dry-run). `bun.lock` wins when both spellings exist. Binary-only projects do not receive `redirect_npm_no_lockfile`. Measured boundaries and the real-Bun matrix: `docs/testing/bun-compatibility.md`), and **vlt** (`vlt-lock.json` without `lockfileVersion`, `0` or `1`; see the vlt hosted-mode contract below). **Rush monorepos**: when `rush.json` is present the rewriter also reads `common/config/rush/pnpm-lock.yaml` and each `common/config/subspaces//pnpm-lock.yaml` (sorted for determinism) under their repo-relative keys and repoints them in place; editing them emits `redirect_rush_repo_state_stale` when `common/config/rush/repo-state.json` exists (the `pnpmShrinkwrapHash` desync is refreshed by `rush update`, which the redirect survives). **maven** is fail-closed via version suffixing: a `mavenSuffixedVersion` + `mavenPomSha256` override pins the Socket-only `-socket.` by rewriting the literal `` (`redirect_maven_dep_version`) or adding a `` entry (`redirect_maven_dep_management_added`), plus optional Trusted Checksums (`redirect_maven_trusted_checksums`, conflicts as `redirect_maven_trusted_checksums_conflict`; when `.mvn/wrapper/maven-wrapper.properties` pins a Maven older than 3.9.4, which ignores those files, the additive warning `redirect_maven_trusted_checksums_unenforced`); a `${property}` version is refused (`redirect_maven_dep_unpinned`), a non-matching literal skipped (`redirect_maven_dep_version_mismatch`), and an override without a suffixed version falls back to same-GAV repository injection (`redirect_maven_same_gav_fallback`, NOT fail-closed). **gradle** (v5.0) is automated wiring, no longer a pasted snippet: the owned settings script `.socket/gradle/socket-patch.hosted.settings.gradle` with its index `.socket/gradle/hosted-index.tsv`, one apply line per build's settings file, every lock entry of the GA moved to the suffixed version, and the suffixed component in an existing `gradle/verification-metadata.xml`. A refused dep writes nothing and keeps `redirect_gradle_manual_snippet` as its fallback; same-GAV grants are refused (`redirect_gradle_same_gav_unsupported`). Rules, refusals and codes: [Gradle builds](#gradle-builds-v50). +The rewriter reads a fixed set of candidate files from the project root: the npm-family locks (`package-lock.json`, `npm-shrinkwrap.json`, `pnpm-lock.yaml`, `shrinkwrap.yaml`, `yarn.lock`, plus `.yarnrc.yml` for the berry cache-config gate, `bun.lock` / `bun.lockb`, and `vlt-lock.json` with `vlt.json` and `node_modules/.vlt-lock.json` read only), `requirements.txt` / `uv.lock` / `Pipfile.lock` (pipfile-spec 6; see the Pipenv section below) / `poetry.lock` (every Poetry lock generation from 1.0 on — the 0.12 `[metadata.hashes]` layout is refused because that installer ignores URL sources; a Poetry < 1.4 writer additionally gets `redirect_poetry_stale_install_risk`, see `docs/testing/poetry-compatibility.md`) / `pdm.lock` (PDM lock formats `2` and `4.3`–`4.5.1`; the identity-losing `3.1` / `4.0`–`4.2` formats and unknown future formats are refused with `redirect_pdm_refused`, and a lock-format-`2` writer additionally gets `redirect_pdm_legacy_sync_required`, see `docs/testing/pdm-compatibility.md`; when `uv.lock` or `poetry.lock` sits beside it they drive and `pdm.lock` is left alone), `Cargo.toml` / `Cargo.lock` / `.cargo/config.toml` (plus the legacy extensionless `.cargo/config` — cargo reads that spelling in preference when both exist, so the managed `[registries.…]` block is written into whichever one is present; **cargo also reads every workspace-member manifest** — the `[workspace] members` globs minus `exclude` — and every in-root path-dependency manifest, recursively, reached without crossing a symbolic link and never under `.socket/`, and pins the crate in each one that declares it, so those `/Cargo.toml` files can appear in `rewrittenFiles`. A crate is redirected only when every declaration pins and every other `Cargo.lock` package depending on it is a planned member: one a registry or git crate — or a path package outside the root or behind a link — also depends on is refused `redirect_cargo_transitive_dependents` (a pin reaches only the declarations it sits on), a crate no manifest declares keeps `redirect_cargo_toml_dep_not_found` with a transitive-only detail naming `--mode vendored`, a crate every declaration of which requires another version (no requirement accepts the patched version) is refused `redirect_cargo_toml_dep_unrewritable`, and so is a requirement that also matches another locked version of the crate — each a transactional skip, never recorded or attested. With NO `Cargo.lock` there is no resolved graph to ask, so the dependents question is answered from the manifests instead: a crate declared beside any other dependency — anything but a path dependency on a manifest this run also pins, or a `workspace = true` inheritor of a table it scans — or beside a workspace member this run did not read (a `members` glob, or a member outside the project or behind a symbolic link, which member discovery drops) is refused `redirect_cargo_lockless_dependents`, whose detail names the remedies (commit a lockfile, or `--mode vendored`); a project whose only dependency is the patched crate has nothing that could pull it in and still redirects. All-CRLF manifests, locks and configs are rewritten with CRLF kept (mixed endings keep refusing where the grammar does not match), and `remove` / rollback match the recorded fragments across a later CRLF↔LF checkout conversion), `composer.lock`, `nuget.config` / `packages.lock.json`, `Gemfile` / `Gemfile.lock`, `pom.xml` (+ `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` for maven Trusted Checksums merge, and, for a Gradle build, every settings, build, `buildSrc`, included-build, applied and plugin-source script, version catalog and lock file the script graph reaches, plus `gradle/verification-metadata.xml`, `gradle/wrapper/gradle-wrapper.properties` and the owned `.socket/gradle/` files), and the sbt build files (`socket-patch.sbt`, `socket-patch-vendor.sbt`, `build.sbt`, `project/build.properties`, `.sbtopts`, `.jvmopts`; `build.sbt.lock` and the Mill / scala-cli build files `build.mill`, `build.mill.yaml`, `build.sc`, `.mill-version`, `project.scala` for their presence only) — read, never edited; `socket-patch.sbt` is the only sbt file hosted mode writes (see **Hosted sbt** below). **npm-family flavor coverage**: package-lock / npm-shrinkwrap (a package the project patches itself with npm ≥ 12.1's native `npm patch` — a root `patchedDependencies` key, or the lock entry's `patched` record — is left on its registry entry in every npm lock, `redirect_npm_patched_dependency_skipped`), pnpm (root OR any nested `*/pnpm-lock.yaml`), yarn classic (a yarn 2+ install migrates a v1 `yarn.lock` and drops its pins, so a run whose v1 lock carries a hosted pin warns `redirect_yarn_classic_berry_migration_risk` — the hosted twin of the vendored `yarn_classic_berry_migration_risk` — unless the root `package.json`, read as advisory input, declares `"packageManager": "yarn@1…"`), **yarn berry** (the pin yarn writes for a root `resolutions` entry: the root `package.json` — edited only beside a berry `yarn.lock` — gains one `"@npm:": ""` selector per locked range (`redirect_yarn_berry_resolution` edits), and only that `yarn.lock` entry is re-keyed `"@"` with the same `resolution:` + `yarnBerry10c0` checksum (`redirect_yarn_berry_entry`), moved to yarn's key order; never an `npm:` locator, whose fetcher sends npm registry auth to the patch host, nor a tarball locator under an `npm:` key, which hardened mode rejects (YN0078). An older release's `npm:::__archiveUrl=` pin is still recognized and is re-pinned on the next run; rollback rebuilds the key from the selectors and drops them. Refused, nothing written: a user-authored `resolutions` entry for the package `redirect_yarn_berry_resolutions_conflict`, no root manifest `redirect_yarn_berry_manifest_missing`, a builtin `patch:` entry wrapping the same descriptor `redirect_yarn_berry_shared_descriptor`, an artifact URL yarn cannot fetch as a tarball `redirect_yarn_berry_artifact_url_unsupported`; cacheKey `10c0` and `.yarnrc.yml compressionLevel 0` gated by `redirect_yarn_berry_cache_unsupported`), and **bun** (text `bun.lock` lockfileVersion 0, 1 or 2 — 0 is the `--save-text-lockfile` opt-in lock of Bun 1.1.39–1.1.45, 1 the 1.2–1.3 default, 2 the 1.4+ default; all three emit one `packages` grammar, so the registry 4-tuple → URL 3-tuple rewrite is version-independent and the lock's own version line is kept. Any other or missing version, or a `packages` section outside bun's single-line grammar, is refused `redirect_bun_lock_unsupported` — the detail is the shared version gate's text (a newer version: update socket-patch, re-locking would reproduce it; no integer: re-lock with Bun ≥ 1.2), identical to the vendored refusal. A version-0 lock holding `workspace:` packages is refused `redirect_bun_workspace_unsupported` (its 2-tuple workspace grammar cannot keep the hosted tuple through a frozen install); the remedy is to delete `bun.lock` and re-run `bun install` with Bun ≥ 1.2, which writes lockfileVersion 1 (accepted). A plain in-place `bun install` bumps the version only when a workspace depends on another workspace (e.g. root → member — the shape the matrix measured); otherwise Bun 1.2.0 keeps version 0 and Bun 1.2.23+ fail to resolve, so the in-place bump is not the documented remedy. Bun lock version, grammar and workspace compatibility are checked before a vendored takeover, including during dry-run: these refusals preserve the existing lock, artifact and vendor ledger. Version-1 and version-2 workspace locks are rewritten, nested versions included. A granted dep with no rewritable entry warns `redirect_bun_entry_not_found`, a grant without a sha512 `redirect_bun_missing_sha512`; a CRLF lock keeps `\r\n` on the rewritten line, and a hosted URL left by an earlier grant of the same `name@version` is re-pinned in place. **Digest-less re-saves (Bun 1.1.39–1.3.9)**: every text-lock Bun below 1.3.10 re-saves a URL tuple WITHOUT its `sha512` whenever the lock is re-saved for another reason (`bun add`, `bun install` after a package.json or workspace change), leaving the 2-tuple `["name@", {meta}]` — the spec Bun installs from is intact. The CLI treats that spelling as its own wiring: a repeat hosted run counts the dep as redirected (no `redirect_bun_entry_not_found`) and HEALS the line back to the 3-tuple with the current `sha512`, recording the heal as a further `redirect_bun_lock_package` edit whose `original` is the 2-tuple (a stale URL is re-pinned from either spelling); `rollback`, scoped `rollback ` / `remove ` and the vendored takeover accept the digest-less spelling of a recorded `new` line (same key, spec and meta, only the trailing `"sha512-…"` missing) and restore the recorded original over it, so the chain always unwinds to the pristine registry line. Anything else — another uuid/token, another version, a re-laid meta object — is still drift. **Native `bun.lockb`**: when no text `bun.lock` exists, binary format versions 1, 2 and 3 are read and rewritten directly. Socket Patch does not invoke Bun or convert the project to a text lockfile. Exact matching package records are rewritten to hosted tarballs with the granted integrity, preserving dependency resolution IDs, workspace/dependency topology and unrelated package metadata; binary pointers and the package metadata hash are updated. Per-package `redirect_bun_lockb_package` snapshots support scoped rollback, repeat runs, superseding grants and hosted ↔ vendored takeover. A regular binary lock is discoverable even with no Bun runtime or `node_modules`; a dry run previews the same binary edits without writing them. A malformed, unreadable, unsupported or unverified binary structure is `redirect_bun_lockb_invalid` (exit 0, `redirected: 0`), and it refuses the npm rewrite before any takeover or sibling npm-family lock mutation. A symlinked binary write target is `redirect_symlinked_file_unsupported` (exit 1, including dry-run). `bun.lock` wins when both spellings exist. Binary-only projects do not receive `redirect_npm_no_lockfile`. Measured boundaries and the real-Bun matrix: `docs/testing/bun-compatibility.md`), and **vlt** (`vlt-lock.json` without `lockfileVersion`, `0` or `1`; see the vlt hosted-mode contract below). **Rush monorepos**: when `rush.json` is present the rewriter also reads `common/config/rush/pnpm-lock.yaml` and each `common/config/subspaces//pnpm-lock.yaml` (sorted for determinism) under their repo-relative keys and repoints them in place; editing them emits `redirect_rush_repo_state_stale` when `common/config/rush/repo-state.json` exists (the `pnpmShrinkwrapHash` desync is refreshed by `rush update`, which the redirect survives). **maven** is fail-closed via version suffixing: a `mavenSuffixedVersion` + `mavenPomSha256` override pins the Socket-only `-socket.` by rewriting the literal `` (`redirect_maven_dep_version`) or adding a `` entry (`redirect_maven_dep_management_added`), plus optional Trusted Checksums (`redirect_maven_trusted_checksums`, conflicts as `redirect_maven_trusted_checksums_conflict`; when `.mvn/wrapper/maven-wrapper.properties` pins a Maven older than 3.9.4, which ignores those files, the additive warning `redirect_maven_trusted_checksums_unenforced`); a `${property}` version is refused (`redirect_maven_dep_unpinned`), a non-matching literal skipped (`redirect_maven_dep_version_mismatch`), and an override without a suffixed version falls back to same-GAV repository injection (`redirect_maven_same_gav_fallback`, NOT fail-closed). **gradle** (v5.0) is automated wiring, no longer a pasted snippet: the owned settings script `.socket/gradle/socket-patch.hosted.settings.gradle` with its index `.socket/gradle/hosted-index.tsv`, one apply line per build's settings file, every lock entry of the GA moved to the suffixed version, and the suffixed component in an existing `gradle/verification-metadata.xml`. A refused dep writes nothing and keeps `redirect_gradle_manual_snippet` as its fallback; same-GAV grants are refused (`redirect_gradle_same_gav_unsupported`). Rules, refusals and codes: [Gradle builds](#gradle-builds-v50). **Hosted sbt (v5.0, additive)**: an sbt build root (`project/build.properties` naming an `sbt.version`, 0.13.18 or later) is wired through ONE generated root file, `socket-patch.sbt` — no user file is edited. It pins every granted Maven patch build-wide (a `ThisBuild` `dependencyOverrides +=` of the Socket-only `-socket.` version plus a `file:` resolver over `.socket/sbt-hosted/maven2/`, moved ahead of the default repositories on sbt 0.13 / 1.x so an unreachable one never blocks it offline), downloads the pinned pom and jar there on the first sbt load (sha256-checked, gitignored by the file itself), and installs a load-time verifier that fails `update` when any project resolves another version or a pinned artifact whose bytes are not pinned. Edits: `redirect_sbt_pin` (added), `redirect_sbt_pin_updated` (an existing row replaced: same GA and base under a new uuid, or the same uuid with new served values; `original` names the previous uuid and version), `redirect_sbt_pin_rechecked` (an existing row re-verified after the build's dependencies changed: its dependency digest is recorded anew, `original`/`new` are `{deps}`). The load-time verifier also fails `update` when a project declares a pinned GA at a version newer than the pin's base (the build-wide override would otherwise force it back down). A new pin is gated on sbt's own resolution records under `target/` (never the machine-wide cache): run-level stops wire nothing, warn once and exit 0 — `redirect_sbt_no_resolution_evidence` (none; run `sbt update` first; always the in-memory engine's answer), `redirect_sbt_resolution_incomplete` (a declared project left no evidence, or the project definitions cannot be read statically), `redirect_sbt_resolution_stale` (a build source is newer than some project's evidence: each project is dated by its own newest record, so a partial `sbt /update` does not vouch for the others). Per-patch refusals (never confirmed): `redirect_sbt_missing_override` (no `maven2` override or no suffixed version), `redirect_sbt_integrity_missing` (jar or pom sha256 missing), `redirect_sbt_unsafe_value` (a value unsafe in a Scala literal, or an index URL not naming the uuid), `redirect_sbt_version_conflict` (some project resolves another version, or a build source declares the GA newer than the patch's base), `redirect_sbt_override_conflict` (two patches for one GA in a run, or another base already pinned), `redirect_sbt_vendored_conflict` (the GA is pinned by `socket-patch-vendor.sbt`, or that file cannot be parsed — then every Maven patch), `redirect_sbt_owned_file_modified` / `redirect_sbt_owned_file_foreign` (`socket-patch.sbt` edited, or not socket-patch's — every Maven patch), `redirect_sbt_owned_file_unreadable` (a whole-run refusal: `socket-patch.sbt` is on disk but cannot be read as UTF-8 text, so writing it would replace it; nothing is written), `redirect_sbt_unsupported_version`, `redirect_sbt_build_root_unknown` (sbt files but no versioned build root — every Maven patch), `redirect_sbt_overrides_assignment` / `redirect_sbt_resolvers_assignment` (a build source reassigns `dependencyOverrides` / `resolvers` with `:=`, `~=` or `--=`), `redirect_sbt_dependency_lock_present` (a `build.sbt.lock`), `redirect_sbt_scala_runtime_unsupported` (`org.scala-lang`), `redirect_sbt_classifier_unsupported`; a GA no library configuration resolves is skipped silently (`redirect_sbt_meta_build_only` when only the meta-build resolves it). Advisories: `redirect_sbt_version_untested` (sbt 2.1+, still wired), `redirect_sbt_override_build_repos` (`sbt.override.build.repos=true`), `redirect_maven_pom_ignored_sbt_build` (a `pom.xml` beside the sbt build, which sbt never reads; the Maven rewriter still edits it for the Maven build). A re-run keeps an existing row and re-checks it. When the build's dependency digest changed since the pin, evidence resolved after the change (fresh, newer than the generated file) re-verifies it and the row's digest is refreshed (`redirect_sbt_pin_rechecked`); the uuid is NOT confirmed on `redirect_sbt_pin_declared_newer` (a build source now declares the GA newer than the pin's base; the row stays, sbt's load-time verifier fails the build, and the remedy is `socket-patch rollback` or declaring the base again), `redirect_sbt_pin_unverifiable` (the digest changed and the evidence predates the change, or the digest cannot be computed: run `sbt update`, then re-run socket-patch), `redirect_sbt_override_shadowed` (the evidence still resolves the base version) or `redirect_sbt_resolved_elsewhere` (the pinned version resolves from outside the pin repository from a file whose sha256 is not the pinned jar's; a copy holding the pinned bytes, such as the Ivy cache a second checkout reads, is fine — at most 64 pinned artifact files of up to 256 MiB are hashed, anything else counts as elsewhere), and also when a build source now reassigns `dependencyOverrides` / `resolvers` or a `build.sbt.lock` appeared (the same `redirect_sbt_overrides_assignment` / `redirect_sbt_resolvers_assignment` / `redirect_sbt_dependency_lock_present` codes; the row stays and sbt's load-time verifier fails the build). For a pure sbt root (no `pom.xml` / Gradle script beside it), maven confirmation is decided only by the sbt rewriter's report; on a mixed root a uuid the sbt rewriter refused is still confirmed by the Maven rewriter's own `pom.xml` pin (the generated sbt files never prove a pin by substring). **Mill and scala-cli** are guidance only: per Maven patch `redirect_mill_manual_snippet` / `redirect_scala_cli_manual_snippet` carry a paste-able snippet (repository + forced suffixed version), nothing is written or confirmed, and a pure Mill / scala-cli root gets no `redirect_maven_no_pom`; there, a Maven patch the server sent without a `maven2` registry override gets `redirect_maven_missing_override` instead of a snippet (with a `pom.xml` beside the Mill / scala-cli files the pom rewriter reports it). `rollback` / `remove` restore `socket-patch.sbt` offline (the rows removed, the file deleted with its last pin; the gitignored downloads are left). Manifest-less VEX reads every strictly parsed pin as a hosted reference but grants it the lockfile basis only when the local evidence shows every recorded version of the GA is the pinned one and every recorded artifact hashes to a pinned sha256 (else `sbt_resolution_unverified`). @@ -226,6 +228,7 @@ patches: - They are matched against a project root's **marker files**, repo-relative: the lockfiles in the root's directory (`package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`, `bun.lock(b)`, `vlt-lock.json`, `rush.json`, `uv.lock`, `poetry.lock`, `pdm.lock`, `Pipfile.lock`, `requirements.txt`, `*.py.lock`/`pylock*.toml`, `Cargo.lock`, `go.mod`, `go.sum`, `composer.lock`, `Gemfile.lock`, `gems.locked`, plus the Maven/NuGet markers). A root is ignored iff **every** marker is ignored; with `includePaths`, it is included iff **any** marker matches; a disk root with no lockfile uses its manifests (`package.json`, `pyproject.toml`, `setup.py`, `Cargo.toml`, `composer.json`, `Gemfile`, and the JVM build files: `pom.xml`, `build.gradle(.kts)`, `settings.gradle(.kts)`, `build.sbt`, `build.mill`, `build.mill.yaml`, `build.sc`, `project.scala`) instead, and one with neither is matched as its directory. So `/package-lock.json`, `**/yarn.lock` and `examples/**` mean what they mean to the scanner; `includePaths: ["/*", "!/*/"]` targets only the repo-root project. - Evaluation order (one combined list): the **built-in defaults** `test/ tests/ fixtures/ __fixtures__/ testdata/`, then `projectIgnorePaths`, then `patches.ignorePaths`. Re-include a default with a negation (`ignorePaths: ["!/e2e/tests/"]`). The defaults apply only to **discovered** roots: hosted/vendored PATH-glob matches and roots the in-memory engine detects. A root you name — `--cwd`, a literal PATH, an in-memory `projectRoots` entry — skips them (the other lists still apply). `node_modules .git .socket .yarn vendor` stay structural excludes of in-memory root detection; no policy negates them. - A workspace member that shares its root's lockfile is part of that root's project: exclude it with `ignorePackages`, not paths. +- **Nested projects in agent mode.** An agent (or report-only) disk scan crawls every nested project's `node_modules` below `--cwd`, so each installed copy is judged by the project root that owns it: the nearest directory above the copy's first `node_modules` that holds a lockfile (none: the scanned root). Nested roots are discovered roots (the built-in defaults apply), and one filtered as a whole is its own `purl: null` entry in `filtered[]`. A package stays in the run when any of its copies' roots admits it: a patch is recorded per package version, so its copies under skipped roots are patched too, and each such selected package gets a `policy_shared_copy` warning naming those roots (and a `note:` line under the human policy line). Hosted and vendored scans rewire only the named root's lockfiles, so they judge just that root. - A hosted/vendored PATH that resolves outside the repository root is a usage error (exit 2): one policy per invocation. **Lookup.** The repo root is the nearest ancestor of `--cwd` (inclusive) holding `.git` (a directory, a file for worktrees and submodules, or a symlink to either), not walking past a `GIT_CEILING_DIRECTORIES` entry or into the home directory (unless `--cwd` is it), and, on Unix, only when `.git` belongs to the current user, to root, or to the user `sudo` ran for (`SUDO_UID`); a root process trusts every owner, since CI containers commonly run as root over a checkout owned by another uid (otherwise warning `socket_yml_repo_untrusted` and `--cwd` is the root). No `.git`: the root is `--cwd`. Only `/socket.yml` and `/socket.yaml` are read, matched by exact directory-entry name (`Socket.yml` is not read: warning `socket_yml_name_case`); nested files never are. A symlinked file is followed only to a regular file inside the repo root. `--global` / `--global-prefix` scans read no file. @@ -271,7 +274,7 @@ patches: - `filtered[]`: `uuid` is null for a package filtered before any patch lookup (path, ecosystem and package reasons — those packages are never queried); a root filtered as a whole is one entry with `purl: null`. A severity entry names the top-ranked patch the floor withheld. `retained[]`: recorded packages the filters hold in place. - Reason codes (stable): `policy_disabled`, `policy_path_excluded`, `policy_path_not_included`, `policy_ecosystem`, `policy_package_not_listed`, `policy_package_ignored`, `policy_severity` (detail `low < high`, `unknown < high`). In-memory `ProjectResult.skipped[]` carries the post-lookup ones (severity, disabled) with the same codes. - Every string copied from the file (patterns, specs, key names) is truncated to 200 characters with control characters stripped. -- Warnings ride scan's top-level `warnings[]` (`{code, detail}`): `socket_yml_ignored_value`, `socket_yml_name_case`, `socket_yml_repo_untrusted`, `patches_disabled`. +- Warnings ride scan's top-level `warnings[]` (`{code, detail}`): `socket_yml_ignored_value`, `socket_yml_name_case`, `socket_yml_repo_untrusted`, `patches_disabled`, `policy_shared_copy`. - Human output: one line after the table when anything was filtered or held, e.g. `Policy (socket.yml): 3 skipped by filters, 1 patched package held.`, then every skipped project and every critical/high patch the severity floor or `enabled: false` held back, by name (a policy must not hide those silently; path, ecosystem and package filters run before any patch lookup, so their severity is unknown); `--verbose` lists every entry. A report-only `--json` run (`--prune` or `--global` with no mode) fetches patch details only when a floor or `enabled: false` could withhold something, so its `filtered[]` matches the human output. - `filtered[]` and `retained[]` are sorted by project, then purl; purls use the canonical spelling (qualifiers stripped, percent-decoded). - Exit code is unchanged by filtering. @@ -362,7 +365,7 @@ Discovery is read-only, never touches the network, and never fails the run: a ma | Ecosystem | Files read | Hosted reference | Vendored reference | Hosted pin (`integrity_required`) | |---|---|---|---|---| -| npm | `package-lock.json` and `npm-shrinkwrap.json` (both when both exist) | `resolved` on the patch host (`packages` in v2/v3; `dependencies` only in v1; `link` / `inBundle` / `bundled` entries skipped, and so is any entry npm installs from a git, URL or `file:` spec, together with every ref for the same `name@version`) | `resolved: file:.socket/vendor/npm//-.tgz` | `integrity`, required | +| npm | `package-lock.json` and `npm-shrinkwrap.json` (both when both exist) | `resolved` on the patch host (`packages` in v2/v3; `dependencies` only in v1; `link` / `inBundle` / `bundled` entries skipped, and so is any entry npm installs from a git, URL or `file:` spec or from a dependency's own `npm-shrinkwrap.json` (beneath a `hasShrinkwrap: true` package, which npm 7–11 install from that shrinkwrap; `redirect_npm_shrinkwrapped_instance_skipped` / `vendor_shrinkwrapped_instance_skipped`), together with every ref for the same `name@version`) | `resolved: file:.socket/vendor/npm//-.tgz` | `integrity`, required | | pnpm | `pnpm-lock.yaml` (every `lockfileVersion`); `shrinkwrap.yaml` only when there is no `pnpm-lock.yaml`; with `rush.json`, `common/config/rush/pnpm-lock.yaml` + `common/config/subspaces/*/pnpm-lock.yaml` | `packages:` `resolution.tarball` on the patch host | `file:.socket/vendor/npm/…` tarball + key | `integrity`, required | | yarn | `yarn.lock` (classic and berry) | classic `resolved`; berry entry keyed and resolved `@` + root `package.json` `resolutions` `"@npm:": ""` (older releases: `resolution: …::__archiveUrl=`) | classic `resolved "file:./.socket/vendor/npm/…#"`; berry `file:` entry **plus** a root `package.json` `resolutions` mapping onto the same artifact (without it the entry is orphaned: diagnosed, no ref) | classic `integrity` / `#sha1`, berry `checksum`, required | | bun | `bun.lock`; `bun.lockb` only when there is no `bun.lock` (bun reads exactly one) | URL tuple / binary remote-tarball resolution; version from the URL leaf | `.socket/vendor/npm//-.tgz` tuple / local-tarball resolution | `sha512-…`, required. A 2-tuple that Bun < 1.3.10 re-saved without its digest is still a reference, but it attests only from an installed tree. | @@ -381,8 +384,9 @@ Recognition rules that hold for every ecosystem: * **Patch hosts.** A hosted reference counts only on `https://patch.socket.dev` or the `--patch-server-url` / `SOCKET_PATCH_SERVER_URL` origin, with no userinfo. The uuid is the URL's LAST canonical-uuid path segment, because grant tokens may themselves be uuid-shaped. The Go module prefix is fixed. `socket-patch-` registry / repository / source names count only through a pin. For a URL on any other host, see **Patch hosts** above. * **Pins, not definitions.** A registry, index or source *definition* alone (cargo `[registries]`, nuget ``, pom ``, uv index tables, `.npmrc`) never makes a reference, because it survives a reverted pin. Sections the package manager ignores are not read: npm's v2 `dependencies` mirror, a `.cargo/config.toml` shadowed by `.cargo/config`. A Socket pin inside a maven `` is diagnosed, never a reference. -* **Contested locks.** When one lock wires a package to a patch and another lock resolves the same `name@version` from a non-Socket source, the build's bytes depend on which package manager runs. The reference is then dropped with a `patched_ref_unattributable` diagnostic naming both files. This applies across npm / pnpm / yarn / bun and across uv / pylock / poetry / pdm / Pipfile.lock / requirements. PEP 723 script locks neither contest nor are contested. The root `requirements.txt` and its in-root `-r` includes count as one lock here: pip reads them as one requirement set, where a direct reference wins over a compatible `==` pin of the same version in another file of the set (#1086). A **bundled** npm copy (`inBundle: true`, or v1 `bundled: true`) of the same `name@version` contests the reference too, in the same lock, in the other npm lock of a shrinkwrap/package-lock pair, or in any other lock. npm unpacks it from the parent package's tarball, so no rewire reaches it and it stays unpatched. Bun and vlt unpack bundled copies the same way (#469, #471). pnpm unpacks bundled copies too, but its lock cannot tie one to a reference (see **Unattested references** below). For Bun, that is a `bun.lock` entry whose meta is `{ "bundled": true }`, or a `bun.lockb` record that a dependency edge with the `bundled` behavior bit reaches, including one Bun shares with a regular install. Such an entry is never a reference, and it contests the reference the same way. For vlt, the lock records no node for a bundled copy, so the copy is found in the installed store: a real package directory inside a store package's own `node_modules`. Hosted and vendored scans skip these copies with `redirect_bun_bundled_instance_skipped` / `redirect_vlt_bundled_instance_skipped` / `vendor_bundled_instance_skipped`. When a bundled copy is the only instance, vendoring refuses with `vendor_lock_entry_not_rewritable`. Another entry of the **same** npm, Bun or yarn lock that resolves the wired `name@version` from a non-Socket source (for example a workspace member added after the rewire, then `npm install` / `bun install` / `yarn install`; for yarn berry, a registry locator such as `left-pad@npm:1.3.0` beside the hosted `…::__archiveUrl=` one) contests the reference too (#588). The package manager installs both entries, and that copy stays unpatched. Re-running `scan` / `vendor` rewires every copy. +* **Contested locks.** When one lock wires a package to a patch and another lock resolves the same `name@version` from a non-Socket source, the build's bytes depend on which package manager runs. The reference is then dropped with a `patched_ref_unattributable` diagnostic naming both files. This applies across npm / pnpm / yarn / bun and across uv / pylock / poetry / pdm / Pipfile.lock / requirements. PEP 723 script locks neither contest nor are contested. The root `requirements.txt` and its in-root `-r` includes count as one lock here: pip reads them as one requirement set, where a direct reference wins over a compatible `==` pin of the same version in another file of the set (#1086). A **bundled** npm copy (`inBundle: true`, or v1 `bundled: true`) of the same `name@version` contests the reference too, in the same lock, in the other npm lock of a shrinkwrap/package-lock pair, or in any other lock. npm unpacks it from the parent package's tarball, so no rewire reaches it and it stays unpatched. Bun and vlt unpack bundled copies the same way (#469, #471). pnpm unpacks bundled copies too, but its lock cannot tie one to a reference (see **Unattested references** below). For Bun, that is a `bun.lock` entry whose meta is `{ "bundled": true }`, or a `bun.lockb` record that a dependency edge with the `bundled` behavior bit reaches, including one Bun shares with a regular install. Such an entry is never a reference, and it contests the reference the same way. For vlt, the lock records no node for a bundled copy, so the copy is found in the installed store: a real package directory inside a store package's own `node_modules`. Hosted and vendored scans skip these copies with `redirect_bun_bundled_instance_skipped` / `redirect_vlt_bundled_instance_skipped` / `vendor_bundled_instance_skipped`. When a bundled copy is the only instance, vendoring refuses with `vendor_lock_entry_not_rewritable`. A reference withheld only because such an unreachable copy installs beside it (a bundled npm / Bun / vlt copy, an npm copy beneath a `hasShrinkwrap` package (#753) or one npm installs from a git / url / `file:` spec, or a yarn classic git or `file:` directory block of the same `name@version`) is still the rewriter's own wiring of exactly one package version: it is never attested, but `rollback`, `remove`, `list` and the hosted → vendored takeover treat it as an ordinary hosted pin and restore its upstream registry entry (#828), rather than refusing it with `hosted_wiring_contested`. Another entry of the **same** npm, Bun or yarn lock that resolves the wired `name@version` from a non-Socket source (for example a workspace member added after the rewire, then `npm install` / `bun install` / `yarn install`; for yarn berry, a registry locator such as `left-pad@npm:1.3.0` beside the hosted `…::__archiveUrl=` one) contests the reference too (#588). The package manager installs both entries, and that copy stays unpatched. Re-running `scan` / `vendor` rewires every copy. * **Unattested references.** Some evidence shows a build may run a copy no wiring reaches, but cannot be tied to the reference's exact `name@version` or cannot say the build runs it. The reference then stays a reference: `list`, `rollback` and `remove` find it, and the ledgers' liveness gates (`vendor --check`, `scan`) keep treating the wiring as live, because re-running `scan` / `vendor` could never clear the evidence. Only `vex` omits it, as a run warning and as the `failed[].reason`. Two cases besides Gradle's (`vex_gradle_lock_above_base`): **pnpm bundled copies** (`vex_pnpm_bundled_copy`): a `packages:` entry's `bundledDependencies:` names the copies pnpm unpacks from that package's own tarball, but pnpm never locks them, so the bundled version is not in the lock. A pnpm reference whose package name a `bundledDependencies` list in the same lock names is omitted whatever its version (a missed attestation when the bundled copy is another version, never a false one), and `bundledDependencies: true` (or a value that cannot be read) omits every reference of that lock. It reaches no other lock. **deno.lock** (`vex_deno_lock_copy`): `deno install` installs a `package.json` project's npm dependencies from `deno.lock` and never reads `package-lock.json` / `pnpm-lock.yaml` / `yarn.lock`, so an npm-family reference whose `name@version` the `deno.lock` npm section also locks is omitted (#406). Whether Deno or another package manager populates `node_modules` (`nodeModulesDir: "manual"` allows either) is not in the files, so this holds whatever `nodeModulesDir` says. +* **Shrinkwrap without a twin (#899).** npm >= 12 never reads `npm-shrinkwrap.json`: on a project whose only npm lock is the shrinkwrap it resolves the tree from the registry and writes a fresh `package-lock.json`, so a Socket reference there reaches npm <= 11 only. The reference is still discovered, so `list`, `rollback` and `remove` manage it, but `vex` omits the patch (`vex_npm_shrinkwrap_only`, as a run warning and as the `failed[].reason`). A `package-lock.json` beside it (wired the same way) makes it attestable again. Hosted and vendored runs still rewire the shrinkwrap and warn `redirect_npm_shrinkwrap_only` / `vendor_npm_shrinkwrap_only`. * **Lockless pins.** With no lock to name a version, a `Cargo.toml` pin (every declaration on `socket-patch-`, that registry defined on the patch host for the same uuid) or an exclusive nuget exact-id mapping is never a reference on its own, so v5.0 does not attest it (nor does `list` show it, or `rollback` / `remove` restore it — restore those files from version control). Only a pre-v5 redirect-ledger record naming a version the pin admits keeps it live. The same holds for a gem wired only in the `Gemfile` (the pre-bundler-2.6 mixed state, lock not converged). **Record resolution.** A candidate's record must carry the patch uuid the lockfile actually **wires**. It is taken from the first source that has one: the manifest (matched qualifier-insensitively), the hosted records above (this run's, then a pre-v5 ledger's), then the vendor ledger's embedded records. If none has it and the run is online, `vex` fetches the patch view by uuid from the patch API — for a v5 hosted checkout this is the normal path. The fetch uses `get`'s API client: the public proxy when no token is configured, and a one-shot 401/403 fallback to the proxy (free patches only). At most 10 fetches run concurrently. Fetched records stay in memory: `vex` never writes the manifest. A candidate still has no record under `--offline`, after a transport error or a 404, or when the patch is refused (paid without an entitled token); it is then omitted as `record_unavailable`, and the run is not aborted. A record whose uuid or package disagrees with the wiring is omitted as `record_mismatch`. The informational `socket-patch.vendor.json` marker is never a record source. When the lockfile wires a package to patch U, a manifest or ledger record for that package under another uuid is superseded, and a human-mode `Note:` says so. @@ -649,7 +653,7 @@ package then fails, exactly the files the eject wrote are put back (the pins' fi restore's files, and every file the vendored apply committed, each only when its bytes changed) — the project stays hosted exactly as before, and every other file (a `--json > report.json` redirect target, a log another process appends to) is left alone, with the -`eject_rolled_back` warning and `partial_failure`, exit 1; if putting the snapshot back itself fails, +`eject_rolled_back` warning (printed to stderr on a human run, which prints no "Vendored N packages" summary and no "Next steps:") and `partial_failure`, exit 1 — each package the vendored apply had vendored is re-reported as a `skipped` event with `errorCode: "eject_rolled_back"`, never `applied`, and the run's other `skipped` advisories for it (`vendor_prebuilt_downloaded`, `vendor_takeover_reverted_redirect`, …) are dropped (#1005); if putting the snapshot back itself fails, the error is `eject_rollback_failed` naming the files to `git checkout`. The eject does not emit the per-purl `vendor_takeover_reverted_redirect` warning (the restore is its own planned step). `--offline` (or `SOCKET_OFFLINE`) refuses the eject up front with `offline_eject_unavailable` — @@ -657,7 +661,12 @@ records and registry entries cannot be fetched offline — making zero network r included). A hosted wiring that discovery cannot attribute (a lock mentioning a recognized hosted uuid it rejected, or a pin with no lockfile) is refused with `hosted_wiring_contested` (exit 1, nothing touched) rather than ejecting a partial set; `rollback`, `remove` and `list` refuse the same -way (`list` degrades to a warning when it can still list). The grant token of an attributed pin's +way (`list` degrades to a warning when it can still list). A pin withheld from attestation only +because an unpatched copy of the same version installs beside it is not contested: a bundled, +git, or `hasShrinkwrap`-nested copy, or a same-lock copy the wiring cannot reach (a pnpm `file:` +directory or tarball, a yarn classic registry or `file:` directory block of the same +`name@version`, a yarn berry `file:` or URL copy under another dependency name). It is restored +like any other pin (#828, #935, #938, #939). The grant token of an attributed pin's own URL is never contested wiring where the same file also names that pin's patch uuid (a uv pin's paired `pyproject.toml` `[tool.uv.sources]` entry, a vlt pin in a `vlt-lock.json` whose pins are withheld from the lock basis); any other unattributed uuid still is. `--vex` works as on the manifest-driven @@ -1253,12 +1262,12 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `hosted_reverted` | `removed` | remove (v5.0): a hosted lockfile pin was restored to its upstream registry entry as part of removing the patch (`verified` on dry-run). Beside a manifest entry it bypasses `summary.removed` like `vendor_reverted`. | | `hosted_revert_failed` | top-level error | remove (v5.0): a matched hosted pin could not be restored to its upstream registry entry (`--offline`, a registry that does not answer, `bun.lockb`, a lock shape the restore refuses — see "Hosted unwind coverage"), or writing the restored files failed; the message names the `git checkout -- ` remedy. The manifest was not modified, exit 1. Rollback's counterpart is a `hosted.failed[]` entry (also `partial_failure` exit 1). v4's `hosted_revert_unsupported` is no longer emitted (every ecosystem has a restore). | | `reinstall_required` | rollback `warnings[]` | rollback (v5.0): vendored/hosted wiring was unwound, but installed trees keep their patched bytes until the next package-manager install — the stale-install advisory. | -| `hosted_state_not_preservable` | rollback `warnings[]` | rollback `--preserve-state` (v5.0): hosted pins were restored to upstream anyway — the lockfile pins are hosted mode's only record, so there is no local state to preserve; re-run `scan --mode hosted` to re-wire. (`remove --preserve-state` prints the same note on stderr.) | +| `hosted_state_not_preservable` | rollback / remove `warnings[]` | rollback `--preserve-state` (v5.0): hosted pins were restored to upstream anyway — the lockfile pins are hosted mode's only record, so there is no local state to preserve; re-run `scan --mode hosted` to re-wire. (`remove --preserve-state` reports the same code in its `warnings[]` — manifest-backed and hosted-only alike — and prints it as a `Note:` on stderr in human mode.) | | `out_of_scope_copies_restored` | rollback `warnings[]` | path-scoped rollback (v5.0): a selected patch had installed copies outside the given patterns; ALL copies were restored (patches are per-package). Informational — never flips the exit. | | `vendor_ledger_entry_unwired` | scan `warnings[]` | a vendored entry's dependency left the lockfile (upgraded or removed), so the ledger supplement skipped it; the detail names the purls and points at `scan --prune`, which reverts them (no warning on a pruning non-hosted run). An entry that prune drift-keeps (its lock entries were re-resolved since vendoring, e.g. an npm uninstall re-locked it away) is reported on the prune's `GC: kept` line and keeps being warned about. | | `path_scope_excluded_supplements` | scan `warnings[]` | path-scoped scan (v5.0): lockfile-only / vendor-ledger supplement packages have no installed path and were excluded from the scoped scan; the detail carries the count. | -| `vendor_commit_failed` | top-level error (`vendor`, and the nested vendor envelope of `scan` / `get --mode vendored`) | v5.0 group commit: the run's lockfile / manifest / ledger edits could not be written (the detail names the I/O error). Exit 1; the project's lockfiles and `.socket/vendor/state.json` are left as they were before the run (a partially-applied commit is put back), and the per-package events describe the uncommitted outcome. When putting a partially-applied commit back fails too, the journal is kept instead and the detail says the next socket-patch command in the project finishes the commit. | -| `redirect_symlinked_file_unsupported` (vendored) | top-level error (`vendor`, and the nested vendor envelope of `scan` / `get --mode vendored`) | v5.0 group commit (#627): a file the run would rewrite — a lockfile, `package.json`, `pnpm-workspace.yaml`, `nuget.config`, … — is a symbolic link. The commit stages each file and renames it over the path, which would replace the link with a detached copy and leave its target (the lock other checkouts read) unpatched, so it refuses before writing anything — the same code and message as the hosted guard. Exit 1; the link, its target and `.socket/vendor/state.json` are left as they were, and the per-package events describe the uncommitted outcome (the artifacts written are unreferenced orphans, as for `vendor_commit_failed`). Backends that check their own targets first (Hatch, uv, Poetry, PDM, Pipenv, requirements, PEP 751 locks, Cargo) refuse per package with this same code (see the per-package row below); a symlinked `bun.lockb` keeps `vendor_bun_lockb_invalid`. | +| `vendor_commit_failed` | top-level error (`vendor`, and the nested vendor envelope of `scan` / `get --mode vendored`); also `failed` (per package) | v5.0 group commit: the run's lockfile / manifest / ledger edits could not be written (the detail names the I/O error). Exit 1; the project's lockfiles and `.socket/vendor/state.json` are left as they were before the run (a partially-applied commit is put back). As for `redirect_symlinked_file_unsupported` (#898), the artifact dirs the run added under `.socket/vendor/` are removed (any that cannot be are named in the detail), each package the run vendored is a `failed` event with this code, not `applied`, its `skipped` advisories from the run are dropped, and the human run prints no "Vendored N packages" count for them and no "Next steps:". When putting a partially-applied commit back fails too, the journal is kept instead, the detail says the next socket-patch command in the project finishes the commit, and the per-package events stand (that commit does complete). | +| `redirect_symlinked_file_unsupported` (vendored) | top-level error (`vendor`, and the nested vendor envelope of `scan` / `get --mode vendored`) | v5.0 group commit (#627): a file the run would rewrite — a lockfile, `package.json`, `pnpm-workspace.yaml`, `nuget.config`, … — is a symbolic link. The commit stages each file and renames it over the path, which would replace the link with a detached copy and leave its target (the lock other checkouts read) unpatched, so it refuses before writing anything — the same code and message as the hosted guard. Exit 1; the link, its target and `.socket/vendor/state.json` are left as they were, the artifact dirs the run added under `.socket/vendor/` are removed (any that cannot be are named in the message, with `vendor --revert` as the remedy), and each package the run vendored is a `failed` event with this code, not `applied` (`summary.applied` does not count it); the `skipped` advisories the run recorded for it (`vendor_prebuilt_downloaded`, `vendor_artifact_reused`, …) are dropped, since they describe the vendoring that was refused. The human run prints no "Vendored N packages" count for them and no "Next steps:" (#898). Backends that check their own targets first (Hatch, uv, Poetry, PDM, Pipenv, requirements, PEP 751 locks, Cargo) refuse per package with this same code (see the per-package row below); a symlinked `bun.lockb` keeps `vendor_bun_lockb_invalid`. | | `vendor_would_refuse_symlinked_file` | `skipped` (advisory event) under `vendor --dry-run`; a `warnings: [{code, detail}]` entry on the `would_vendor` / `would_revendor` row of the `vendor` preview under `scan` / `get --mode vendored --dry-run` (human: an `[warning] (): ` line) | dry run (#627): a dry run captures no writes, so for each package it would vendor (not one previewed as in sync, whose re-run writes nothing) it names every symlinked file of that package's ecosystem a vendored run may rewrite (the registry's vendored rewrite targets plus `pnpm-workspace.yaml`, `nuget.config`, `packages.lock.json`, the root `pom.xml`, `.mvn/maven.config` and `hatch.toml`; files a vendored run only reads, such as `.yarnrc.yml` or `vlt.json`, are never named); the wet run refuses with `redirect_symlinked_file_unsupported` if it must rewrite one. Does not change the exit code. | | `vendor_state_unreadable` | rollback `warnings[]`; remove top-level error | corrupt-ledger containment (v5.0). Rollback: an unreadable vendor ledger skips the vendored leg + manifest cleanup + GC and drives `partial_failure` exit 1 while the agent and hosted legs still run. Remove: a hard top-level error before any mutation. Also the Bun vendored preflight's refusal code: `get` / `scan --mode vendored`, `vendor`'s pre-takeover check and the `--dry-run` `would_refuse` preview report an unreadable `.socket/vendor/state.json` as itself (`errorCode` in `patches[]` / `download.patches[]`, or `get `'s top-level `error.code`), fail-closed — nothing is exempt — instead of a Bun lock code. (v4's `redirect_state_unreadable` is no longer emitted: v5 never reads the redirect ledger on these paths.) | | `manifest_write_failed` | rollback `warnings[]` | rollback (v5.0): the post-rollback manifest update could not be written; no entries were removed (`manifest.removedEntries: []`) and the run exits `partial_failure` 1. | @@ -1269,6 +1278,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_stale_artifact_removed` | `removed` | vendor / scan `--mode vendored`: re-vendor under a newer patch uuid removed the previous uuid's orphaned artifact dir. | | `vendor_unsupported_ecosystem` | `skipped` | vendor: no vendor backend for this purl's ecosystem (jsr). | | `already_vendored` | `skipped` | vendor: artifact + wiring already in sync for this patch uuid. | +| `vendor_prebuilt_pending` / `vendor_prebuilt_unavailable` | `skipped` | vendor / scan / get `--mode vendored` (npm family, #954): the package is already vendored at another patch uuid, and the patch service has no prebuilt artifact for the selected (superseding) uuid — still building (`pending_build`) / not served (`build_failed`, `not_found`, `withdrawn`, no usable artifact). Nothing is touched: the older patch's artifact, wiring and ledger entry stay in force, the detail names both uuids (`kept the vendored patch : prebuilt artifact is still building for patch `), and the run does not fail (exit 0), the way hosted mode keeps its pin and lists the upgrade in `redirect.skipped[]`. A re-run picks the new patch up once the service serves it. A package with no vendored patch to keep still fails (`failed`, error `prebuilt artifact is still building` / `prebuilt artifact unavailable: `), as does a request/transport failure. | | `unsafe_coordinates` | `failed` | vendor: purl/uuid would escape `.socket/vendor/` (tampered manifest/state); refused before any write. | | `revert_failed` | `failed` | vendor --revert: a recorded entry could not be reverted. | | `vendor_ledger_missing` | `failed` (artifact-level: `uuid` + `details.{ecosystem,path}`, no purl) | repair (v5.0) and `vendor --check`: a lockfile references `.socket/vendor///` but the vendor ledger has no entry for it; repair no longer rebuilds ledger entries from lockfiles. Recovery: restore `.socket/vendor/state.json` from version control and re-run `repair`, or `git checkout -- ` and re-vendor. | @@ -1283,7 +1293,11 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `not_applied` / `hash_mismatch` / `file_not_found` / `no_matching_variant` | `failed` | `apply --check` (v5.0): an installed copy of the patch does not verify (still unpatched; neither the original nor the patched bytes; a patched file missing; a copy of a release-variant base that holds none of the manifest's variants, keyed by the base purl). Exit 1, status `partialFailure`. | | `redirect_unconfirmed` | `redirect.patches[]` `unpinned` row | hosted `scan` / `get` (v5.0, additive): the patch was granted but no lockfile entry pinning it could be rewritten. The status and exit code are unchanged for now, pending the open hosted exit-policy decision (#704); `--silent` hides the human line. | | `lockfile_unreadable` / `lockfile_unparseable` / `patched_ref_invalid` / `patched_ref_unattributable` | run-level `warnings[]` | vex (every form): lockfile-discovery diagnostics — see "Manifest-less VEX (lockfile discovery)". Never flip the exit on their own. | +| `vex_npm_shrinkwrap_only` | run warning and `failed[].reason` | vex (every form, #899): the patch is wired only in a root `npm-shrinkwrap.json` with no `package-lock.json` twin, which npm >= 12 never reads; no statement until the twin exists. `list` / `rollback` / `remove` still manage the wiring. | | `vendor_multiple_lockfiles` / `pypi_multiple_lockfiles` | `skipped` (warning) | vendor: a sibling lockfile of another package manager (for PyPI, also a root `requirements.txt` that pins the package beside the wired tool lock) will still install UNPATCHED bytes; names the wired winner + the ignored locks. | +| `vendor_npm_shrinkwrap_only` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #899): the package was wired into `npm-shrinkwrap.json`, the only npm lock (also on an in-sync re-run). npm >= 12 never reads the shrinkwrap and installs the unpatched registry bytes; rename the lock to `package-lock.json` (or commit a copy under that name) and re-run. | +| `vendor_workspace_member_skipped` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #688): a lock entry with the package's `name@version` is the project's own source (a workspace member or a `file:` directory), so it is left alone while the lock's registry copies are vendored; patch that source directly if it needs the fix. When it is the only instance, vendoring refuses instead. | +| `vendor_npm_allow_file` | `skipped` (warning, printed as `Warning (vendor_npm_allow_file)`) | vendor / scan / get `--mode vendored` (package-lock, #969): the effective npm `allow-file` setting (env > project `.npmrc` > user > global > builtin) is not `all`, so npm >= 11.14 refuses the vendored `file:` tarball (EALLOWFILE) on install. The setting is respected, never rewritten; the detail names where it comes from and the remedy (`allow-file=all`). `vendor --check` fails the entry for the same reason. | | `vendor_yarn_berry_unsupported` | `failed` | vendor (npm): yarn-berry Plug'n'Play layout; use its native `yarn patch` workflow. | | `vendor_bun_lockb_invalid` | `failed` | vendor / scan / get `--mode vendored`: the binary lock is malformed, unreadable, unsupported or cannot be rewritten safely. The detail names the parser, hash or filesystem error. Refused before patch downloads and before hosted takeover; `patches[]` / `download.patches[]` carry `errorCode` and `error`, while `get ` also carries top-level `error.code`. Dry-run predicts the same refusal. | | `vendor_bun_workspace_unsupported` | `failed` | vendor / scan / get `--mode vendored` (bun): the text lock holds `workspace:` packages and its `lockfileVersion` is below 2 — Bun 1.2–1.3 resolve a workspace member's local-tarball path relative to the member; a committed version-2 lock is the proof every consumer runs Bun ≥ 1.4 (deliberate over-approximation: root-only declared packages would install on version 1 too). Detail names the version integer and a version-specific remedy: delete `bun.lock` and re-lock with Bun ≥ 1.4 (an in-place `bun install` keeps the existing version) — then, for a version-1 lock, "or use `--mode hosted`, which accepts version-1 workspace locks"; for a version-0 lock, "or delete `bun.lock`, re-lock with Bun ≥ 1.2 (which writes lockfileVersion 1) and use `--mode hosted`" (hosted refuses version-0 workspace locks, so a bare hosted pointer would send the user into a second refusal). Refused before any write — in the pre-download preflight on `get`/`scan` (see `vendor_bun_lockb_invalid` for the placements); in the shared preflight that `vendor` and the vendor step run BEFORE a hosted → vendored takeover's revert (a hosted-redirected purl stays hosted-wired, ledger and lock untouched; `vendor --dry-run` previews the same `failed` code); and in the engine when the run would write a NEW local tuple. Exempt: purls the vendor ledger wires at the selected uuid, purls whose every `bun.lock` instance is already a `.socket/vendor/npm/` tuple (any uuid), in-sync re-runs and `repair` redownloads. | @@ -1305,7 +1319,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `redirect_pnpm_settings_elsewhere` | top-level `error.code` (`status: "error"`) | scan / get `--mode hosted`: the project directory is a pnpm workspace member with its own v9 `pnpm-lock.yaml` (`sharedWorkspaceLockfile: false`) and no `pnpm-workspace.yaml` of its own, so its pnpm settings come from the nearest ancestor `pnpm-workspace.yaml`, which pnpm reads alone (a member's own file is ignored). When that file neither carries `trustLockfile: true` nor explicitly sets another value, the trust auto-config has nowhere to go: refused before any takeover or write, `--dry-run` included; the message names the root file to add `trustLockfile: true` to (or `--no-trust-lockfile-config` pins without it); exit 1. Once the root file trusts the lock (or opts out), the member is pinned and no nested `pnpm-workspace.yaml` is created; the `redirect_pnpm_trust_lockfile` warning names the root file. Disk runs only. | | `eject_refused` | top-level `error.code` (`status: "error"`) | vendor eject (v5.0): a record fetch failed or a pin's upstream restore was refused while planning; nothing was changed, exit 1. | | `eject_planned` | `applied` (reason) | vendor eject `--dry-run` (v5.0): the pin would be restored upstream and vendored; nothing written. | -| `eject_rolled_back` | warning | vendor eject (v5.0): a package failed after the restore began; every touched file was put back from the pre-eject snapshot, so the project is still hosted; `partial_failure`, exit 1. | +| `eject_rolled_back` | warning; `skipped` event | vendor eject (v5.0): a package failed after the restore began; every touched file was put back from the pre-eject snapshot, so the project is still hosted; `partial_failure`, exit 1. Also the `errorCode` of the `skipped` event re-reporting each package the eject had vendored before it was rolled back (not counted in `summary.applied`). | | `eject_rollback_failed` | top-level `error.code` | vendor eject (v5.0): putting the pre-eject snapshot back failed; the detail names the files to `git checkout --`; exit 1. | | `offline_eject_unavailable` | top-level `error.code` | vendor eject under `--offline` / `SOCKET_OFFLINE` (v5.0): records and registry entries cannot be fetched offline; zero network requests, nothing touched, exit 1. | | `hosted_wiring_contested` | top-level `error.code` (list: warning when it can still list) | rollback / remove / vendor eject / list (v5.0): a lockfile mentions a recognized hosted patch uuid that discovery rejected (or a pin with no lockfile), so the hosted set is not known exactly; refused with nothing touched, exit 1. Remedy: fix or `git checkout` the named lockfile. | @@ -1333,7 +1347,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_artifact_redownload_failed` | `failed` | repair: download unavailable, integrity mismatch, or downloaded bytes/inventory differ from the ledger. Existing files are preserved. | | `vendor_artifact_unrepairable` | `failed` | repair: the ledger identity or patch record cannot be trusted or recovered. | | `vendor_uuid_mismatch` | `skipped` | repair: the manifest's patch uuid moved past the vendored artifact — a re-vendor (`vendor` / `scan --mode vendored`) is pending; repair does not cross patch generations. | -| `content_mismatch_overwritten` | `skipped` (warning) | apply (default policy): a file matched NEITHER beforeHash nor afterHash and was overwritten with the full verified patched content. This includes a file the patch adds (empty beforeHash) that already exists with other content. `--strict` turns this case into a `failed` event instead. | +| `content_mismatch_overwritten` | `skipped` (warning) | apply (default policy): a file matched NEITHER beforeHash nor afterHash and was overwritten with the full verified patched content. This includes a file the patch adds (empty beforeHash) that already exists with other content. `--strict` turns this case into a `failed` event instead. Agent-mode `get` / `scan --json` carry it as a `(content_mismatch_overwritten) …` `warnings[]` entry (#1004). | | `vendor_lock_checksums_unsupported` / `vendor_stale_lock_checksum` | `failed` | vendor (gem): an ambiguous/platform CHECKSUMS entry, or a v1-wired lock whose stale token blocks the hot path (run `vendor --revert` + re-vendor). | | `redirect_unattributable` | `redirect.skipped[].reason` | scan/get `--mode hosted`: the rewriters would pin the patch, but lockfile discovery over the result reads that pin as contested (another lock or requirements file resolves the same version elsewhere, or the pin is not one the package manager consumes), so `vex`, `rollback`, `remove` and `vendor` would refuse it. The candidate is left out of the rewrite, so nothing is written for it; the detail carries discovery's findings. Exit code unchanged. | | `redirect_pin_lockless` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (nuget, cargo): the pin was written without a lockfile that records its version, so `vex` cannot attest it and `rollback` / `remove` / `vendor` refuse it as unattributable. The detail names the lockfile to create (`dotnet restore --use-lock-file`, `cargo generate-lockfile`) before re-running the hosted scan. | @@ -1362,6 +1376,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `redirect_bun_lockb_invalid` | `redirect.warnings[]` (warning) | scan/get `--mode hosted`: the native binary lock is malformed, unreadable, unsupported or cannot be rewritten safely. No installer is spawned and no binary or sibling npm lock edit or takeover occurs; dry-run reports the same format error. Exit 0, `redirected: 0`. | | `redirect_bun_entry_not_found` / `redirect_bun_missing_sha512` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (bun): the lock has no rewritable entry at the granted version (re-resolved, or occupied by an unowned URL/file spec) / the grant carries no sha512 integrity. Per-dep; nothing rewritten for it; exit 0. NOT emitted for the digest-less 2-tuple Bun 1.1.39–1.3.9 re-save our URL tuple as — that entry counts as redirected and is healed. | | `redirect_bun_patched_dependency_skipped` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (bun, `bun.lock` and `bun.lockb`): the project patches the granted `name@version` itself with `bun patch` (a `patchedDependencies` key for `name@version`, or the bare name, in the root `package.json` or mirrored in `bun.lock`). Bun applies that patch only to the registry resolution, so the entry is left on its registry tuple instead of silently losing the user's patch (#367). Per-dep; the detail names the key and the remedy (fold the Socket fix into the user's patch, or drop the `patchedDependencies` entry and re-run); the in-run VEX never assumes the uuid applied. Vendored mode refuses the same package `vendor_lock_entry_unsupported` before any write or download. Exit 0. | +| `redirect_npm_patched_dependency_skipped` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (`package-lock.json` / `npm-shrinkwrap.json`): the project patches the granted `name@version` itself with npm ≥ 12.1's native `npm patch` (a root `package.json` `patchedDependencies` key for `name@version` or the bare name, or a `packages` entry carrying npm's `patched` record, which npm writes in a lockfileVersion 4 lock). npm applies the user's diff on top of whatever tarball the lock names and fails the install `EPATCHFAILED` when it no longer applies, so a hosted pin would break every later `npm ci` / `npm install` (or install bytes VEX can never attest); the dep is left on its registry entry in every present npm lock instead (#711). Per-dep; the detail names the key or lock entry and the remedy (fold the Socket fix into the user's patch, or drop the `patchedDependencies` entry and re-run); when an earlier hosted run already pinned the entry, the detail says so and names `socket-patch rollback ` to restore the registry entry instead of claiming it is unchanged; the in-run VEX never assumes the uuid applied, and no sibling lock confirms it. Vendored mode refuses the lockfileVersion 4 lock `vendor_lockfile_version_unsupported`, its detail naming `npm patch` / `patchedDependencies`. Exit 0. | | `redirect_vlt_lock_unsupported` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): `vlt-lock.json` has a `lockfileVersion` other than absent, `0` or `1` (decided on the raw JSON token), is not a JSON object, starts with a UTF-8 BOM, or its `nodes` section is not vlt's one-node-per-line layout. Nothing rewritten; also refuses a vendored → hosted takeover of a `flavor: "vlt"` entry before its revert (`redirect.skipped[].reason`). Exit 0. | | `redirect_takeover_kept_vendored` | `redirect.warnings[]` (warning) | scan/get `--mode hosted`: a vendored → hosted takeover the hosted rewrite would not pin was retracted (see **Staged takeover**): the package keeps its vendored wiring, ledger entry and artifact byte-identical and stays patched. The detail names the cause code, which is also the purl's `redirect.skipped[].reason`. Exit 0. Replaces v5.0-pre `redirect_takeover_unpatched`, which reported a package left unpatched in both modes and is no longer emitted. | | `redirect_takeover_not_pinned` | `redirect.skipped[].reason` | scan/get `--mode hosted`: the skip reason of a retracted takeover when no rewriter warning names the cause. | @@ -1371,7 +1386,9 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `redirect_vlt_custom_registry_skipped` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): same-`name@version` nodes under a named alias, a scoped registry or jsr, or git, remote-tarball or local-directory nodes of the same package name (vlt records no version for those; a remote tarball whose `-.tgz` leaf names another version does not count), were left untouched (hosted mode only redirects vlt's default registry). The dep is still redirected, but the run's `--vex` does not attest it, and neither does a later `vex` from the lock alone. | | `redirect_vlt_lockfile_version_missing` / `redirect_vlt_old_lockfile_ignored` / `redirect_vlt_scalar_registry_ignored` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): the lock has no `lockfileVersion` (vlt ≥ 1.0.0-rc.15 re-resolves it) / a legacy default-registry id without `"modifiers"` in `vlt.json` (vlt 0.0.0-16 … 0.0.0-24 ignore the lock) / a scalar `registry` option that vlt 1.0.0-rc.7 … rc.29 honor over the lock. The deps stay redirected, but the run's `--vex` does not attest them. | | `redirect_vlt_sibling_lockfiles` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): `vlt-lock.json` and another npm-family lock are both present and vlt's install state (`node_modules/.vlt-lock.json` or `node_modules/.vlt/`) is not, so both locks were rewritten and the other lock's rules confirm. | +| `redirect_npm_replace_registry_host` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (npm): the effective npm `replace-registry-host` (env var, project `.npmrc`, or user / global / builtin config) is `always` or a pinned hosted hostname, so npm rewrites the hosted pins to the configured registry and every install fails E404; the detail names the layer and the remedies (`replace-registry-host=npmjs` in the project `.npmrc`, or vendored mode). See the npm `replace-registry-host` contract. | | `redirect_vlt_no_lockfile` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): `vlt.json` or vlt's install state is present without `vlt-lock.json`; replaces `redirect_npm_no_lockfile` for vlt projects. | +| `redirect_npm_shrinkwrap_only` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (npm, #899): the root `npm-shrinkwrap.json` is the only npm lock and carries a hosted redirect (this run's or an earlier one's — every run repeats it until the project gains a `package-lock.json`). npm >= 12 never reads the shrinkwrap and installs the unpatched registry bytes; npm <= 11 installs the patch. Rename the lock to `package-lock.json` (or commit a copy under that name) and re-run. | | `redirect_vlt_artifact_unverifiable` | `redirect.warnings[]` (warning), `redirect.skipped[].reason` | scan/get `--mode hosted` (vlt): before any takeover or rewrite (dry runs included), each granted artifact with a default-registry instance in `vlt-lock.json` (or, for a purl a `flavor: "vlt"` vendored entry claims, its vendored node, probed before the takeover reverts it) is fetched once as vlt fetches it (`accept-encoding: gzip;q=1.0, identity;q=0.5`, no `Authorization`, up to 10 redirects) and must return 200 with no content encoding (or `identity`) and the granted sha512. On failure (`content-encoding `, `sha512 mismatch`, `http `, `fetch error `, `offline`) the dep is withheld from every rewriter when vlt drives or it is vlt-vendored (which also keeps it vendored), and from the vlt rewrite only otherwise (detail "…; vlt-lock.json was not changed for {purl}"; only the sibling lock this run rewrote can confirm it). A lock already pinned by an earlier run is left pinned, and neither confirmed nor attested. Projects without `vlt-lock.json` make no such request. The detail quotes the artifact URL (and any fetch error that echoes it) with its grant-token path level, the one just before the patch uuid, spelled ``; host, uuid and leaf stay. The in-memory hosted engine (`hosted-bundle`, the Node addon) has no network for this fetch, so it judges every in-scope artifact as `--offline` does (withheld, never pinned; the vendored takeover it refuses anyway). Exit 0. | | `redirect_vlt_reinstall_required` | `redirect.warnings[]` (advisory); rollback/remove `warnings[]` (+ human stderr) | vlt: `vlt-lock.json` pins (or, after rollback/remove, no longer pins) Socket-patched packages, and vlt never refreshes an installed copy. The heal removes `node_modules/.vlt-lock.json` and each stale `node_modules/.vlt/` of a Socket-owned node (never a link's target, never outside the project, never a copy it cannot judge) unless `--no-vlt-install-cleanup` or `--dry-run`. It never removes an optional node's copy (lock flags 1 or 3, or flags it cannot read): `vlt install` does not put a removed optional dependency back (its link dangles) unless the same install also reinstalls a non-optional node, so such a copy is left stale and the detail says to run `vlt ci` (or delete `node_modules` and run `vlt install`); vlt 0.0.0-30 … 1.0.4 install no optional dependency from the lock of a project that declares only optional dependencies, so there both commands remove the installed copy and the detail says to upgrade vlt to 1.0.5 or later first. The detail says whether copies were removed, left stale by a skipped cleanup, could not be checked, or none were stale, and adds how many optional copies were kept whenever there are any. The kept optional copies are named by what they are: `unpatched copies of optional dependencies` after `scan`/`get`, `patched copies of optional dependencies` after `rollback`/`remove`, and `installed copies of the vendored optional dependencies` after a hosted → vendored takeover (the copy the hosted pin left installed, which may still be the registry bytes). Stale or unchecked copies are not attested by the run's `--vex`, nor is a confirmed vlt pin the heal did not check (a URL on a host other than patch.socket.dev and the configured `--patch-server-url`/`--api-url`). A hidden lock that cannot be removed keeps every store entry. Invalidation failures only warn. | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | @@ -1537,6 +1554,19 @@ on the same object (`apply` in `scan`'s envelope); `status` stays `partial_failure`. v5.0: this replaced the top-level `errorCode` + string `error` pair (MAJOR). +Agent-mode mismatch overwrites (#1004): when the nested apply's default +mismatch policy overwrites a file that matched neither the patch's +beforeHash nor its afterHash (a local edit, a `patch-package` / `npm +patch` change), the same object's string `warnings[]` (`apply` in +`scan`'s envelope) carries one +`(content_mismatch_overwritten) : did not match the patch's +expected original content; the full verified patched content was applied` +entry per file — the warning `apply --json` reports as a +`content_mismatch_overwritten` `skipped` event and the human run prints on +stderr. The patch record keeps its `added` / `updated` / `skipped` action +and counts as applied; the status and exit code are unchanged (`--strict` +turns the case into an apply failure instead). + `vulnerabilities[]` is always sorted by `id` so consumer diffs and test snapshots are stable. `severity` at the top level is the max across the array using the ordering `critical > high > medium = moderate > low > (unknown)`. @@ -1814,7 +1844,13 @@ resolves any more (`dependency removed`; `scan --mode vendored --prune` reverts the entry). For a package-lock entry, drift also includes a `package-lock.json` / `npm-shrinkwrap.json` entry for the vendored `name@version` that `vendor` would rewire but that does not resolve to the vendored artifact -(#588); the reason names that entry. Missing ledger entries fail with +(#588); the reason names that entry. It also includes any entry for that +`name@version` beneath a `hasShrinkwrap: true` package, which npm 7–11 install +from that package's own `npm-shrinkwrap.json` whatever the lock says (#753); the +reason names the entry and its shrinkwrapping dependency, which must be updated +since re-vendoring cannot reach that copy. A package-lock entry also fails when +npm's effective `allow-file` setting refuses its vendored `file:` tarball (#969, +see the npm vendored `allow-file` contract). Missing ledger entries fail with `vendor_ledger_missing`: a manifest patch with no ledger entry, and a lockfile reference to `.socket/vendor///` that no ledger entry owns (the artifact-level event repair emits: `uuid` plus `details.{ecosystem,path}`, no diff --git a/crates/socket-patch-cli/src/commands/agent_download.rs b/crates/socket-patch-cli/src/commands/agent_download.rs index 096b00146..88e22b130 100644 --- a/crates/socket-patch-cli/src/commands/agent_download.rs +++ b/crates/socket-patch-cli/src/commands/agent_download.rs @@ -1389,6 +1389,22 @@ pub(crate) async fn run_nested_apply( report } +/// The nested apply's non-fatal warnings (today the default policy's +/// `content_mismatch_overwritten` overwrites, #1004) as `get`'s string +/// `warnings[]` entries, code-prefixed like `get`'s `fold_narrowing_into_result`. +/// A JSON caller's nested apply is silent, so the envelope is the only +/// place these surface; a human caller's apply already printed them. +pub(crate) fn apply_warning_lines(report: Option<&ApplyRunReport>) -> Vec { + report + .map(|r| { + r.warnings + .iter() + .map(|w| format!("({}) {}", w.code, w.detail)) + .collect() + }) + .unwrap_or_default() +} + /// Whether apply's package key `key` covers the patch record purl /// `record`: the same purl, or `key` is the unqualified base of a /// qualified record (apply keys a release-variant base by its base purl). @@ -1665,7 +1681,9 @@ pub async fn download_and_apply_patches_with( } // Surface release-narrowing fallbacks (uninstalled package / no // matching variant) so JSON consumers can see why all variants were - // kept. Omitted entirely when narrowing was clean. + // kept, and the apply's mismatch overwrites. Omitted entirely when + // both were clean. + warnings.extend(apply_warning_lines(apply_report.as_ref())); if !warnings.is_empty() { result_json["warnings"] = serde_json::json!(warnings); } diff --git a/crates/socket-patch-cli/src/commands/apply.rs b/crates/socket-patch-cli/src/commands/apply.rs index 8bcb3722d..16aa38c9b 100644 --- a/crates/socket-patch-cli/src/commands/apply.rs +++ b/crates/socket-patch-cli/src/commands/apply.rs @@ -86,6 +86,25 @@ fn mismatch_event_detail(file: &str, dry_run: bool) -> String { ) } +/// One `content_mismatch_overwritten` run warning per mismatch-overwritten +/// file across `results`, in the event detail's words prefixed with the +/// package purl — what a nested apply hands back to `get` / `scan --mode +/// agent` (#1004). +fn mismatch_overwrite_warnings(results: &[ApplyResult], dry_run: bool) -> Vec { + results + .iter() + .flat_map(|r| { + let purl = normalize_purl(&r.package_key); + mismatch_overwritten_files(r) + .into_iter() + .map(move |file| RunWarning { + code: "content_mismatch_overwritten".to_string(), + detail: format!("{purl}: {}", mismatch_event_detail(&file, dry_run)), + }) + }) + .collect() +} + /// `1 mismatched file` / `2 mismatched files`, with the verb agreeing. fn mismatched_files_fail(n: usize) -> String { if n == 1 { @@ -1219,15 +1238,20 @@ pub(crate) struct ApplyRunReport { /// failed run's caller can count exactly what applied. Filled only /// when `code != 0`. pub applied: Vec, + /// Non-fatal per-file warnings the caller's envelope must carry: one + /// `content_mismatch_overwritten` per file the default mismatch policy + /// overwrote (#1004). A nested apply never prints JSON and is silent + /// for a JSON caller, so this is their only channel. Filled whenever + /// the apply loop ran, whatever the exit code. + pub warnings: Vec, } impl ApplyRunReport { fn run_failure(code: i32, error_code: &str, error: impl Into) -> Self { Self { code, - failures: Vec::new(), run_error: Some((error_code.to_string(), error.into())), - applied: Vec::new(), + ..Self::default() } } } @@ -1627,10 +1651,16 @@ pub(crate) async fn run_locked( .await; } + // The mismatch overwrites, for a nested caller's envelope (the + // JSON events above are the standalone apply's copy). + let warnings = mismatch_overwrite_warnings(&results, args.common.dry_run); // A requested-but-failed VEX flips an otherwise-successful // apply to a non-zero exit (fail-the-command contract). if success && !vex_failed { - return ApplyRunReport::default(); + return ApplyRunReport { + warnings, + ..ApplyRunReport::default() + }; } let failures = if success { Vec::new() @@ -1668,6 +1698,7 @@ pub(crate) async fn run_locked( failures, run_error, applied, + warnings, } } Err(e) => { diff --git a/crates/socket-patch-cli/src/commands/get.rs b/crates/socket-patch-cli/src/commands/get.rs index 77e287e2f..2331c68de 100644 --- a/crates/socket-patch-cli/src/commands/get.rs +++ b/crates/socket-patch-cli/src/commands/get.rs @@ -30,11 +30,12 @@ use crate::args::{apply_env_toggles, GlobalArgs}; // `commands::get` paths (the in-process tests and embedders call them); // the engine itself lives in the shared `agent_download` helper. use crate::commands::agent_download::{ - decide_patch_action, download_patch_records_preflighted, download_patch_records_reusing, - filter_to_installed_releases, fold_apply_failures, max_vuln_severity, merge_metadata, - nested_apply_args, patch_event_metadata, report_error, report_lock_failure, run_nested_apply, - run_outcome, unwind_new_blobs, warn_on_vendored_uuid_drift, write_all_patch_blobs, - DetachedDownload, PatchAction, VendorRefusals, + apply_warning_lines, decide_patch_action, download_patch_records_preflighted, + download_patch_records_reusing, filter_to_installed_releases, fold_apply_failures, + max_vuln_severity, merge_metadata, nested_apply_args, patch_event_metadata, report_error, + report_lock_failure, run_nested_apply, run_outcome, unwind_new_blobs, + warn_on_vendored_uuid_drift, write_all_patch_blobs, DetachedDownload, PatchAction, + VendorRefusals, }; pub use crate::commands::agent_download::{ download_and_apply_patches_with, DownloadParams, DownloadRun, @@ -2062,6 +2063,7 @@ async fn save_and_apply_patch(args: &GetArgs, client: &ApiClient, patch: &PatchR result_json["applied"] = serde_json::json!(applied); } // Same contract as `download_and_apply_patches_with`: omitted when clean. + warnings.extend(apply_warning_lines(apply_report.as_ref())); if !warnings.is_empty() { result_json["warnings"] = serde_json::json!(warnings); } @@ -3168,6 +3170,7 @@ mod tests { failures, run_error: None, applied: applied.iter().map(|p| p.to_string()).collect(), + warnings: Vec::new(), } } @@ -3324,6 +3327,7 @@ mod tests { failures: Vec::new(), run_error: Some(("yarn_pnp_unsupported".to_string(), "pnp".to_string())), applied: Vec::new(), + warnings: Vec::new(), }; assert_eq!(fold_apply_failures(&mut env, &report, |_| None), 0); assert!(env.get("errorCode").is_none(), "{env}"); diff --git a/crates/socket-patch-cli/src/commands/mod.rs b/crates/socket-patch-cli/src/commands/mod.rs index 7193ddee8..45329a503 100644 --- a/crates/socket-patch-cli/src/commands/mod.rs +++ b/crates/socket-patch-cli/src/commands/mod.rs @@ -96,9 +96,18 @@ pub(crate) async fn discover_wiring( common: &crate::args::GlobalArgs, root: &Path, ) -> socket_patch_core::vex::discover::Discovery { + #[cfg(test)] + DISCOVERIES.with(|n| n.set(n.get() + 1)); socket_patch_core::vex::discover_patched_refs_with(root, &discover_options(common)).await } +#[cfg(test)] +thread_local! { + /// How many times this thread ran [`discover_wiring`]: discovery walks + /// every lockfile, so tests pin the paths that must not repeat it. + pub(crate) static DISCOVERIES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + /// [`discover_wiring`] of the snapshot's root, reading through `snapshot`. pub(crate) async fn discover_wiring_in( common: &crate::args::GlobalArgs, @@ -128,13 +137,8 @@ pub(crate) async fn hosted_inventory( ) } -/// The project's hosted state, v5-style: v5 hosted mode keeps no ledger, -/// so the hosted pins [`discover_wiring`] finds in the lockfiles are the -/// whole record. Shaped as a [`RedirectState`] for the readers that classify -/// hosted against vendored state (one uuid-only record per pinned purl, no -/// edits) — it is never persisted. -/// -/// [`RedirectState`]: socket_patch_core::patch::redirect::RedirectState +/// [`hosted_state_from_pins`] over a fresh [`discover_wiring`] of `root` +/// (the unit tests' load-then-derive entry point). #[cfg(test)] pub(crate) async fn hosted_state_from_lockfiles( common: &crate::args::GlobalArgs, @@ -147,8 +151,14 @@ pub(crate) async fn hosted_state_from_lockfiles( ) } -/// [`hosted_state_from_lockfiles`] over already-discovered pins. A purl -/// pinned to several uuids (different lockfiles) keeps the first. +/// The project's hosted state, v5-style: v5 hosted mode keeps no ledger, +/// so the hosted pins [`discover_wiring`] finds in the lockfiles are the +/// whole record. Shaped as a [`RedirectState`] for the readers that classify +/// hosted against vendored state (one uuid-only record per pinned purl, no +/// edits) — it is never persisted. A purl pinned to several uuids +/// (different lockfiles) keeps the first. +/// +/// [`RedirectState`]: socket_patch_core::patch::redirect::RedirectState pub(crate) fn hosted_state_from_pins( pins: &[socket_patch_core::patch::redirect::upstream::HostedPin], ) -> socket_patch_core::patch::redirect::RedirectState { diff --git a/crates/socket-patch-cli/src/commands/remove.rs b/crates/socket-patch-cli/src/commands/remove.rs index 71e86ead6..748472791 100644 --- a/crates/socket-patch-cli/src/commands/remove.rs +++ b/crates/socket-patch-cli/src/commands/remove.rs @@ -102,6 +102,19 @@ fn print_hosted_leg_warnings(common: &GlobalArgs, warnings: &[(String, String)]) } } +/// `--preserve-state` restored hosted pins anyway (hosted has no +/// preservable local state): say so on stderr (`Note: …`, never under +/// `--silent` / `--json`) and return the `hosted_state_not_preservable` +/// run warning for the envelope's `warnings[]` — the same code +/// `rollback --preserve-state` reports. +fn note_hosted_state_not_preservable(common: &GlobalArgs) -> (String, String) { + let warning = super::rollback::hosted_state_not_preservable_warning(); + if !common.silent && !common.json { + eprintln!("Note: {}.", warning.1); + } + warning +} + /// Emit a `remove` error envelope and return. Used by the many error /// paths in `run` so they all share the same JSON shape. `dry_run` rides /// the envelope so preview failures report `dryRun: true`. @@ -785,11 +798,8 @@ pub async fn run(args: RemoveArgs) -> i32 { return 1; } }; - if args.preserve_state && !leg.reverted.is_empty() && loud { - eprintln!( - "Note: hosted wiring has no preservable local state; its lockfile pins \ - now resolve upstream." - ); + if args.preserve_state && !leg.reverted.is_empty() { + hosted_leg_warnings.push(note_hosted_state_not_preservable(&args.common)); } // `run_hosted_leg` printed one line per restored purl. printed_progress |= loud && !leg.reverted.is_empty(); @@ -1432,7 +1442,11 @@ async fn remove_hosted_only( } else { PatchAction::Removed }; - for (code, detail) in &leg.warnings { + let mut warnings = leg.warnings.clone(); + if args.preserve_state && !leg.reverted.is_empty() { + warnings.push(note_hosted_state_not_preservable(&args.common)); + } + for (code, detail) in &warnings { env.warnings.push(crate::json_envelope::RunWarning { code: code.clone(), detail: detail.clone(), diff --git a/crates/socket-patch-cli/src/commands/rollback.rs b/crates/socket-patch-cli/src/commands/rollback.rs index 1dd1fbb2a..a9d289190 100644 --- a/crates/socket-patch-cli/src/commands/rollback.rs +++ b/crates/socket-patch-cli/src/commands/rollback.rs @@ -76,6 +76,19 @@ pub(crate) fn join_clauses(clauses: &[String]) -> String { } } +/// The `hosted_state_not_preservable` run warning: a `--preserve-state` +/// run (rollback or remove) restored hosted pins to upstream anyway — the +/// lockfile pins are hosted mode's only record, so there is no local +/// state to keep. +pub(crate) fn hosted_state_not_preservable_warning() -> (String, String) { + ( + "hosted_state_not_preservable".into(), + "hosted wiring has no preservable local state: the lockfile pins are the only \ + record, and they now resolve upstream; re-run `scan --mode hosted` to re-wire" + .into(), + ) +} + /// Capitalize the first character and end with `?`. pub(crate) fn as_question(text: &str) -> String { if text.is_empty() { @@ -1490,13 +1503,7 @@ pub async fn run(args: RollbackArgs) -> i32 { )); } if args.preserve_state && !hosted_leg.reverted.is_empty() { - run_warnings.push(( - "hosted_state_not_preservable".into(), - "hosted wiring has no preservable local state: the lockfile pins are \ - the only record, and they now resolve upstream; re-run \ - `scan --mode hosted` to re-wire" - .into(), - )); + run_warnings.push(hosted_state_not_preservable_warning()); } if !path_scope.is_empty() { let scope = path_scope.bind(&cwd); diff --git a/crates/socket-patch-cli/src/commands/scan/hosted.rs b/crates/socket-patch-cli/src/commands/scan/hosted.rs index 11bee9f8d..1663859db 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted.rs @@ -1419,48 +1419,54 @@ pub(crate) async fn run_redirect_selected( // Classified over the lockfiles as this run left them and the vendored // ledger as the takeover left it. let mut takeover_warnings: Vec = Vec::new(); - // The lockfiles as this run left them: the gate's (or scan's) discovery - // when it provably describes them, else a fresh one. A takeover's - // reverts are writes too (the gate's discovery saw them in the overlay; - // a dry run drops them). - let created: Vec<&str> = created_paths.iter().map(String::as_str).collect(); - let written = if takeover_migrated.is_empty() - && !rewrite - .files - .keys() - .chain(rewrite.binary_files.keys()) - .any(|rel| !socket_patch_core::patch::redirect::sbt::is_synthetic_key(rel)) - { - Written::Nothing - } else if common.dry_run { - Written::Previewed + // Nothing vendored, nothing to overlap: skip the lockfile walk (#993). + let vendor_now = vendor_state.as_ref().ok().filter(|v| !v.entries.is_empty()); + let superseded = if vendor_now.is_none() { + Vec::new() } else { - Written::Landed { created: &created } - }; - let fresh_now; - let discovery_now = match discovery_after_writes( - prior_discovery, - done.final_discovery.as_ref(), - written, - vlt_stale.healed_store.as_ref(), - ) { - Some(discovery) => discovery, - None => { - fresh_now = crate::commands::discover_wiring(common, &common.cwd).await; - &fresh_now - } + // The lockfiles as this run left them: the gate's (or scan's) discovery + // when it provably describes them, else a fresh one. A takeover's + // reverts are writes too (the gate's discovery saw them in the overlay; + // a dry run drops them). + let created: Vec<&str> = created_paths.iter().map(String::as_str).collect(); + let written = if takeover_migrated.is_empty() + && !rewrite + .files + .keys() + .chain(rewrite.binary_files.keys()) + .any(|rel| !socket_patch_core::patch::redirect::sbt::is_synthetic_key(rel)) + { + Written::Nothing + } else if common.dry_run { + Written::Previewed + } else { + Written::Landed { created: &created } + }; + let fresh_now; + let discovery_now = match discovery_after_writes( + prior_discovery, + done.final_discovery.as_ref(), + written, + vlt_stale.healed_store.as_ref(), + ) { + Some(discovery) => discovery, + None => { + fresh_now = crate::commands::discover_wiring(common, &common.cwd).await; + &fresh_now + } + }; + let hosted_now = crate::commands::hosted_state_from_pins( + &socket_patch_core::patch::redirect::upstream::HostedPin::all(discovery_now), + ); + super::classify_overlap_takeover_with( + &common.cwd, + Some(&hosted_now), + vendor_now, + discovery_now, + ) + .await + .redirect }; - let hosted_now = crate::commands::hosted_state_from_pins( - &socket_patch_core::patch::redirect::upstream::HostedPin::all(discovery_now), - ); - let superseded = super::classify_overlap_takeover_with( - &common.cwd, - Some(&hosted_now), - vendor_state.as_ref().ok(), - discovery_now, - ) - .await - .redirect; if !superseded.is_empty() { takeover_warnings.push(serde_json::json!({ "code": super::REDIRECT_SUPERSEDES_VENDORED, diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index 2a73c0d74..a27f058d4 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -874,7 +874,7 @@ pub(super) const REDIRECT_PRUNE_IGNORED_DETAIL: &str = `scan --mode vendored --prune` to garbage-collect"; /// The PURLs claimed by BOTH a hosted pin (`redirect`, the lockfiles' -/// hosted state — see [`crate::commands::hosted_state_from_lockfiles`]) and +/// hosted state — see [`crate::commands::hosted_state_from_pins`]) and /// the vendored state ledger (`.socket/vendor/state.json`), sorted. A /// non-empty result means one of the two is stale for each PURL (a /// lockfile entry can point only one way). `None`, an empty vendor ledger, @@ -920,19 +920,29 @@ pub(super) async fn classify_overlap_takeover(common: &GlobalArgs, cwd: &Path) - // A malformed vendor ledger classifies like a missing one (this path // only feeds takeover warnings; corruption is a hard error on the // write/attest paths). + let vendor = socket_patch_core::vendor::load_state(cwd).await.ok(); + // Nothing vendored, nothing to overlap: skip the lockfile walk (#993). + let Some(vendor) = vendor.filter(|v| !v.entries.is_empty()) else { + return OverlapTakeover::default(); + }; let discovery = crate::commands::discover_wiring(common, cwd).await; let redirect = crate::commands::hosted_state_from_pins( &socket_patch_core::patch::redirect::upstream::HostedPin::all(&discovery), ); - let vendor = socket_patch_core::vendor::load_state(cwd).await.ok(); - classify_overlap_takeover_with(cwd, Some(&redirect), vendor.as_ref(), &discovery).await + classify_overlap_takeover_with(cwd, Some(&redirect), Some(&vendor), &discovery).await } /// [`classify_overlap_takeover`] over already-loaded state (the hosted /// engine classifies against its post-takeover vendor ledger) and /// `discovery`, the lockfile discovery of `cwd` as it is now -/// ([`crate::commands::discover_wiring`]). `None` for either state yields -/// no overlap. +/// ([`crate::commands::discover_wiring`]). `None` for either state, or an +/// empty vendored ledger, yields no overlap. +/// +/// Callers hand in a discovery they already hold and should skip +/// discovering at all when the vendored ledger has no entries: discovery +/// re-walks every lockfile of the project, which is a real share of a +/// hosted scan's time (#993), and a project that never vendored (the +/// hosted common case) has nothing for it to decide. pub(super) async fn classify_overlap_takeover_with( cwd: &Path, redirect: Option<&socket_patch_core::patch::redirect::RedirectState>, @@ -940,7 +950,7 @@ pub(super) async fn classify_overlap_takeover_with( discovery: &socket_patch_core::vex::discover::Discovery, ) -> OverlapTakeover { let mut out = OverlapTakeover::default(); - let Some(vendor) = vendor else { + let Some(vendor) = vendor.filter(|v| !v.entries.is_empty()) else { return out; }; let overlap = overlap_from_states(redirect, vendor); @@ -1296,7 +1306,7 @@ async fn gradle_scan( /// The scanned purls whose HOSTED redirect wiring is still live: a hosted /// pin names the purl (`redirect_state`, the lockfiles' hosted state — see -/// [`crate::commands::hosted_state_from_lockfiles`]) AND lockfile discovery +/// [`crate::commands::hosted_state_from_pins`]) AND lockfile discovery /// proves the current lockfile still routes it to that hosted patch — core /// `Discovery::redirect_record_live`, the same liveness rule `vex` gates /// hosted attestations on. @@ -1693,6 +1703,15 @@ async fn run_scan( explicit, args.common.is_global(), )); + // Agent and report-only scans patch the crawled copies in place, so a + // copy under a nested project's `node_modules` is judged by that + // project's root (#554). Hosted and vendored scans only rewire this + // root's lockfiles. + if !args.common.is_global() + && !matches!(args.mode, Some(ScanMode::Hosted) | Some(ScanMode::Vendored)) + { + policy.judge_nested_roots(invocation, &args.common.cwd); + } // Positional PATH globs (see `ScanArgs::paths`). An unparseable glob // is a usage error, same exit-2 shape as the mode conflicts. @@ -1995,10 +2014,16 @@ async fn run_scan( // The socket.yml root/ecosystem/package filters, after the flags // (which only narrow further) and after the prune-universe capture. - let filtered_crawled: Vec<_> = filtered_crawled - .into_iter() - .filter(|pkg| policy.admit_crawled(&pkg.purl)) - .collect(); + if policy.judges_nested_roots() && args.common.ecosystem_selected(Ecosystem::Npm) { + let nm_roots = socket_patch_core::crawlers::NpmCrawler::new() + .get_node_modules_paths(&crawler_options) + .await + .unwrap_or_default(); + policy + .locate_nested_copies(&nm_roots, &filtered_crawled) + .await; + } + let filtered_crawled = policy.admit_crawled_copies(filtered_crawled, &supplement_purls); // Gradle discovery notes (m2 gating, the user home) ride the run-level // warnings; the lock set only annotates `packages[]` below. @@ -4439,6 +4464,43 @@ mod tests { assert!(takeover.vendored.is_empty(), "{takeover:?}"); } + /// The takeover classifier runs after every hosted rewrite, so it must + /// not re-walk the lockfiles when no vendored ledger entry could + /// overlap (the hosted common case), and walks them once otherwise + /// (#993: it used to discover twice — once for the hosted pins, once + /// for liveness — and even with no vendored ledger at all). + #[tokio::test] + async fn takeover_classifier_discovers_at_most_once() { + let discoveries = || crate::commands::DISCOVERIES.with(|n| n.get()); + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + write_lock_pointing_at_hosted(root, "minimist", "1.2.2").await; + + // No vendored ledger, then an empty one: nothing to overlap. + for write_empty in [false, true] { + if write_empty { + socket_patch_core::vendor::save_state(root, &VendorState::new()) + .await + .unwrap(); + } + let before = discoveries(); + assert_eq!( + classify_overlap_takeover(&common_at(root), root).await, + OverlapTakeover::default() + ); + assert_eq!(discoveries(), before, "no vendored entry, no discovery"); + } + + write_vendor_ledger_wired(root, &["pkg:npm/minimist@1.2.2"]).await; + let before = discoveries(); + let takeover = classify_overlap_takeover(&common_at(root), root).await; + assert_eq!(discoveries(), before + 1, "one discovery serves both sides"); + assert_eq!( + takeover.redirect, + vec!["pkg:npm/minimist@1.2.2".to_string()] + ); + } + #[tokio::test] async fn half_migrated_locks_naming_both_stay_silent() { // One lockfile pins minimist hosted while another still routes it to diff --git a/crates/socket-patch-cli/src/commands/scan/policy.rs b/crates/socket-patch-cli/src/commands/scan/policy.rs index 8a6711844..17f795c0d 100644 --- a/crates/socket-patch-cli/src/commands/scan/policy.rs +++ b/crates/socket-patch-cli/src/commands/scan/policy.rs @@ -9,6 +9,7 @@ use std::sync::Mutex; use socket_patch_core::api::ranking::cmp_search_results; use socket_patch_core::api::types::PatchSearchResult; +use socket_patch_core::crawlers::types::CrawledPackage; use socket_patch_core::manifest::schema::PatchManifest; use socket_patch_core::policy::{ find_repo_root_with_warnings, patch_severity_order, policy_block, repo_relative_checked, @@ -81,21 +82,7 @@ pub(crate) fn load_invocation_policy(args: &ScanArgs) -> Result Vec { - let mut markers: Vec = std::fs::read_dir(dir) - .map(|entries| { - entries - .filter_map(|e| e.ok()) - .filter(|e| e.file_type().is_ok_and(|t| t.is_file() || t.is_symlink())) - .filter_map(|e| e.file_name().into_string().ok()) - .filter(|name| { - marker_ecosystem(name).is_some() - || UNSUPPORTED_MARKERS - .iter() - .any(|(_, names)| names.contains(&name.as_str())) - }) - .collect() - }) - .unwrap_or_default(); + let mut markers = lock_markers(dir); if markers.is_empty() { // No lockfile: the manifests say what the project is. Every name // here, JVM build files included, must be a regular file: the @@ -113,6 +100,26 @@ pub(crate) fn dir_markers(dir: &Path) -> Vec { markers } +/// The lockfile markers in `dir` (unsorted): what makes a directory a +/// project root of its own rather than a member of an enclosing one. +fn lock_markers(dir: &Path) -> Vec { + std::fs::read_dir(dir) + .map(|entries| { + entries + .filter_map(|e| e.ok()) + .filter(|e| e.file_type().is_ok_and(|t| t.is_file() || t.is_symlink())) + .filter_map(|e| e.file_name().into_string().ok()) + .filter(|name| { + marker_ecosystem(name).is_some() + || UNSUPPORTED_MARKERS + .iter() + .any(|(_, names)| names.contains(&name.as_str())) + }) + .collect() + }) + .unwrap_or_default() +} + /// Manifests that stand in as markers for a root with no lockfile (plus /// every JVM build file, `layout::JVM_PROJECT_MARKERS`). const MANIFEST_MARKERS: [&str; 6] = [ @@ -131,9 +138,35 @@ struct Report { retained_purls: BTreeSet, filtered_purls: HashSet, update_purls: HashSet, + /// Admitted purls with a copy under a nested project root the policy + /// skips: canonical purl -> those roots (see + /// [`ScanPolicy::admit_crawled_copies`]). + shared_copies: BTreeMap>, + /// The [`Self::shared_copies`] entries a patch was selected for. + shared_selected: BTreeSet, human_printed: bool, } +/// A copy's owning root: its repo-relative dir and the root filter's verdict. +type Owner = (String, Result<(), FilterReason>); + +/// The nested project roots under an agent / report-only scan's root +/// (#554): such a scan crawls every nested project's `node_modules` and +/// patches copies in place, so each copy is judged by the root that owns +/// it, not only the scan root. +struct NestedRoots { + repo_root: PathBuf, + /// The scan root as given and canonicalized (crawled paths are built + /// from the former). + scan_dirs: Vec, + /// Owning root per `node_modules` parent directory: `None` for the scan + /// root, else the root's repo-relative dir and verdict. + owners: HashMap>, + /// More installed copies per purl ([`ScanPolicy::locate_nested_copies`]): + /// the crawl keeps one copy per purl. + more_copies: HashMap>, +} + /// One project root's view of the invocation policy (disk scans run one /// root per `run_scan`). pub(crate) struct ScanPolicy { @@ -147,6 +180,7 @@ pub(crate) struct ScanPolicy { root_verdict: Result<(), FilterReason>, /// Recorded patches of this root: canonical purl → uuid. recorded: HashMap, + nested: Option, report: Mutex, } @@ -200,10 +234,230 @@ impl ScanPolicy { project, root_verdict, recorded: HashMap::new(), + nested: None, report: Mutex::new(report), } } + /// Judge crawled copies under a nested project's `node_modules` by that + /// project's own root (agent and report-only scans, which patch the + /// crawled copies in place, see [`NestedRoots`]). A nested root is a + /// directory below `scan_dir` holding a lockfile; a member without one + /// belongs to the enclosing root. Nested roots are discovered, so the + /// built-in default ignores apply to them. + pub(crate) fn judge_nested_roots(&mut self, invocation: &InvocationPolicy, scan_dir: &Path) { + let mut scan_dirs = vec![scan_dir.to_path_buf()]; + if let Ok(canonical) = std::fs::canonicalize(scan_dir) { + if canonical != scan_dir { + scan_dirs.push(canonical); + } + } + self.nested = Some(NestedRoots { + repo_root: invocation.repo_root.clone(), + scan_dirs, + owners: HashMap::new(), + more_copies: HashMap::new(), + }); + } + + /// Whether [`Self::judge_nested_roots`] is on. + pub(crate) fn judges_nested_roots(&self) -> bool { + self.nested.is_some() + } + + /// The nested root owning the crawled copy at `path` (`None`: the scan + /// root owns it). A newly seen root filtered as a whole is reported as + /// one entry, like the scan root. + fn nested_owner(&mut self, path: &Path) -> Option { + let nested = self.nested.as_mut()?; + let (scan_dir, rel) = nested + .scan_dirs + .iter() + .find_map(|dir| path.strip_prefix(dir).ok().map(|rel| (dir.clone(), rel)))?; + // Only the directories above the first `node_modules`: anything + // inside one is a package, not a project. + let parts: Vec<&std::ffi::OsStr> = rel + .components() + .map_while(|c| match c { + std::path::Component::Normal(part) => Some(part), + _ => None, + }) + .collect(); + let depth = parts.iter().position(|p| *p == "node_modules")?; + let mut dir = scan_dir.clone(); + dir.extend(&parts[..depth]); + if let Some(owner) = nested.owners.get(&dir) { + return owner.clone(); + } + let owner = (1..=depth).rev().find_map(|k| { + let mut root = scan_dir.clone(); + root.extend(&parts[..k]); + if lock_markers(&root).is_empty() { + return None; + } + let canonical = std::fs::canonicalize(&root).unwrap_or(root.clone()); + let project = repo_relative_checked(&nested.repo_root, &canonical)?; + let markers = dir_markers(&root); + let verdict = self.policy.admits_root(&Root { + rel_dir: &project, + markers: &markers, + explicit: false, + }); + Some((project, verdict)) + }); + if let Some((project, Err(reason))) = &owner { + let mut report = self.report.lock().unwrap_or_else(|e| e.into_inner()); + let known = report + .filtered + .iter() + .any(|f| f.purl.is_none() && &f.project == project); + if !known { + report.filtered.push(FilteredEntry { + purl: None, + uuid: None, + project: project.clone(), + reason: reason.clone(), + severity: None, + }); + } + } + nested.owners.insert(dir, owner.clone()); + owner + } + + /// The owner of a crawled copy: its nested root, else the scan root. + fn copy_owner(&mut self, path: &Path) -> Owner { + self.nested_owner(path) + .unwrap_or_else(|| (self.project.clone(), self.root_verdict.clone())) + } + + /// The crawl keeps one installed copy per purl, but a package can be + /// installed under several roots. When some `node_modules` roots + /// (`nm_roots`, as the npm crawler walks them) are skipped and others + /// are not, find the other copies of each npm package whose crawled + /// copy sits on the other side, so [`Self::admit_crawled_copies`] sees + /// every root that installs it. + pub(crate) async fn locate_nested_copies( + &mut self, + nm_roots: &[PathBuf], + pkgs: &[CrawledPackage], + ) { + if self.nested.is_none() { + return; + } + let roots: Vec<(PathBuf, bool)> = nm_roots + .iter() + .map(|root| (root.clone(), self.copy_owner(root).1.is_ok())) + .collect(); + if roots.iter().all(|(_, ok)| *ok) || roots.iter().all(|(_, ok)| !*ok) { + return; + } + let mut admitted_purls = Vec::new(); + let mut skipped_purls = Vec::new(); + for pkg in pkgs.iter().filter(|p| p.purl.starts_with("pkg:npm/")) { + if self.copy_owner(&pkg.path).1.is_ok() { + admitted_purls.push(pkg.purl.clone()); + } else { + skipped_purls.push(pkg.purl.clone()); + } + } + let crawler = socket_patch_core::crawlers::NpmCrawler::new(); + let mut more: HashMap> = HashMap::new(); + for (root, ok) in roots { + // An admitted root may hold a copy of a package crawled under a + // skipped one, and the reverse. + let wanted = if ok { &skipped_purls } else { &admitted_purls }; + if wanted.is_empty() { + continue; + } + let Ok(found) = crawler.find_by_purls(&root, wanted).await else { + continue; + }; + for (purl, copies) in found { + more.entry(purl) + .or_default() + .extend(copies.into_iter().map(|c| c.path)); + } + } + if let Some(nested) = self.nested.as_mut() { + nested.more_copies = more; + } + } + + /// Step 3 over the whole crawl: [`Self::admit_crawled`], with each copy + /// judged by its owning root when nested roots are on. A package stays + /// when any copy's root admits it: patches are recorded per package + /// version, so its copies under skipped roots are patched too (noted + /// once a patch is selected for it). Supplement purls (`supplements`, + /// no installed copy) belong to the scan root. + pub(crate) fn admit_crawled_copies( + &mut self, + pkgs: Vec, + supplements: &HashSet, + ) -> Vec { + if self.nested.is_none() { + return pkgs + .into_iter() + .filter(|p| self.admit_crawled(&p.purl)) + .collect(); + } + // Each purl's first admitting root, else its first root. + let mut owners: BTreeMap = BTreeMap::new(); + let mut skipped: BTreeMap> = BTreeMap::new(); + let more_copies = self + .nested + .as_mut() + .map(|n| std::mem::take(&mut n.more_copies)) + .unwrap_or_default(); + for pkg in &pkgs { + let mut paths: Vec<&Path> = vec![&pkg.path]; + if supplements.contains(&pkg.purl) { + paths.clear(); + } else if let Some(more) = more_copies.get(&pkg.purl) { + paths.extend(more.iter().map(PathBuf::as_path)); + } + let mut copy_owners: Vec = paths + .into_iter() + .map(|path| self.copy_owner(path)) + .collect(); + if copy_owners.is_empty() { + copy_owners.push((self.project.clone(), self.root_verdict.clone())); + } + for (project, verdict) in copy_owners { + if verdict.is_err() { + skipped + .entry(pkg.purl.clone()) + .or_default() + .insert(project.clone()); + } + let slot = owners + .entry(pkg.purl.clone()) + .or_insert_with(|| (project.clone(), verdict.clone())); + if slot.1.is_err() && verdict.is_ok() { + *slot = (project, verdict); + } + } + } + let mut admitted = HashSet::new(); + for (purl, (project, verdict)) in owners { + let root_ok = verdict.is_ok(); + if !self.admit_in(&purl, &project, verdict) { + continue; + } + if root_ok && self.policy.admits_purl(&purl).is_ok() { + if let Some(projects) = skipped.remove(&purl) { + self.report() + .shared_copies + .insert(PurlKey::new(&purl).into_string(), projects); + } + } + admitted.insert(purl); + } + pkgs.into_iter() + .filter(|p| admitted.contains(&p.purl)) + .collect() + } + /// Whether selection can filter anything (a floor, or patching /// disabled): report-only runs select only for the report then. pub(crate) fn reports_selection(&self) -> bool { @@ -237,7 +491,9 @@ impl ScanPolicy { } fn recorded_uuid(&self, purl: &str) -> Option<&str> { - self.recorded.get(&PurlKey::new(purl).into_string()).map(String::as_str) + self.recorded + .get(&PurlKey::new(purl).into_string()) + .map(String::as_str) } /// Step 3: the root, ecosystem and package filters. Returns whether the @@ -245,10 +501,14 @@ impl ScanPolicy { /// exclude stays in the query (so `upgradeAvailable` can be reported) /// but joins the retained set, which never reaches a writer. pub(crate) fn admit_crawled(&self, purl: &str) -> bool { - let verdict = self - .root_verdict - .clone() - .and_then(|()| self.policy.admits_purl(purl)); + self.admit_in(purl, &self.project, self.root_verdict.clone()) + } + + /// [`Self::admit_crawled`] for a copy owned by the root `project` + /// with the verdict `root_verdict`. + fn admit_in(&self, purl: &str, project: &str, root_verdict: Result<(), FilterReason>) -> bool { + let root_excluded = root_verdict.is_err(); + let verdict = root_verdict.and_then(|()| self.policy.admits_purl(purl)); let reason = match verdict { Ok(()) => return true, Err(reason) => reason, @@ -259,7 +519,7 @@ impl ScanPolicy { if report.retained_purls.insert(key.clone()) { report.retained.push(RetainedEntry { purl: key, - project: self.project.clone(), + project: project.to_string(), recorded_uuid: uuid.to_string(), reason, upgrade_available: false, @@ -267,13 +527,16 @@ impl ScanPolicy { } return true; } - if self.root_verdict.is_err() { + if root_excluded { // Already reported as the root's one entry. - } else if report.filtered_purls.insert(PurlKey::new(purl).into_string()) { + } else if report + .filtered_purls + .insert(PurlKey::new(purl).into_string()) + { report.filtered.push(FilteredEntry { purl: Some(PurlKey::new(purl).into_string()), uuid: None, - project: self.project.clone(), + project: project.to_string(), reason, severity: None, }); @@ -284,7 +547,10 @@ impl ScanPolicy { /// Record the purls with a newer patch (`updates[]`), for /// `retained[].upgradeAvailable`. pub(crate) fn set_update_purls<'a>(&self, purls: impl IntoIterator) { - self.report().update_purls = purls.into_iter().map(|p| PurlKey::new(p).into_string()).collect(); + self.report().update_purls = purls + .into_iter() + .map(|p| PurlKey::new(p).into_string()) + .collect(); } /// Steps 5-6: group the tier-accessible offers, keep retained packages @@ -298,7 +564,10 @@ impl ScanPolicy { { let report = self.report(); for offer in accessible { - if report.retained_purls.contains(&PurlKey::new(&offer.purl).into_string()) { + if report + .retained_purls + .contains(&PurlKey::new(&offer.purl).into_string()) + { continue; } grouped.entry(offer.purl.clone()).or_default().push(offer); @@ -371,6 +640,14 @@ impl ScanPolicy { } } if let Some(i) = chosen { + if report + .shared_copies + .contains_key(&PurlKey::new(&purl).into_string()) + { + report + .shared_selected + .insert(PurlKey::new(&purl).into_string()); + } offers.selected.insert(purl.clone(), group[i].clone()); } offers.unfiltered.insert(purl, group); @@ -402,7 +679,7 @@ impl ScanPolicy { .entry("warnings") .or_insert_with(|| serde_json::json!([])); if let Some(arr) = warnings.as_array_mut() { - for w in &self.warnings { + for w in self.warnings.iter().chain(&self.shared_copy_warnings()) { let present = arr .iter() .any(|e| e["code"] == w.code && e["detail"] == w.detail.as_str()); @@ -416,6 +693,23 @@ impl ScanPolicy { } } + /// One `policy_shared_copy` warning per selected package that is also + /// installed under a nested root the policy skips. + fn shared_copy_warnings(&self) -> Vec { + let report = self.report(); + report + .shared_selected + .iter() + .filter_map(|purl| { + let projects = report.shared_copies.get(purl)?; + Some(PolicyWarning { + code: POLICY_SHARED_COPY, + detail: shared_copy_detail(purl, projects), + }) + }) + .collect() + } + /// Print the policy warnings (stderr) once, human path. pub(crate) fn print_warnings(&self, silent: bool) { if silent || !self.announce_warnings { @@ -465,8 +759,8 @@ impl ScanPolicy { } let what = match &f.purl { Some(purl) => sanitize(&normalize_purl(purl)), - None if self.project.is_empty() => "this project".to_string(), - None => format!("project {}", sanitize(&self.project)), + None if f.project.is_empty() => "this project".to_string(), + None => format!("project {}", sanitize(&f.project)), }; let severity = f .severity @@ -475,6 +769,11 @@ impl ScanPolicy { .unwrap_or_default(); println!(" skipped {what}{severity}: {}", f.reason.detail()); } + for purl in &report.shared_selected { + if let Some(projects) = report.shared_copies.get(purl) { + println!(" note: {}", shared_copy_detail(purl, projects)); + } + } if verbose { for r in &report.retained { println!( @@ -488,6 +787,30 @@ impl ScanPolicy { } } +/// Warning code: a selected package is also installed under a nested +/// project root the policy skips, and that copy is patched too. +pub(crate) const POLICY_SHARED_COPY: &str = "policy_shared_copy"; + +fn shared_copy_detail(purl: &str, projects: &BTreeSet) -> String { + let projects: Vec = projects + .iter() + .map(|p| { + if p.is_empty() { + "the repo root".to_string() + } else { + sanitize(p) + } + }) + .collect(); + format!( + "{} is patched for an included project, so its installed copy under skipped project{} {} \ + is patched too (patches are recorded per package version)", + sanitize(&normalize_purl(purl)), + if projects.len() == 1 { "" } else { "s" }, + projects.join(", ") + ) +} + /// The JSON error object for a policy file that cannot be honored: scan's /// error shape, `error: {code, message}`. pub(crate) fn policy_error_json(err: &PolicyError, paths: &[String]) -> serde_json::Value { @@ -575,3 +898,188 @@ pub(crate) fn policy_bypass_warnings( } out } + +#[cfg(test)] +mod tests { + use super::*; + + fn invocation(repo: &Path, yml: &str) -> InvocationPolicy { + std::fs::write(repo.join("socket.yml"), yml).unwrap(); + let (policy, warnings) = SelectionPolicy::load( + &DiskPolicyFs::new(repo), + &socket_patch_core::policy::PolicyOverrides::default(), + ) + .unwrap(); + InvocationPolicy { + policy, + repo_root: repo.to_path_buf(), + warnings, + warned: Default::default(), + } + } + + fn copy(dir: &Path, name: &str) -> CrawledPackage { + let path = dir.join("node_modules").join(name); + std::fs::create_dir_all(&path).unwrap(); + std::fs::write( + path.join("package.json"), + format!(r#"{{"name": "{name}", "version": "1.0.0"}}"#), + ) + .unwrap(); + CrawledPackage { + name: name.to_string(), + version: "1.0.0".to_string(), + namespace: None, + purl: format!("pkg:npm/{name}@1.0.0"), + path, + } + } + + fn project(dir: &Path, lock: bool) { + std::fs::create_dir_all(dir).unwrap(); + std::fs::write(dir.join("package.json"), "{}").unwrap(); + if lock { + std::fs::write(dir.join("package-lock.json"), "{}").unwrap(); + } + } + + /// #554: copies under a nested project's `node_modules` are judged by + /// that project's root, not only the scan root. + #[test] + fn nested_copies_follow_their_own_root() { + let tmp = tempfile::tempdir().unwrap(); + let repo = std::fs::canonicalize(tmp.path()).unwrap(); + project(&repo, true); + project(&repo.join("services/legacy"), true); + project(&repo.join("tests/e2e"), true); + // A lockless workspace member belongs to the repo root. + project(&repo.join("packages/member"), false); + let invocation = invocation( + &repo, + "version: 2\npatches:\n ignorePaths: [\"/services/\"]\n", + ); + let mut policy = ScanPolicy::for_root(&invocation, &repo, true, false); + policy.judge_nested_roots(&invocation, &repo); + let crawl = vec![ + copy(&repo, "shared"), + copy(&repo.join("services/legacy"), "shared"), + copy(&repo.join("services/legacy"), "legacy-only"), + copy( + &repo.join("services/legacy/node_modules/legacy-only"), + "deep", + ), + copy(&repo.join("tests/e2e"), "fixture-only"), + copy(&repo.join("packages/member"), "member-dep"), + ]; + let kept: Vec = policy + .admit_crawled_copies(crawl, &HashSet::new()) + .into_iter() + .map(|p| p.purl) + .collect(); + assert_eq!( + kept, + [ + "pkg:npm/shared@1.0.0", + "pkg:npm/shared@1.0.0", + "pkg:npm/member-dep@1.0.0" + ] + ); + let doc = policy.json(); + let roots: Vec<(&str, &str)> = doc["filtered"] + .as_array() + .unwrap() + .iter() + .map(|f| { + ( + f["project"].as_str().unwrap(), + f["reason"].as_str().unwrap(), + ) + }) + .collect(); + assert_eq!( + roots, + [ + ("services/legacy", "policy_path_excluded"), + ("tests/e2e", "policy_path_excluded") + ] + ); + // The shared package is noted only once a patch is selected for it. + assert!(policy.shared_copy_warnings().is_empty()); + let offer: PatchSearchResult = serde_json::from_value(serde_json::json!({ + "uuid": "11111111-1111-4111-8111-111111111111", + "purl": "pkg:npm/shared@1.0.0", + "publishedAt": "2024-01-01T00:00:00Z", + "description": "d", + "license": "MIT", + "tier": "free", + "vulnerabilities": {} + })) + .unwrap(); + policy.select(vec![offer]); + let warnings = policy.shared_copy_warnings(); + assert_eq!(warnings.len(), 1); + assert_eq!(warnings[0].code, POLICY_SHARED_COPY); + assert!( + warnings[0].detail.contains("services/legacy"), + "{}", + warnings[0].detail + ); + } + + /// The crawl keeps one copy per purl: when that copy sits under a + /// skipped root, the copy under an admitted root is looked up. + #[tokio::test] + async fn a_package_crawled_under_a_skipped_root_is_found_under_an_admitted_one() { + let tmp = tempfile::tempdir().unwrap(); + let repo = std::fs::canonicalize(tmp.path()).unwrap(); + project(&repo, true); + project(&repo.join("services/legacy"), true); + let invocation = invocation( + &repo, + "version: 2\npatches:\n ignorePaths: [\"/services/\"]\n", + ); + let mut policy = ScanPolicy::for_root(&invocation, &repo, true, false); + policy.judge_nested_roots(&invocation, &repo); + copy(&repo, "shared"); + let crawl = vec![ + copy(&repo.join("services/legacy"), "shared"), + copy(&repo.join("services/legacy"), "legacy-only"), + ]; + let nm_roots = [ + repo.join("node_modules"), + repo.join("services/legacy/node_modules"), + ]; + policy.locate_nested_copies(&nm_roots, &crawl).await; + let kept: Vec = policy + .admit_crawled_copies(crawl, &HashSet::new()) + .into_iter() + .map(|p| p.purl) + .collect(); + assert_eq!(kept, ["pkg:npm/shared@1.0.0"]); + assert_eq!( + policy.report().shared_copies.get("pkg:npm/shared@1.0.0"), + Some(&BTreeSet::from(["services/legacy".to_string()])) + ); + } + + /// Without nested roots (hosted / vendored), every copy follows the + /// scan root as before. + #[test] + fn without_nested_roots_every_copy_follows_the_scan_root() { + let tmp = tempfile::tempdir().unwrap(); + let repo = std::fs::canonicalize(tmp.path()).unwrap(); + project(&repo, true); + project(&repo.join("services/legacy"), true); + let invocation = invocation( + &repo, + "version: 2\npatches:\n ignorePaths: [\"/services/\"]\n", + ); + let mut policy = ScanPolicy::for_root(&invocation, &repo, true, false); + let kept = policy.admit_crawled_copies( + vec![copy(&repo.join("services/legacy"), "a")], + &HashSet::new(), + ); + assert_eq!(kept.len(), 1); + assert_eq!(policy.json()["counts"]["filtered"], 0); + } +} diff --git a/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs b/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs index cfe3423d0..ecd012af5 100644 --- a/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs +++ b/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs @@ -378,7 +378,7 @@ async fn vendor_under_lock( detached: true, force: false, prior, - committed: None, + eject: None, }, &mut env, ) diff --git a/crates/socket-patch-cli/src/commands/vendor.rs b/crates/socket-patch-cli/src/commands/vendor.rs index 4c24ee052..6a135851d 100644 --- a/crates/socket-patch-cli/src/commands/vendor.rs +++ b/crates/socket-patch-cli/src/commands/vendor.rs @@ -110,9 +110,80 @@ pub struct VendorArgs { /// request is still fully satisfied when these are the only non-successes. fn refusal_is_benign(code: &str) -> bool { matches!(code, "vendor_unsupported_ecosystem" | "already_vendored") + // An older vendored patch kept in force (see [`keep_older_vendored_patch`]). + || matches!(code, vendor::VENDOR_PREBUILT_PENDING | vendor::VENDOR_PREBUILT_UNAVAILABLE) || socket_patch_core::vendor::jvm::sbt_gate::SKIP_CODES.contains(&code) } +/// #954: the patch service has no artifact for `uuid` yet (still building) +/// or at all (`build_failed`, `not_found`, …) — the backend's failed `Done` +/// carries [`vendor::VENDOR_PREBUILT_PENDING`] / +/// [`vendor::VENDOR_PREBUILT_UNAVAILABLE`] — while the ledger already holds +/// `purl` vendored at another patch. The backend touched nothing, so that +/// older vendoring is still in force: like hosted mode, which keeps its pin +/// and skips the upgrade, the package is a benign skip under the unserved +/// code instead of a failure that would fail every re-run until the server +/// builds the artifact. Only an older vendoring whose wiring is still live +/// (the [`Discovery::vendor_entry_live`] verdict `vendor --check` and `vex` +/// use) is in force: once a relock dropped its `.socket/vendor/` reference +/// the package is patched in neither mode, so the unserved upgrade stays a +/// failure. Any other outcome passes through, a failure minus the unserved +/// marker (its error already says it). +/// +/// [`Discovery::vendor_entry_live`]: socket_patch_core::vex::discover::Discovery::vendor_entry_live +async fn keep_older_vendored_patch( + outcome: Option, + state: &VendorState, + purl: &str, + uuid: &str, + common: &GlobalArgs, +) -> Option { + let Some(VendorOutcome::Done { + result, + entry, + mut warnings, + }) = outcome + else { + return outcome; + }; + if !result.success { + if let Some(i) = warnings.iter().position(|w| { + matches!( + w.code, + vendor::VENDOR_PREBUILT_PENDING | vendor::VENDOR_PREBUILT_UNAVAILABLE + ) + }) { + let unserved = warnings.remove(i); + if let Some(kept) = lookup_entry(&state.entries, purl).filter(|e| e.uuid != uuid) { + let root = common.project_root(); + if !crate::commands::discover_wiring(common, &root) + .await + .vendor_entry_live(&root, kept) + .await + { + return Some(VendorOutcome::Done { + result, + entry, + warnings, + }); + } + return Some(VendorOutcome::Refused { + code: unserved.code, + detail: format!( + "kept the vendored patch {}: {} for patch {uuid}", + kept.uuid, unserved.detail + ), + }); + } + } + } + Some(VendorOutcome::Done { + result, + entry, + warnings, + }) +} + /// The `vendor_dir_symlink_unsupported` detail when `purl`'s vendor dir /// (`.socket/vendor`, its ecosystem dir, or the `uuid` unit) is a link. fn linked_vendor_dir_refusal(project_root: &Path, purl: &str, uuid: &str) -> Option { @@ -1824,8 +1895,11 @@ async fn run_eject(args: &VendorArgs, pins: Vec) -> i32 { .next() .or_else(|| restore.flush_error.clone()); // Every file the vendored apply's group commit wrote, with its bytes - // from before: the rollback's scope beyond the restore's own files. - let mut committed: Vec = Vec::new(); + // from before (the rollback's scope beyond the restore's own files), + // and the flavors it wired for the close printed below. + let mut capture = EjectCapture::default(); + // The vendored apply's events start here: a rollback retracts them. + let events_start = env.events.len(); let mut exit: i32; if let Some(why) = restore_failure { env.mark_error(EnvelopeError::new("redirect_revert_failed", why.clone())); @@ -1854,7 +1928,7 @@ async fn run_eject(args: &VendorArgs, pins: Vec) -> i32 { detached: true, force: false, prior: None, - committed: Some(&mut committed), + eject: Some(&mut capture), }, &mut env, ) @@ -1889,19 +1963,33 @@ async fn run_eject(args: &VendorArgs, pins: Vec) -> i32 { } } } + let committed = &capture.committed; if exit != 0 { - match snapshot.restore(&restore.reverted_files, &committed).await { - Ok(()) => env.warnings.push(RunWarning { - code: "eject_rolled_back".to_string(), - detail: "the eject did not complete, so every file it touched was restored: the \ - project is still hosted, exactly as before" - .to_string(), - }), + match snapshot.restore(&restore.reverted_files, committed).await { + Ok(()) => { + let detail = "the eject did not complete, so every file it touched was \ + restored: the project is still hosted, exactly as before"; + // Nothing the vendored apply did survives the rollback: a + // package it vendored is still hosted, not applied (#1005). + env.retract_applied( + events_start, + PatchAction::Skipped, + "eject_rolled_back", + "vendored, then rolled back with the rest of the eject: still hosted", + ); + if !common.json && !common.silent { + eprintln!("Warning: {detail}"); + } + env.warnings.push(RunWarning { + code: "eject_rolled_back".to_string(), + detail: detail.to_string(), + }); + } Err(e) => { let detail = format!( "the eject did not complete and restoring the pre-eject files failed ({e}); \ restore them from version control (`git checkout -- {}`)", - snapshot.files_hint(&restore.reverted_files, &committed) + snapshot.files_hint(&restore.reverted_files, committed) ); if !common.json { eprintln!("Error: {detail}"); @@ -1912,6 +2000,11 @@ async fn run_eject(args: &VendorArgs, pins: Vec) -> i32 { if env.error.is_none() { env.mark_partial_failure(); } + } else { + // The vendored summary and its "Next steps:", deferred until the + // eject is known to stand: a failed one is rolled back (or needs + // the manual restore its error names), so neither applies. + print_vendor_closing(common, &env, 0, &capture.wired_flavors, true); } note_classic_migration_risk(&mut env, &common.cwd, common); drop(guard); @@ -2054,7 +2147,7 @@ async fn run_vendor( detached: false, force: args.force, prior: None, - committed: None, + eject: None, }, env, ) @@ -2537,9 +2630,13 @@ pub(crate) async fn vendor_records_reusing( service: Option<&VendorServiceConfig>, ledger: std::io::Result, prior: Option<&NpmCrawlSnapshot>, - committed: Option<&mut Vec>, + mut eject: Option<&mut EjectCapture>, ) -> bool { let mut has_errors = false; + // This run's events start here (`scan --mode vendored` hands over an + // envelope that already holds its own): the ones a refused commit + // retracts. + let events_start = env.events.len(); // Lockfile flavors the backends wired THIS run (from the returned ledger // entries, not the whole ledger — an old pnpm entry must not re-flavor // the hints of a run that vendored only cargo). Drives the human @@ -2759,6 +2856,21 @@ pub(crate) async fn vendor_records_reusing( // line prints on a clean line. let mut status = StatusLine::stderr(common.json, common.silent); let total = all_packages.len(); + // The vendored artifact dirs before this run wrote any: a commit the + // symlink check refuses (or that fails with nothing written) removes + // the ones the loop added, so it leaves no orphan artifact behind + // (#898). A dir the pre-run ledger + // already names (an artifact redownloaded in place) is kept: it needs + // no commit to be referenced. + let vendor_dirs_before = (!common.dry_run).then(|| VendorDirsBefore { + dirs: EjectSnapshot::vendor_dir_set(&common.cwd), + referenced: state + .entries + .values() + .map(|e| common.cwd.join(&e.artifact.path)) + .collect(), + }); + // Service downloads, fetched ahead of this serial loop (the wiring and // every write stay here, in order). The plan is EXACT — only the // records the loop will ask the service for (see @@ -3278,6 +3390,8 @@ pub(crate) async fn vendor_records_reusing( } } + let outcome = + keep_older_vendored_patch(outcome, &state, candidate, &record.uuid, common).await; match outcome { None => { env.record( @@ -3469,31 +3583,48 @@ pub(crate) async fn vendor_records_reusing( // others failed — a failed package's backend already put back what it // had touched, in the captured state — so a completed run ends exactly // where committing after every package would have left it. + let mut commit_failed = false; if let Some(group) = group { socket_patch_core::utils::failpoint::hit("vendor_group_commit"); match group.commit_changes().await { Ok(changes) => { - if let Some(committed) = committed { - committed.extend(changes); + if let Some(capture) = eject.as_deref_mut() { + capture.committed.extend(changes); } for stale in stale_artifacts { sweep_stale_artifact(common, env, &state, stale).await; } } Err(e) if socket_patch_core::utils::group_commit::symlinked_target(&e).is_some() => { - // Refused before anything was written: the hosted refusal, - // same code and wording. + // Refused before any lockfile, manifest or ledger was + // written: the hosted refusal, same code and wording. The + // artifacts the loop already downloaded are removed, and + // the packages it vendored are reported as refused, not + // applied — nothing of them was committed (#898). has_errors = true; + commit_failed = true; let linked = socket_patch_core::utils::group_commit::symlinked_target(&e) .unwrap_or_default(); let refusal = socket_patch_core::hosted::engine::symlink_refusal(linked); + let mut message = refusal.message; + let leftovers = remove_new_vendor_dirs(common, vendor_dirs_before.as_ref()).await; + if !leftovers.is_empty() { + message = format!( + "{} (except the downloaded artifacts that could not be removed: {}; \ + `socket-patch vendor --revert` removes them)", + message, + leftovers.join("; ") + ); + } + env.retract_applied(events_start, PatchAction::Failed, &refusal.code, &message); if !common.json { - eprintln!("Error: {}", refusal.message); + eprintln!("Error: {message}"); } - env.mark_error(EnvelopeError::new(refusal.code, refusal.message)); + env.mark_error(EnvelopeError::new(refusal.code, message)); } Err(e) => { has_errors = true; + commit_failed = true; let detail = if socket_patch_core::utils::group_commit::is_pending(&e) { // Some files were replaced and could not be put back: // the journal left behind makes the next locked command @@ -3504,11 +3635,30 @@ pub(crate) async fn vendor_records_reusing( this project finishes it" ) } else { - format!( + // Nothing was committed: like the symlink refusal + // (#898), the artifacts the loop downloaded are removed + // and its packages are reported failed, not applied. + let mut detail = format!( "could not commit the vendored lockfile, manifest and ledger edits: \ {e}; the project's lockfiles and .socket/vendor/state.json are \ unchanged" - ) + ); + let leftovers = + remove_new_vendor_dirs(common, vendor_dirs_before.as_ref()).await; + if !leftovers.is_empty() { + detail = format!( + "{detail} (except the downloaded artifacts that could not be \ + removed: {}; `socket-patch vendor --revert` removes them)", + leftovers.join("; ") + ); + } + env.retract_applied( + events_start, + PatchAction::Failed, + "vendor_commit_failed", + &detail, + ); + detail }; if !common.json { eprintln!("Error: {detail}"); @@ -3617,68 +3767,128 @@ pub(crate) async fn vendor_records_reusing( } } - if !common.json && !common.silent { - let tally = VendorTally::from_envelope(env, common.dry_run, dry_in_sync); - println!("{}", format_vendor_summary(common.dry_run, &tally)); - if env.summary.applied > 0 && !common.dry_run { - // pnpm >=11 reads `overrides` ONLY from pnpm-workspace.yaml (the - // package.json `pnpm.overrides` mirror is ignored), so pnpm-wired - // runs must name that file among the committables: a checkout - // that loses it silently unvendors on the next install. - let commit = commit_hint(&wired_flavors); - let mut installs: Vec<&str> = wired_flavors + // A rolled-back eject's summary would describe a vendoring that was + // undone: the eject prints it itself once it knows the outcome (#1005). + match eject { + Some(capture) => capture.wired_flavors = wired_flavors, + None => print_vendor_closing(common, env, dry_in_sync, &wired_flavors, !commit_failed), + } + + has_errors +} + +/// The human close of a vendor run: the summary line, then — when +/// something was vendored and `next_steps` (the run's commit landed) — the +/// commit and reinstall "Next steps:" for the lockfile flavors the run +/// wired. +fn print_vendor_closing( + common: &GlobalArgs, + env: &Envelope, + dry_in_sync: u32, + wired_flavors: &HashSet, + next_steps: bool, +) { + if common.json || common.silent { + return; + } + let tally = VendorTally::from_envelope(env, common.dry_run, dry_in_sync); + println!("{}", format_vendor_summary(common.dry_run, &tally)); + if env.summary.applied > 0 && !common.dry_run && next_steps { + // pnpm >=11 reads `overrides` ONLY from pnpm-workspace.yaml (the + // package.json `pnpm.overrides` mirror is ignored), so pnpm-wired + // runs must name that file among the committables: a checkout + // that loses it silently unvendors on the next install. + let commit = commit_hint(wired_flavors); + let mut installs: Vec<&str> = wired_flavors + .iter() + .filter_map(|f| flavor_install_command(f)) + .collect(); + installs.sort_unstable(); + installs.dedup(); + let jvm_only = !installs.is_empty() + && wired_flavors .iter() - .filter_map(|f| flavor_install_command(f)) - .collect(); - installs.sort_unstable(); - installs.dedup(); - let jvm_only = !installs.is_empty() - && wired_flavors - .iter() - .filter(|f| flavor_install_command(f).is_some()) - .all(|f| JVM_TOOLS.contains(&f.as_str())); - let reinstall = if jvm_only { - let cmds: Vec = installs.iter().map(|c| format!("`{c}`")).collect(); - format!( - "Run {} so the build resolves the vendored artifacts (the generated root \ - file points it at .socket/vendor/)", - cmds.join(" and ") - ) - } else if installs.is_empty() { - "Reinstall from the updated lockfile so the installed packages pick up the \ - vendored artifacts" - .to_string() - } else { - let cmds: Vec = installs.iter().map(|c| format!("`{c}`")).collect(); - format!( - "Run {} to update the installed tree (vendoring rewires the lockfile \ - only; the current install keeps the unpatched bytes until reinstalled)", - cmds.join(" and ") - ) - }; - let mut extra = Vec::new(); - if wired_flavors.contains("bun") && common.cwd.join("bun.lockb").exists() { - extra.push( - "For binary Bun workspaces, also commit the workspace members' \ - .socket/vendor/ tarballs recorded in the vendor ledger." - .to_string(), - ); - } - if wired_flavors.contains("composer") { - if let Ok(lock) = socket_patch_core::utils::fs::read_regular_to_string_sync( - &common.cwd.join("composer.lock"), - ) { - let packages = super::composer_hints::vendored_composer_packages(&lock); - extra.extend(super::composer_hints::vendored_reinstall_hints(&packages)); - } - } - for line in crate::ui::next_steps(&commit, &reinstall, &extra) { - println!("{line}"); + .filter(|f| flavor_install_command(f).is_some()) + .all(|f| JVM_TOOLS.contains(&f.as_str())); + let reinstall = if jvm_only { + let cmds: Vec = installs.iter().map(|c| format!("`{c}`")).collect(); + format!( + "Run {} so the build resolves the vendored artifacts (the generated root \ + file points it at .socket/vendor/)", + cmds.join(" and ") + ) + } else if installs.is_empty() { + "Reinstall from the updated lockfile so the installed packages pick up the \ + vendored artifacts" + .to_string() + } else { + let cmds: Vec = installs.iter().map(|c| format!("`{c}`")).collect(); + format!( + "Run {} to update the installed tree (vendoring rewires the lockfile \ + only; the current install keeps the unpatched bytes until reinstalled)", + cmds.join(" and ") + ) + }; + let mut extra = Vec::new(); + if wired_flavors.contains("bun") && common.cwd.join("bun.lockb").exists() { + extra.push( + "For binary Bun workspaces, also commit the workspace members' \ + .socket/vendor/ tarballs recorded in the vendor ledger." + .to_string(), + ); + } + if wired_flavors.contains("composer") { + if let Ok(lock) = socket_patch_core::utils::fs::read_regular_to_string_sync( + &common.cwd.join("composer.lock"), + ) { + let packages = super::composer_hints::vendored_composer_packages(&lock); + extra.extend(super::composer_hints::vendored_reinstall_hints(&packages)); } } + for line in crate::ui::next_steps(&commit, &reinstall, &extra) { + println!("{line}"); + } } +} - has_errors +/// The vendored artifact dirs a run started with (see +/// [`EjectSnapshot::vendor_dir_set`]), and the artifact paths its ledger +/// named then. +struct VendorDirsBefore { + dirs: std::collections::BTreeSet, + referenced: Vec, +} + +/// Remove the vendored artifact dirs that appeared since `before`, except +/// one holding an artifact the pre-run ledger names (redownloaded in place, +/// referenced without any commit), returning what could not be removed. +async fn remove_new_vendor_dirs( + common: &GlobalArgs, + before: Option<&VendorDirsBefore>, +) -> Vec { + let Some(before) = before else { + return Vec::new(); + }; + let mut leftovers = Vec::new(); + for dir in EjectSnapshot::vendor_dir_set(&common.cwd).difference(&before.dirs) { + if before.referenced.iter().any(|p| p.starts_with(dir)) { + continue; + } + if let Err(e) = remove_tree_and_prune(dir, &common.cwd.join(SOCKET_DIR)).await { + leftovers.push(format!("{}: {e}", dir.display())); + } + } + leftovers +} + +/// What a hosted→vendored eject's vendored apply hands back instead of +/// printing its own close: every project file its group commit wrote (with +/// the bytes from before, for the rollback) and the lockfile flavors it +/// wired (for the "Next steps:" the eject prints only when it completes). +#[derive(Default)] +pub(crate) struct EjectCapture { + pub(crate) committed: Vec, + pub(crate) wired_flavors: HashSet, } /// The pseudo-flavors of vendored JVM builds whose wiring is a generated diff --git a/crates/socket-patch-cli/src/commands/vendored_backend/mod.rs b/crates/socket-patch-cli/src/commands/vendored_backend/mod.rs index 816b89026..2b0461f30 100644 --- a/crates/socket-patch-cli/src/commands/vendored_backend/mod.rs +++ b/crates/socket-patch-cli/src/commands/vendored_backend/mod.rs @@ -16,13 +16,12 @@ use std::collections::HashMap; use std::path::Path; use socket_patch_core::manifest::schema::{PatchManifest, PatchRecord}; -use socket_patch_core::utils::group_commit::CommittedFile; use socket_patch_core::vendor::{ save_state, RevertOpts, VendorServiceConfig, VendorState, VendorWarning, }; use crate::args::GlobalArgs; -use crate::commands::vendor::{dispatch_revert_one_opts, vendor_records_reusing}; +use crate::commands::vendor::{dispatch_revert_one_opts, vendor_records_reusing, EjectCapture}; use crate::ecosystem_dispatch::NpmCrawlSnapshot; use crate::json_envelope::Envelope; @@ -48,9 +47,12 @@ pub(crate) struct ApplyRequest<'a> { /// The npm half of a crawl this process already made over an untouched /// tree (see [`vendor_records_reusing`]). pub(crate) prior: Option<&'a NpmCrawlSnapshot>, - /// Receives every project file the run's group commit wrote, with the - /// bytes it held before (the eject's rollback undoes exactly these). - pub(crate) committed: Option<&'a mut Vec>, + /// The hosted→vendored eject's hand-back: every project file the run's + /// group commit wrote, with the bytes it held before (the eject's + /// rollback undoes exactly these), and the wired flavors. With it, the + /// run's human summary and "Next steps:" are left to the eject, which + /// prints them only once it knows the eject stands. + pub(crate) eject: Option<&'a mut EjectCapture>, } impl<'a> VendoredBackend<'a> { @@ -85,7 +87,7 @@ impl<'a> VendoredBackend<'a> { self.service, req.ledger, req.prior, - req.committed, + req.eject, )) .await } diff --git a/crates/socket-patch-cli/src/commands/vex_consumed.rs b/crates/socket-patch-cli/src/commands/vex_consumed.rs index b55788ef4..2c0ac9605 100644 --- a/crates/socket-patch-cli/src/commands/vex_consumed.rs +++ b/crates/socket-patch-cli/src/commands/vex_consumed.rs @@ -23,7 +23,7 @@ //! | golang | the REPLACEMENT module `$GOMODCACHE/patch.socket.dev/gopatch/@` (the ref's `url`, else go.mod's hosted `replace`) | the original `M@v` | //! | cargo | `registry/src/-/-` for the lock source's host; several such registries (one per patch uuid) are narrowed to the one whose cached `.crate` has the lock's pinned checksum. A `vendor/` source tree or `--global-prefix` is taken as given | crates.io's / any other registry's extraction | //! | maven | `///-socket./` (the version the pom or the hosted Gradle wiring pins) in `~/.m2` and every Gradle `files-2.1` holding it (hash dirs expanded), its artifact files matched under the suffixed name | the `` version dir | -//! | npm | every `node_modules` copy the crawler finds (pnpm and vlt store copies included), every peer / modifier / registry variant of those in the same `.pnpm` / `.vlt` store, plus alias installs (`node_modules/` holding the package) in the root's and every workspace member's tree | — each serves some dependent: ALL must verify | +//! | npm | every `node_modules` copy the crawler finds (pnpm and vlt store copies included), every peer / modifier / registry variant of those in the same `.pnpm` / `.vlt` store, alias installs (`node_modules/` holding the package) in the root's and every workspace member's tree included | — each serves some dependent: ALL must verify | //! | pypi | every copy in the crawler's environment set (the project's venvs when it has any, else the interpreters) | — any may be the one that runs the project: ALL must verify | //! | gem | every copy in bundler's gem path | — bundler loads whichever `Gem.path` home it hits first: ALL must verify | //! @@ -43,7 +43,7 @@ use std::path::{Path, PathBuf}; #[cfg(not(test))] use socket_patch_core::crawlers::npm_crawler::with_store_peer_variant_copies; use socket_patch_core::crawlers::{ - CargoCrawler, CrawlerOptions, Ecosystem, GoCrawler, MavenCrawler, NpmCrawler, + CargoCrawler, CrawlerOptions, Ecosystem, GoCrawler, MavenCrawler, }; use socket_patch_core::utils::purl::purl_parts; use socket_patch_core::vendor::go_mod_edit::{ @@ -68,8 +68,8 @@ use crate::ecosystem_dispatch::{ /// variants. The shared-location ecosystems read it instead of crawling /// the tree a second time. `prior` /// (embedded hosted `scan --vex` only) is scan's npm crawl of the same -/// tree: the alias walk takes its `node_modules` roots and the identity -/// fallback its packages instead of walking the tree again. +/// tree: the identity fallback takes its packages instead of crawling the +/// tree again. pub(crate) async fn hosted_consumed_copies( common: &GlobalArgs, hosted: &BTreeMap, @@ -96,55 +96,16 @@ pub(crate) async fn hosted_consumed_copies( .flatten() .filter_map(|purl| Some((purl.clone(), installed.get(purl)?.clone()))) .collect(); - let mut aliases = match shared.get(&Ecosystem::Npm) { - Some(npm) => npm_alias_copies_reusing(&options, npm, prior).await, - None => HashMap::new(), - }; - npm_identity_fallback_reusing( - shared.get(&Ecosystem::Npm), - &options, - &mut all, - &aliases, - prior, - ) - .await; + npm_identity_fallback_reusing(shared.get(&Ecosystem::Npm), &options, &mut all, prior).await; let npm: Vec<&String> = shared.get(&Ecosystem::Npm).into_iter().flatten().collect(); for purl in shared.values().flatten() { let mut paths = all.remove(purl).unwrap_or_default(); - // The installed-tree lookup already resolves importer-tree - // aliases, so most of the walk's finds are in `paths` already. - let extra: Vec = aliases - .remove(purl) - .unwrap_or_default() - .into_iter() - .filter(|alias| !paths.contains(alias)) - .collect(); - if npm.contains(&purl) && installed.get(purl).is_some_and(|p| !p.is_empty()) { - // The installed lookup already expanded these copies. - // Expanding its N variants again scans the store N times. - // Only aliases are new; expand them before merging so a - // different alias/store can still contribute more copies. - if !extra.is_empty() { - let added = with_store_peer_variant_copies(extra).await; - let mut seen = std::collections::HashSet::new(); - for path in &paths { - seen.insert(tokio::fs::canonicalize(path).await.unwrap_or(path.clone())); - } - for path in added { - let canonical = - tokio::fs::canonicalize(&path).await.unwrap_or(path.clone()); - if seen.insert(canonical) { - paths.push(path); - } - } - } - } else if npm.contains(&purl) { - // No installed copies: the identity fallback and aliases - // have not had their store variants enumerated yet. - paths.extend(extra); + // The installed lookup already resolved every copy, importer + // tree aliases included, and expanded their store variants. + // Without installed copies the identity fallback's have not + // had their store variants enumerated yet. + if npm.contains(&purl) && installed.get(purl).is_none_or(Vec::is_empty) { paths = with_store_peer_variant_copies(paths).await; - } else { - paths.extend(extra); } out.insert( purl.clone(), @@ -176,142 +137,12 @@ fn not_installed() -> HostedCopies { HostedCopies::default() } -// ── npm aliases ────────────────────────────────────────────────────────── - -/// Upper bound on the package dirs [`npm_alias_copies`] inspects: a -/// pathological (or hostile) tree cannot turn one `vex` run into an -/// unbounded walk. -const ALIAS_WALK_MAX_DIRS: usize = 200_000; - -/// ALIAS installs of the hosted npm `purls`, keyed by purl: a dependency -/// declared `"mm": "npm:minimist@1.2.2"` (npm, yarn and bun alike) lands in -/// `node_modules/mm`, a dir the crawler's name-keyed lookup never probes -/// (it matches `node_modules/` by design, so apply cannot patch the -/// wrong package). For a hosted ref that copy is what the aliased import -/// loads, so it is consumed evidence like any other (otherwise an alias -/// that is the ONLY copy reads as "not installed" and attests from the lock -/// pin alone). -/// -/// Walks EVERY importer `node_modules` tree the crawler resolves the -/// installed copies from ([`NpmCrawler::get_node_modules_paths`]: the -/// root's AND each workspace member's in a project, the prefix under -/// `--global-prefix`) once, matching each package dir's `package.json` -/// `(name, version)`; only dirs whose on-disk key DIFFERS from the -/// package's name are aliases (the crawler already returns the rest). -/// Member trees matter because the identity fallback only fills purls with -/// NO copy, so a member alias beside a root copy is found only here. -/// Hidden entries (`.bin`, pnpm's `.pnpm` and vlt's `.vlt` stores — the -/// crawler probes them) and symlinks (pnpm's and vlt's importer links, -/// `npm link` targets) are not traversed. Under vlt EVERY importer entry, -/// an alias included, is a link into `.vlt//node_modules/`, -/// so this walk finds no vlt alias at all: the store copy is named after -/// the real package, and the crawler's store pass resolves it (the -/// identity fallback covers the rest). A plain `--global` run is not -/// walked: its roots come from spawning every package manager again, and -/// the identity fallback covers an alias that is the only global copy. -#[cfg(test)] -async fn npm_alias_copies( - options: &CrawlerOptions, - purls: &[String], -) -> HashMap> { - npm_alias_copies_reusing(options, purls, None).await -} - -/// [`npm_alias_copies`], taking the importer `node_modules` roots from -/// `prior` when it was crawled with `options` (the same roots -/// `NpmCrawler::get_node_modules_paths` returns) instead of walking the tree -/// for them; the per-root BFS below is unchanged. -async fn npm_alias_copies_reusing( - options: &CrawlerOptions, - purls: &[String], - prior: Option<&NpmCrawlSnapshot>, -) -> HashMap> { - let wanted: HashMap<(String, String), &String> = purls - .iter() - .filter_map(|purl| { - let (ty, name, version) = purl_parts(purl)?; - (ty == "npm").then_some(((name, version), purl)) - }) - .collect(); - let mut out: HashMap> = HashMap::new(); - if wanted.is_empty() { - return out; - } - if options.global && options.global_prefix.is_none() { - return out; - } - let roots = match prior.and_then(|p| p.roots_for(options)) { - Some(roots) => roots.to_vec(), - None => NpmCrawler::new() - .get_node_modules_paths(options) - .await - .unwrap_or_default(), - }; - let mut queue = std::collections::VecDeque::from(roots); - let mut visited = 0usize; - while let Some(nm) = queue.pop_front() { - // (package dir, the key it is installed under) - let mut packages: Vec<(PathBuf, String)> = Vec::new(); - for (path, name) in real_subdirs(&nm).await { - if name.starts_with('@') { - for (pkg, bare) in real_subdirs(&path).await { - packages.push((pkg, format!("{name}/{bare}"))); - } - } else { - packages.push((path, name)); - } - } - for (pkg, key) in packages { - visited += 1; - if visited > ALIAS_WALK_MAX_DIRS { - return out; - } - let found = socket_patch_core::crawlers::npm_crawler::read_package_json( - &pkg.join("package.json"), - ) - .await; - if let Some((name, version)) = found { - if name != key { - if let Some(purl) = wanted.get(&(name, version)) { - let copies = out.entry((*purl).clone()).or_default(); - if !copies.contains(&pkg) { - copies.push(pkg.clone()); - } - } - } - } - queue.push_back(pkg.join("node_modules")); - } - } - out -} - -/// `(path, name)` of the non-hidden, non-symlink directories directly in -/// `dir` (empty when `dir` is missing or unreadable). -async fn real_subdirs(dir: &Path) -> Vec<(PathBuf, String)> { - let mut out = Vec::new(); - let Ok(mut entries) = tokio::fs::read_dir(dir).await else { - return out; - }; - while let Ok(Some(entry)) = entries.next_entry().await { - let name = entry.file_name().to_string_lossy().into_owned(); - if name.starts_with('.') { - continue; - } - // `DirEntry::file_type` does not follow symlinks: a link is skipped. - if entry.file_type().await.is_ok_and(|t| t.is_dir()) { - out.push((entry.path(), name)); - } - } - out -} +// ── npm identity fallback ──────────────────────────────────────────────── -/// Last-resort identity lookup for an npm purl that neither the targeted -/// resolver nor the [`npm_alias_copies`] walk found. An npm ALIAS dependency -/// (`"lp": "npm:left-pad@1.3.0"`) is installed under its dependency key -/// (`node_modules/lp`), not its package name, so the targeted resolver — -/// which probes `node_modules/` — reports it not installed, and the -/// walk skips symlinked importer entries (yarn's pnpm linker, a global +/// Last-resort identity lookup for an npm purl the targeted resolver did +/// not find. The resolver takes real alias dirs (`node_modules/lp` holding +/// `left-pad@1.3.0`) as copies, but not an alias reached through a symlinked +/// importer entry (yarn's pnpm linker, `npm link`, a global /// `--global-prefix` tree). For a hosted purl "not installed" is exactly what /// the lockfile basis excuses, so such a copy must still be hash-verified /// ("installed evidence wins"). Resolve every npm purl the targeted lookup @@ -322,9 +153,8 @@ async fn npm_identity_fallback( npm: Option<&Vec>, options: &CrawlerOptions, all: &mut HashMap>, - aliases: &HashMap>, ) { - npm_identity_fallback_reusing(npm, options, all, aliases, None).await + npm_identity_fallback_reusing(npm, options, all, None).await } /// [`npm_identity_fallback`], answering from `prior`'s crawled packages @@ -334,15 +164,12 @@ async fn npm_identity_fallback_reusing( npm: Option<&Vec>, options: &CrawlerOptions, all: &mut HashMap>, - aliases: &HashMap>, prior: Option<&NpmCrawlSnapshot>, ) { let missing: Vec<&String> = npm .into_iter() .flatten() - .filter(|purl| { - all.get(*purl).is_none_or(Vec::is_empty) && aliases.get(*purl).is_none_or(Vec::is_empty) - }) + .filter(|purl| all.get(*purl).is_none_or(Vec::is_empty)) .collect(); match prior.and_then(|p| p.packages_for(options)) { Some(installed) => all.extend(npm_paths_by_identity_in(installed, &missing)), @@ -624,6 +451,7 @@ async fn maven_copies(options: &CrawlerOptions, purl: &str, wiring: &HostedWirin #[cfg(test)] mod tests { use super::*; + use socket_patch_core::crawlers::NpmCrawler; tokio::task_local! { // Observe real expansion work only in the regression's own task; @@ -659,6 +487,17 @@ mod tests { .await } + /// The installed-tree lookup `vex` hands [`hosted_consumed_copies`]. + async fn installed_copies(common: &GlobalArgs, purl: &str) -> HashMap> { + crate::ecosystem_dispatch::find_manifest_package_copies_reusing( + &[purl.to_string()], + common, + true, + None, + ) + .await + } + #[cfg(unix)] fn peer_copies(store: &Path, count: usize) -> Vec { (0..count) @@ -685,13 +524,7 @@ mod tests { ..GlobalArgs::default() }; let purl = "pkg:npm/left-pad@1.3.0".to_string(); - let installed = crate::ecosystem_dispatch::find_manifest_package_copies_reusing( - std::slice::from_ref(&purl), - &common, - true, - None, - ) - .await; + let installed = installed_copies(&common, &purl).await; assert_eq!(installed[&purl].len(), peers.len()); let (paths, calls) = tracked_npm_hosted(&common, &installed).await; assert_eq!(paths, installed[&purl]); @@ -700,53 +533,42 @@ mod tests { "already-expanded copies were rescanned: {calls:?}" ); - // A real alias is absent from the name-keyed installed set. Its - // store variants overlap that set canonically, including the - // importer link's physical copy; keep the alias once and preserve - // the original importer-first path choices. + // The resolver takes a real alias as a copy and expands its store + // variants, which overlap the set canonically (the importer link's + // physical copy included): the alias is listed once, the importer + // link stays first, and nothing is expanded again. let alias = nm.join("lp"); pkg(&alias, "left-pad", "1.3.0"); - let (paths, calls) = tracked_npm_hosted(&common, &installed).await; - assert_eq!(calls, vec![vec![alias.clone()]]); + let with_alias = installed_copies(&common, &purl).await; + let (paths, calls) = tracked_npm_hosted(&common, &with_alias).await; + assert!(calls.is_empty(), "{calls:?}"); + assert_eq!(paths, with_alias[&purl]); + assert_eq!(paths[0], installed[&purl][0]); let mut expected = installed[&purl].clone(); expected.push(alias); - assert_eq!(paths, expected); + expected.sort(); + assert_eq!(sorted(paths), expected); - // An alias beneath a real nested host can reach another store. - // The installed root copy makes the name-keyed resolver skip - // those peers, so alias expansion must still add them even when - // installed copies are already present. + // An alias beneath a real nested host can reach another store. The + // installed root copy does not keep the resolver from that store's + // peers. let host = nm.join("host"); pkg(&host, "host", "1.0.0"); let host_nm = host.join("node_modules"); let nested_peers = peer_copies(&host_nm.join(".pnpm"), 2); let nested_alias = host_nm.join("lp"); pkg(&nested_alias, "left-pad", "1.3.0"); - let installed_again = crate::ecosystem_dispatch::find_manifest_package_copies_reusing( - std::slice::from_ref(&purl), - &common, - true, - None, - ) - .await; - // Since #605 the name-keyed resolver probes bundled trees itself, so - // it already returns the aliases and the nested store's peers. Feed - // the earlier, alias-free set to keep exercising alias expansion; - // the resolver's own set is checked against the same result below. - let (paths, calls) = tracked_npm_hosted(&common, &installed).await; - assert_eq!(calls.len(), 1); - let mut inputs = calls[0].clone(); - inputs.sort(); - let mut aliases = vec![nm.join("lp"), nested_alias.clone()]; - aliases.sort(); - assert_eq!(inputs, aliases); - assert_eq!(&paths[..installed[&purl].len()], installed[&purl]); + let installed_again = installed_copies(&common, &purl).await; + let (paths, calls) = tracked_npm_hosted(&common, &installed_again).await; + assert!(calls.is_empty(), "{calls:?}"); + assert_eq!(paths, installed_again[&purl]); + assert_eq!(paths[0], installed[&purl][0]); expected.push(nested_alias); expected.extend(nested_peers); let mut actual = paths.clone(); actual.sort(); expected.sort(); - assert_eq!(actual, expected); + assert_eq!(actual, expected, "the resolver's own copy set"); assert_eq!( paths .iter() @@ -755,9 +577,6 @@ mod tests { .len(), paths.len() ); - let (mut resolved, _) = tracked_npm_hosted(&common, &installed_again).await; - resolved.sort(); - assert_eq!(resolved, expected, "the resolver's own copy set"); } #[cfg(unix)] @@ -781,16 +600,10 @@ mod tests { ..GlobalArgs::default() }; let purl = "pkg:npm/left-pad@1.3.0".to_string(); - let installed = crate::ecosystem_dispatch::find_manifest_package_copies_reusing( - std::slice::from_ref(&purl), - &common, - true, - None, - ) - .await; - // Since #605 the name-keyed resolver reaches the alias and its - // sibling peers on its own. An alias-only set (what an alias-blind - // resolver returns) must still expand to the same copies. + let installed = installed_copies(&common, &purl).await; + // The resolver reaches the alias and its sibling peers on its own. + // With no installed set, the identity fallback's alias copy must + // still expand to the same copies. let (mut paths, calls) = tracked_npm_hosted(&common, &HashMap::new()).await; assert_eq!(calls, vec![vec![alias.clone()]]); let mut expected = peers; @@ -824,11 +637,11 @@ mod tests { ..GlobalArgs::default() }; let purl = "pkg:npm/left-pad@1.3.0".to_string(); - assert!( - npm_alias_copies(&common.crawler_options(), std::slice::from_ref(&purl)) - .await - .is_empty() - ); + // The resolver never takes a link as a copy. + assert!(installed_copies(&common, &purl) + .await + .get(&purl) + .is_none_or(Vec::is_empty)); let installed = HashMap::from([(purl, Vec::new())]); let (mut paths, calls) = tracked_npm_hosted(&common, &installed).await; assert_eq!(calls, vec![vec![alias.clone()]]); @@ -877,7 +690,7 @@ mod tests { "pkg:npm/absent@2.0.0".to_string(), ]; let mut all: HashMap> = HashMap::new(); - npm_identity_fallback(Some(&purls), &options, &mut all, &HashMap::new()).await; + npm_identity_fallback(Some(&purls), &options, &mut all).await; assert_eq!( all.get("pkg:npm/left-pad@1.3.0"), Some(&vec![tmp.path().join("node_modules/lp")]) @@ -888,20 +701,14 @@ mod tests { // fallback only fills misses). let found = tmp.path().join("node_modules/left-pad"); let mut all = HashMap::from([(purls[0].clone(), vec![found.clone()])]); - npm_identity_fallback(Some(&purls), &options, &mut all, &HashMap::new()).await; + npm_identity_fallback(Some(&purls), &options, &mut all).await; assert_eq!(all[&purls[0]], vec![found.clone()]); - - // A purl the alias walk already found is not re-resolved. - let mut all: HashMap> = HashMap::new(); - let walked = HashMap::from([(purls[0].clone(), vec![found.clone()])]); - npm_identity_fallback(Some(&purls), &options, &mut all, &walked).await; - assert!(!all.contains_key(&purls[0]), "{all:?}"); } /// The identity fallback answered from the crawl snapshot finds the /// same copies as crawling again — here an alias installed through a - /// symlink (yarn's pnpm linker, `npm link`), which neither the targeted - /// lookup nor the alias walk (it skips symlinks) finds, among other + /// symlink (yarn's pnpm linker, `npm link`), which the targeted lookup + /// does not take as a copy (it skips symlinks), among other /// crawled packages so it is not the snapshot's first entry. #[cfg(unix)] #[tokio::test] @@ -926,10 +733,18 @@ mod tests { "pkg:npm/is-odd@1.3.0".to_string(), "pkg:npm/absent@2.0.0".to_string(), ]; - let aliases = npm_alias_copies(&options, &purls).await; + let resolved = installed_copies( + &GlobalArgs { + cwd: root.to_path_buf(), + ecosystems: Some(vec!["npm".to_string()]), + ..GlobalArgs::default() + }, + &purls[0], + ) + .await; assert!( - aliases.is_empty(), - "the alias walk skips symlinks: {aliases:?}" + resolved.get(&purls[0]).is_none_or(Vec::is_empty), + "the resolver skips symlinks: {resolved:?}" ); let (_, _, _, snapshot) = @@ -941,16 +756,9 @@ mod tests { ); let mut walked: HashMap> = HashMap::new(); - npm_identity_fallback(Some(&purls), &options, &mut walked, &aliases).await; + npm_identity_fallback(Some(&purls), &options, &mut walked).await; let mut reused: HashMap> = HashMap::new(); - npm_identity_fallback_reusing( - Some(&purls), - &options, - &mut reused, - &aliases, - Some(&snapshot), - ) - .await; + npm_identity_fallback_reusing(Some(&purls), &options, &mut reused, Some(&snapshot)).await; assert_eq!(reused, walked); for purl in &purls[..2] { assert_eq!( @@ -962,9 +770,23 @@ mod tests { assert!(!reused.contains_key("pkg:npm/absent@2.0.0"), "{reused:?}"); } - /// Only dirs whose install key differs from the package name are - /// aliases; scoped keys, nested trees and scoped targets are walked; - /// hidden dirs, other versions and symlinks are not copies. + fn npm_scope(root: &Path) -> GlobalArgs { + GlobalArgs { + cwd: root.to_path_buf(), + ecosystems: Some(vec!["npm".to_string()]), + ..GlobalArgs::default() + } + } + + fn sorted(mut paths: Vec) -> Vec { + paths.sort(); + paths + } + + /// The installed lookup `vex` judges hosted npm purls by takes alias + /// installs as copies beside the own-name one: scoped keys, nested trees + /// and scoped targets count; hidden dirs, other versions and symlinks do + /// not (#856: no second alias walk backs it up). #[tokio::test] async fn npm_alias_copies_finds_only_alias_installs() { let tmp = tempfile::tempdir().unwrap(); @@ -988,23 +810,29 @@ mod tests { "pkg:npm/minimist@1.2.2".to_string(), "pkg:npm/%40scope/pkg@1.0.0".to_string(), ]; - let found = npm_alias_copies(&local(root), &purls).await; - let mut got: Vec = found["pkg:npm/minimist@1.2.2"].clone(); - got.sort(); - let mut want = vec![ - nm.join("@me/mm"), - nm.join("dep/node_modules/deep"), - nm.join("mm"), - ]; - want.sort(); - assert_eq!(got, want); + let found = crate::ecosystem_dispatch::find_manifest_package_copies_reusing( + &purls, + &npm_scope(root), + true, + None, + ) + .await; + assert_eq!( + sorted(found["pkg:npm/minimist@1.2.2"].clone()), + sorted(vec![ + nm.join("@me/mm"), + nm.join("dep/node_modules/deep"), + nm.join("minimist"), + nm.join("mm"), + ]) + ); assert_eq!(found["pkg:npm/%40scope/pkg@1.0.0"], vec![nm.join("sc")]); } /// A workspace member's alias install is a consumed copy even when the - /// root holds the package under its own name. Every importer tree the - /// crawler enumerates is walked; `--global-prefix` walks the prefix; a - /// plain `--global` run walks nothing (the fallback covers it). + /// root holds the package under its own name: every importer tree the + /// crawler enumerates is searched, and `--global-prefix` searches the + /// prefix. #[tokio::test] async fn npm_alias_copies_walks_every_workspace_members_tree() { let tmp = tempfile::tempdir().unwrap(); @@ -1025,43 +853,35 @@ mod tests { "left-pad", "1.3.0", ); - // Not an alias: the member's own-name copy is the crawler's. pkg( &root.join("apps/web/node_modules/left-pad"), "left-pad", "1.3.0", ); - let purls = vec!["pkg:npm/left-pad@1.3.0".to_string()]; - let mut got = - npm_alias_copies(&local(root), &purls).await["pkg:npm/left-pad@1.3.0"].clone(); - got.sort(); - let mut want = vec![ - root.join("apps/web/node_modules/@me/lp"), - root.join("packages/a/node_modules/dep/node_modules/deep"), - root.join("packages/a/node_modules/lp"), - ]; - want.sort(); - assert_eq!(got, want); + let purl = "pkg:npm/left-pad@1.3.0"; + let got = installed_copies(&npm_scope(root), purl).await; + assert_eq!( + sorted(got[purl].clone()), + sorted(vec![ + root.join("apps/web/node_modules/@me/lp"), + root.join("apps/web/node_modules/left-pad"), + root.join("node_modules/left-pad"), + root.join("packages/a/node_modules/dep/node_modules/deep"), + root.join("packages/a/node_modules/lp"), + ]) + ); let prefix = root.join("packages/a/node_modules"); - let under_prefix = CrawlerOptions { + let under_prefix = GlobalArgs { global_prefix: Some(prefix.clone()), - ..local(root) + ..npm_scope(root) }; - let mut got = - npm_alias_copies(&under_prefix, &purls).await["pkg:npm/left-pad@1.3.0"].clone(); - got.sort(); + let got = installed_copies(&under_prefix, purl).await; assert_eq!( - got, + sorted(got[purl].clone()), vec![prefix.join("dep/node_modules/deep"), prefix.join("lp")] ); - - let global = CrawlerOptions { - global: true, - ..local(root) - }; - assert!(npm_alias_copies(&global, &purls).await.is_empty()); } #[cfg(unix)] @@ -1104,7 +924,7 @@ mod tests { } /// vlt twin of the `.pnpm` case: every importer entry is a link into - /// the `.vlt` store (the alias `lp` too), so the alias walk yields + /// the `.vlt` store (the alias `lp` too), so no importer dir is an alias /// nothing, while the crawler resolves the alias's package from its /// store copy, which is named after the real package. That store copy /// is the consumed evidence. @@ -1133,7 +953,6 @@ mod tests { .unwrap(); let purls = vec!["pkg:npm/left-pad@1.1.3".to_string()]; - assert!(npm_alias_copies(&local(root), &purls).await.is_empty()); let found = NpmCrawler::new().find_by_purls(&nm, &purls).await.unwrap(); assert_eq!( @@ -1145,7 +964,7 @@ mod tests { ); let mut all: HashMap> = HashMap::new(); - npm_identity_fallback(Some(&purls), &local(root), &mut all, &HashMap::new()).await; + npm_identity_fallback(Some(&purls), &local(root), &mut all).await; assert_eq!(all.get(&purls[0]), Some(&vec![nm.join("lp")])); } diff --git a/crates/socket-patch-cli/src/commands/vex_sources.rs b/crates/socket-patch-cli/src/commands/vex_sources.rs index 307f4d0cf..994695efc 100644 --- a/crates/socket-patch-cli/src/commands/vex_sources.rs +++ b/crates/socket-patch-cli/src/commands/vex_sources.rs @@ -225,9 +225,18 @@ fn unattested_note(kind: UnattestedKind) -> (&'static str, &'static str) { intact, so re-running `scan` / `vendor` does not change this (drop the entry from \ deno.lock if Deno does not install this project's npm dependencies)", ), + UnattestedKind::NpmShrinkwrapOnly => ( + NOTE_NPM_SHRINKWRAP_ONLY, + "not attested until a package-lock.json wires it", + ), } } +/// Omission tag and note: the patch is wired only in a root +/// `npm-shrinkwrap.json` with no `package-lock.json` twin, which npm >= 12 +/// never reads (`vex::Unattested`, #899). +pub(crate) const NOTE_NPM_SHRINKWRAP_ONLY: &str = "vex_npm_shrinkwrap_only"; + fn note(code: &'static str, detail: String) -> PlanNote { PlanNote { code, detail } } @@ -363,8 +372,10 @@ pub(crate) async fn plan(common: &GlobalArgs, sources: Sources, assume_live: &[S let superseded = attach_discovered(&mut cands, &discovery, &vendor, &conflicts); // Wired, but a build bypasses the pin (`Unattested`: a Gradle lock // above the hosted base resolves the newer upstream release, a pnpm - // bundled copy, a deno.lock copy): the ref keeps rollback, remove, - // list and the ledgers' liveness working, and the patch is omitted. + // bundled copy, a deno.lock copy, an npm shrinkwrap with no + // package-lock.json twin, which npm >= 12 ignores): the ref keeps + // rollback, remove, list and the ledgers' liveness working, and the + // patch is omitted. cands.retain(|c| { let pkg = canonical_base_purl(&c.key); let Some(u) = discovery diff --git a/crates/socket-patch-cli/src/ecosystem_dispatch.rs b/crates/socket-patch-cli/src/ecosystem_dispatch.rs index c0fa50176..21c84b038 100644 --- a/crates/socket-patch-cli/src/ecosystem_dispatch.rs +++ b/crates/socket-patch-cli/src/ecosystem_dispatch.rs @@ -548,14 +548,13 @@ impl NpmRootsCrawler<'_> { /// The installed copy of each npm purl in `purls`, found by its /// `package.json` identity (a full npm crawl) instead of its install path. -/// An npm ALIAS dependency (`"lp": "npm:left-pad@1.3.0"`) is installed under -/// its dependency key (`node_modules/lp`), which the name-keyed resolvers -/// above never probe; callers use this as the last lookup before calling a +/// An npm ALIAS dependency (`"lp": "npm:left-pad@1.3.0"`) reached through a +/// symlinked importer entry is one the name-keyed resolvers above never +/// take as a copy; callers use this as the last lookup before calling a /// package missing (`vendor`, and `vex` for a purl nothing else found). /// The crawl dedups by name@version, so this yields ONE copy per purl — the /// first the crawl reaches — never every alias beside a normal install -/// (`vex`'s per-dir alias walk, `vex_consumed::npm_alias_copies`, finds -/// those). Keyed by the caller's spelling; a purl with no copy has no +/// (the core resolver's alias pass returns those). Keyed by the caller's spelling; a purl with no copy has no /// entry. No crawl runs when `purls` is empty. pub(crate) async fn npm_paths_by_identity( options: &CrawlerOptions, diff --git a/crates/socket-patch-cli/src/json_envelope.rs b/crates/socket-patch-cli/src/json_envelope.rs index 2a76a9811..68bfcdfe5 100644 --- a/crates/socket-patch-cli/src/json_envelope.rs +++ b/crates/socket-patch-cli/src/json_envelope.rs @@ -152,6 +152,65 @@ impl Envelope { self.events.push(event); } + /// Re-tag every `Applied` event recorded at or after index `since` as + /// `action` (`Skipped` or `Failed`) with `code` and `message`, keeping + /// the summary in step: for packages a later step of the same run + /// undid (a refused group commit, a rolled-back eject), which must not + /// be reported or counted as applied. Their file lists are dropped (the + /// files are no longer there). The `skipped` advisories recorded for a + /// retracted package in the same span (`vendor_prebuilt_downloaded` + /// "vendored … from the patch service", `vendor_artifact_reused`, …) + /// describe that undone vendoring, so they are dropped too: the + /// re-tagged event is the package's one account. Returns how many + /// events were re-tagged. + pub fn retract_applied( + &mut self, + since: usize, + action: PatchAction, + code: &str, + message: &str, + ) -> usize { + let since = since.min(self.events.len()); + let retracted_purls: std::collections::HashSet = self.events[since..] + .iter() + .filter(|e| e.action == PatchAction::Applied) + .filter_map(|e| e.purl.clone()) + .collect(); + let mut index = 0; + let summary = &mut self.summary; + self.events.retain(|e| { + let keep = index < since + || e.action != PatchAction::Skipped + || !e.purl.as_ref().is_some_and(|p| retracted_purls.contains(p)); + index += 1; + if !keep { + summary.skipped = summary.skipped.saturating_sub(1); + } + keep + }); + let mut retracted = 0; + for event in self.events.iter_mut().skip(since) { + if event.action != PatchAction::Applied { + continue; + } + self.summary.applied = self.summary.applied.saturating_sub(1); + self.summary.bump(action); + event.action = action; + event.files.clear(); + event.error_code = Some(code.to_string()); + if action == PatchAction::Failed { + event.error = Some(message.to_string()); + } else { + event.reason = Some(message.to_string()); + } + retracted += 1; + } + if retracted > 0 && action == PatchAction::Failed { + self.mark_partial_failure(); + } + retracted + } + /// Mark the run as a partial failure. Idempotent. pub fn mark_partial_failure(&mut self) { if !matches!(self.status, Status::Error) { @@ -796,6 +855,52 @@ mod tests { assert_eq!(env.events.len(), 3); } + #[test] + fn retract_applied_retags_only_later_applied_events() { + let mut env = Envelope::new(Command::Vendor); + env.record(PatchEvent::new(PatchAction::Applied, "pkg:npm/early@1.0.0")); + let since = env.events.len(); + env.record(PatchEvent::new(PatchAction::Applied, "pkg:npm/a@1.0.0")); + env.record( + PatchEvent::new(PatchAction::Skipped, "pkg:npm/a@1.0.0") + .with_reason("vendor_prebuilt_downloaded", "advisory"), + ); + // An advisory for a package that was NOT retracted is kept. + env.record( + PatchEvent::new(PatchAction::Skipped, "pkg:npm/other@1.0.0") + .with_reason("vendor_bundled_instance_skipped", "advisory"), + ); + let n = env.retract_applied(since, PatchAction::Skipped, "eject_rolled_back", "undone"); + assert_eq!(n, 1); + assert_eq!(env.summary.applied, 1, "the earlier event is kept"); + // The retracted package's "vendored … from the patch service" + // advisory described the undone vendoring (#898, #1005): dropped. + assert_eq!(env.events.len(), 3, "{:?}", env.events); + assert_eq!(env.summary.skipped, 2); + assert!(!env + .events + .iter() + .any(|e| e.error_code.as_deref() == Some("vendor_prebuilt_downloaded"))); + assert_eq!( + env.events[2].error_code.as_deref(), + Some("vendor_bundled_instance_skipped") + ); + assert_eq!(env.events[1].action, PatchAction::Skipped); + assert_eq!( + env.events[1].error_code.as_deref(), + Some("eject_rolled_back") + ); + assert_eq!(env.events[1].reason.as_deref(), Some("undone")); + assert_eq!(env.status, Status::Success); + + let n = env.retract_applied(0, PatchAction::Failed, "refused", "boom"); + assert_eq!(n, 1); + assert_eq!(env.summary.applied, 0); + assert_eq!(env.summary.failed, 1); + assert_eq!(env.events[0].error.as_deref(), Some("boom")); + assert_eq!(env.status, Status::PartialFailure); + } + #[test] fn recording_failed_event_marks_partial_failure() { // The `status` invariant — "PartialFailure when any event has diff --git a/crates/socket-patch-cli/tests/covgap_commands_get.rs b/crates/socket-patch-cli/tests/covgap_commands_get.rs index e890104db..c3a71be73 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_get.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_get.rs @@ -3278,3 +3278,95 @@ async fn get_uuid_json_nested_apply_failure_names_the_patch() { "json={json}" ); } + +// =========================================================================== +// Nested apply mismatch-overwrite warnings in the JSON envelope (#1004) +// =========================================================================== + +/// Locally edited bytes: neither the patch's beforeHash nor its afterHash. +const LOCAL_EDIT_BYTES: &[u8] = b"vulnerable\n// local edit\n"; + +/// Whether `json["warnings"]` carries a `content_mismatch_overwritten` +/// entry naming `purl` and the overwritten file. +fn has_mismatch_warning(json: &serde_json::Value, purl: &str) -> bool { + json["warnings"].as_array().is_some_and(|ws| { + ws.iter().filter_map(|w| w.as_str()).any(|w| { + w.starts_with("(content_mismatch_overwritten) ") + && w.contains(purl) + && w.contains("package/index.js") + }) + }) +} + +/// The agent engine (`scan --mode agent --json`'s path) over an installed +/// file a local edit changed: the default policy overwrites it, and the +/// envelope must say so — the warning `apply --json` reports as a +/// `content_mismatch_overwritten` event — while the patch still counts as +/// applied. +#[tokio::test] +#[serial] +async fn engine_nested_apply_mismatch_overwrite_reaches_the_json_envelope() { + let server = MockServer::start().await; + mount_real_view(&server, UUID, PURL).await; + + let tmp = tempfile::tempdir().unwrap(); + write_project(tmp.path()); + let installed = tmp.path().join("node_modules").join(NAME).join("index.js"); + std::fs::write(&installed, LOCAL_EDIT_BYTES).unwrap(); + let mut params = engine_params(tmp.path()); + params.save_only = false; + let selected = vec![search_result(UUID, PURL)]; + let (code, json) = download_and_apply_patches(&selected, ¶ms, &server.uri()).await; + + assert_eq!(code, 0, "json={json}"); + assert_eq!(json["status"], "success", "json={json}"); + assert_eq!(json["applied"], 1, "json={json}"); + assert_eq!(json["patches"][0]["action"], "added", "json={json}"); + assert!( + has_mismatch_warning(&json, PURL), + "the overwrite must be reported: {json}" + ); + assert_eq!(std::fs::read(&installed).unwrap(), AFTER_BYTES); +} + +/// A clean apply carries no mismatch warning. +#[tokio::test] +#[serial] +async fn engine_nested_apply_clean_has_no_mismatch_warning() { + let server = MockServer::start().await; + mount_real_view(&server, UUID, PURL).await; + + let tmp = tempfile::tempdir().unwrap(); + write_project(tmp.path()); + let mut params = engine_params(tmp.path()); + params.save_only = false; + let selected = vec![search_result(UUID, PURL)]; + let (code, json) = download_and_apply_patches(&selected, ¶ms, &server.uri()).await; + assert_eq!(code, 0, "json={json}"); + assert!(json.get("warnings").is_none(), "json={json}"); +} + +/// `get --json` over a locally edited installed file: one JSON +/// document whose `warnings[]` names the overwrite, and nothing on stderr +/// (the envelope is the JSON caller's channel). +#[tokio::test] +async fn get_uuid_json_mismatch_overwrite_is_reported() { + let server = MockServer::start().await; + mount_real_view(&server, UUID, PURL).await; + + let tmp = tempfile::tempdir().unwrap(); + write_project(tmp.path()); + let installed = tmp.path().join("node_modules").join(NAME).join("index.js"); + std::fs::write(&installed, LOCAL_EDIT_BYTES).unwrap(); + let (code, stdout, stderr) = run_get_bin(tmp.path(), &server.uri(), &[UUID, "--json"]); + assert_eq!(code, 0, "stdout={stdout}\nstderr={stderr}"); + let json = parse_single_json_doc(&stdout); + assert_eq!(json["status"], "success", "json={json}"); + assert_eq!(json["applied"], 1, "json={json}"); + assert!( + has_mismatch_warning(&json, PURL), + "the overwrite must be reported: {json}" + ); + assert!(!stderr.contains("did not match"), "stderr={stderr}"); + assert_eq!(std::fs::read(&installed).unwrap(), AFTER_BYTES); +} diff --git a/crates/socket-patch-cli/tests/covgap_commands_scan_hosted.rs b/crates/socket-patch-cli/tests/covgap_commands_scan_hosted.rs index fba984c3e..6a9c07706 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_scan_hosted.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_scan_hosted.rs @@ -2605,6 +2605,91 @@ async fn vlt_decides_before_binary_bun_and_a_refused_uuid_is_never_confirmed() { assert_eq!(confirmed["redirect"]["redirected"], 1, "{confirmed:#}"); } +/// REGRESSION (#899): npm 12 never reads npm-shrinkwrap.json. A project +/// whose only npm lock is the shrinkwrap is still redirected (npm <= 11 +/// installs from it), but the run warns `redirect_npm_shrinkwrap_only` — +/// in `--json` and on human stderr — and the in-run `--vex` attests nothing +/// for it (`vex_npm_shrinkwrap_only`), as a lockfile-only `vex` does. +#[tokio::test] +async fn shrinkwrap_only_project_warns_npm12_ignores_it_and_vex_omits_it() { + let server = MockServer::start().await; + mock_discovery(&server, PURL, UUID).await; + mock_granted_reference(&server, UUID, PURL, HOSTED_URL).await; + mock_view(&server, UUID, PURL).await; + + let tmp = tempfile::tempdir().unwrap(); + write_npm_project(tmp.path(), NAME); + std::fs::rename( + tmp.path().join("package-lock.json"), + tmp.path().join("npm-shrinkwrap.json"), + ) + .unwrap(); + + let (code, doc) = scan_hosted_json( + tmp.path(), + &server.uri(), + &[ + "--vex", + "out.vex.json", + "--vex-product", + "pkg:npm/consumer@0.0.0", + "--patch-server-url", + "http://patch.test", + ], + &[], + ); + // Still redirected: npm <= 11 installs from the shrinkwrap. + assert_eq!(doc["redirect"]["redirected"], 1, "{doc:#}"); + let lock = std::fs::read_to_string(tmp.path().join("npm-shrinkwrap.json")).unwrap(); + assert!(lock.contains(HOSTED_URL), "{lock}"); + assert!( + !tmp.path().join("package-lock.json").exists(), + "no package-lock.json is invented" + ); + let detail = warning_detail(&doc, "redirect_npm_shrinkwrap_only"); + for needle in ["covgap-hosted@1.0.0", "npm >= 12", "no package-lock.json"] { + assert!(detail.contains(needle), "{needle}: {detail}"); + } + // The in-run VEX omits it, so the requested VEX fails the run. + assert_eq!(code, 1, "{doc:#}"); + assert_eq!(doc["error"]["code"], "no_applicable_patches", "{doc:#}"); + assert!( + doc["warnings"] + .as_array() + .is_some_and(|w| w.iter().any(|w| w["code"] == "vex_npm_shrinkwrap_only" + && w["detail"].as_str().is_some_and(|d| d.contains(PURL)))), + "{doc:#}" + ); + assert!(!tmp.path().join("out.vex.json").exists()); + + // With the package-lock.json twin npm 12 reads, the re-run rewires it + // too, the warning is gone and the in-run VEX attests. + std::fs::copy( + tmp.path().join("npm-shrinkwrap.json"), + tmp.path().join("package-lock.json"), + ) + .unwrap(); + let (code, doc) = scan_hosted_json( + tmp.path(), + &server.uri(), + &[ + "--vex", + "out.vex.json", + "--vex-product", + "pkg:npm/consumer@0.0.0", + "--patch-server-url", + "http://patch.test", + ], + &[], + ); + assert_eq!(code, 0, "{doc:#}"); + assert!( + !warning_codes(&doc).contains(&"redirect_npm_shrinkwrap_only".to_string()), + "{doc:#}" + ); + assert_eq!(doc["vex"]["statements"], 1, "{doc:#}"); +} + // ───────────────── yarn classic berry-migration advisory ───────────────── /// Yarn classic project: package.json (with `package_manager` when given) + diff --git a/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs b/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs index 48217a391..f85c2eeb4 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs @@ -2744,3 +2744,42 @@ async fn scan_agent_json_nested_apply_failure_reaches_the_apply_block() { ); assert_eq!(std::fs::read(member.join("index.js")).unwrap(), b"before\n",); } + +/// `scan --mode agent --json` over an installed file a local edit changed +/// (neither beforeHash nor afterHash): the default policy overwrites it, +/// and the `apply` block's `warnings[]` must report that overwrite — the +/// same `content_mismatch_overwritten` warning `apply --json` and the human +/// scan print — instead of dropping it (#1004). +#[tokio::test] +async fn scan_agent_json_mismatch_overwrite_reaches_the_apply_block() { + let mock = MockServer::start().await; + let purl = "pkg:npm/locally-edited@1.0.0"; + mount_one_patch_api(&mock, purl, b"before\n").await; + + let tmp = tempfile::tempdir().unwrap(); + write_root_package_json(tmp.path()); + write_npm_package( + tmp.path(), + "locally-edited", + "1.0.0", + b"before\n// local edit\n", + ); + + let (code, stdout, stderr) = run_scan_agent(tmp.path(), &mock.uri(), &["--json"]); + assert_eq!(code, 0, "stdout={stdout}\nstderr={stderr}"); + let v: serde_json::Value = serde_json::from_str(stdout.trim()).expect("one JSON envelope"); + let apply = &v["apply"]; + assert_eq!(apply["applied"], 1, "{v}"); + let warned = apply["warnings"].as_array().is_some_and(|ws| { + ws.iter().filter_map(|w| w.as_str()).any(|w| { + w.starts_with("(content_mismatch_overwritten) ") + && w.contains(purl) + && w.contains("package/index.js") + }) + }); + assert!(warned, "the overwrite must be reported: {v}"); + assert_eq!( + std::fs::read(tmp.path().join("node_modules/locally-edited/index.js")).unwrap(), + b"after\n" + ); +} diff --git a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs index 7f3f5cca1..3a3b33f2e 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs @@ -1163,11 +1163,12 @@ async fn reconcile_state_write_failure_reports_failed_after_removal() { /// writes) after the backend already wrote the artifact: the run's ONE /// commit of the lock rewire and the ledger fails as a whole, so neither is /// written — the lock keeps its pre-run bytes, no ledger appears — and the -/// run exits 1 with the top-level `vendor_commit_failed` error. The -/// package's `Applied` event still reports what the backend did; the -/// artifact is an orphan the next run re-vendors over. (Before the group -/// commit this was a per-package `vendor_state_write_failed` next to an -/// already-rewired lock.) +/// run exits 1 with the top-level `vendor_commit_failed` error. Like the +/// symlink refusal (#898), nothing of the package stands: it is a `failed` +/// event with that code, not `applied`, the artifact the backend wrote is +/// removed, and the human run claims no vendored package. (Before the +/// group commit this was a per-package `vendor_state_write_failed` next to +/// an already-rewired lock.) #[cfg(unix)] #[tokio::test] async fn vendor_state_write_failure_reports_failed_event() { @@ -1180,12 +1181,19 @@ async fn vendor_state_write_failure_reports_failed_event() { let (code, env) = vendor_cli(fx.root(), &[]); assert_eq!(code, 1, "{env:#}"); - let applied = find_event(&env, "applied", None); - assert_eq!(applied["purl"], PURL, "the backend vendored: {env:#}"); assert_eq!(env["error"]["code"], "vendor_commit_failed", "{env:#}"); + assert_eq!(env["summary"]["applied"], 0, "{env:#}"); + let failed = find_event(&env, "failed", Some("vendor_commit_failed")); + assert_eq!(failed["purl"], PURL, "{env:#}"); assert!( - fx.tgz_path().is_file(), - "the artifact the backend wrote is on disk" + !events(&env) + .iter() + .any(|e| e["action"] == "applied" || e["errorCode"] == "vendor_prebuilt_downloaded"), + "nothing reports the uncommitted vendoring as done: {env:#}" + ); + assert!( + !fx.tgz_path().exists(), + "the uncommitted run leaves no orphan artifact" ); assert!(!fx.state_path().exists(), "the ledger write failed"); assert_eq!( @@ -1193,6 +1201,17 @@ async fn vendor_state_write_failure_reports_failed_event() { fx.original_lock, "the lock rewire is committed with the ledger or not at all" ); + + let (code, stdout, stderr) = run_cli( + fx.root(), + &["vendor", "--cwd", fx.root().to_str().unwrap()], + &[], + ); + assert_eq!(code, 1, "{stdout}\n{stderr}"); + assert!( + !stdout.contains("Vendored 1 package") && !stdout.contains("Next steps"), + "{stdout}\n{stderr}" + ); } /// #627: a symlinked lockfile (a lock shared with another checkout) is @@ -1264,9 +1283,70 @@ async fn vendor_refuses_a_symlinked_lock_instead_of_replacing_it() { !fx.state_path().exists(), "{linked}: no ledger entry was committed" ); + // #898: "nothing was written" holds — the artifact the loop + // downloaded before the commit was refused is removed — and the + // package is reported refused, not applied. + assert!( + !fx.tgz_path().exists(), + "{linked}: the refused run leaves no orphan artifact" + ); + assert_eq!(env["summary"]["applied"], 0, "{linked}: {env:#}"); + let refused = find_event(&env, "failed", Some("redirect_symlinked_file_unsupported")); + assert_eq!(refused["purl"], PURL, "{linked}: {env:#}"); + + // The human run neither claims a vendored package nor advises + // committing `.socket/vendor/`. + let (code, stdout, stderr) = run_cli( + fx.root(), + &["vendor", "--cwd", fx.root().to_str().unwrap()], + &[], + ); + assert_eq!(code, 1, "{linked}: {stdout}\n{stderr}"); + assert!( + stderr.contains(&format!("{linked} is a symbolic link")), + "{linked}: {stdout}\n{stderr}" + ); + assert!( + !stdout.contains("Vendored 1 package") && !stdout.contains("Next steps"), + "{linked}: {stdout}\n{stderr}" + ); + assert!(!fx.tgz_path().exists(), "{linked}"); } } +/// #898: a refused commit removes only the artifact dirs the run ADDED. +/// An artifact the pre-run ledger already names, deleted and redownloaded +/// in place by this run, is referenced without any commit, so the refusal +/// keeps it (the ledger still points there). +#[cfg(unix)] +#[tokio::test] +async fn refused_commit_keeps_an_artifact_the_ledger_already_names() { + let fx = npm_fixture(); + assert_eq!(vendor_run(vendor_args(fx.root())).await, 0, "stage vendor"); + let state_before = std::fs::read(fx.state_path()).unwrap(); + // The artifact is lost and the lock re-resolved from the registry + // (e.g. a fresh `npm install`), and the lock is now a shared symlink. + std::fs::remove_dir_all(fx.tgz_path().parent().unwrap()).unwrap(); + let shared = tempfile::tempdir().unwrap(); + let target = shared.path().join("package-lock.json"); + std::fs::write(&target, &fx.original_lock).unwrap(); + std::fs::remove_file(fx.lock_path()).unwrap(); + std::os::unix::fs::symlink(&target, fx.lock_path()).unwrap(); + + let (code, env) = vendor_cli(fx.root(), &[]); + assert_eq!(code, 1, "{env:#}"); + assert_eq!( + env["error"]["code"], "redirect_symlinked_file_unsupported", + "{env:#}" + ); + assert!( + fx.tgz_path().is_file(), + "the artifact the pre-run ledger names is kept: {env:#}" + ); + assert_eq!(std::fs::read(fx.state_path()).unwrap(), state_before); + assert_eq!(std::fs::read(&target).unwrap(), fx.original_lock); +} + /// #627 follow-up: an already-vendored package whose lock is LATER made a /// symlink writes nothing on a re-run, so neither the dry run nor the wet /// run may predict or raise the symlink refusal: the dry run previews it as diff --git a/crates/socket-patch-cli/tests/e2e_redirect_npm_build.rs b/crates/socket-patch-cli/tests/e2e_redirect_npm_build.rs index a10fa957e..d1513aba2 100644 --- a/crates/socket-patch-cli/tests/e2e_redirect_npm_build.rs +++ b/crates/socket-patch-cli/tests/e2e_redirect_npm_build.rs @@ -996,7 +996,9 @@ fn post_install_vex(fresh: &Path, server: &str) { /// renamed), npm 12's shrinkwrap + package-lock.json twin (the command is /// gone and installs read the twin). The redirect must land in EVERY /// committed lock, the fresh `npm ci` must install the patched bytes, and -/// the manifest-less VEX tail must attest them (and stop once reverted). +/// the manifest-less VEX tail must attest them (and stop once reverted) — +/// a shrinkwrap-only checkout only once its package-lock.json twin is +/// committed (npm 12 never reads the shrinkwrap, #899). #[tokio::test(flavor = "multi_thread")] #[ignore = "wall-bound real-npm install (~150s); runs on all 3 OSes as an e2e CI matrix leg"] async fn npm_redirect_shrinkwrap_fresh_checkout_and_manifestless_vex() { diff --git a/crates/socket-patch-cli/tests/e2e_socket_yml_policy.rs b/crates/socket-patch-cli/tests/e2e_socket_yml_policy.rs index 4a742d2dd..16ea58d3b 100644 --- a/crates/socket-patch-cli/tests/e2e_socket_yml_policy.rs +++ b/crates/socket-patch-cli/tests/e2e_socket_yml_policy.rs @@ -1092,6 +1092,119 @@ async fn agent_mode_retains_a_recorded_patch_the_policy_now_excludes() { assert_eq!(doc["policy"]["retained"][0]["upgradeAvailable"], true); } +/// #554: an agent scan at the repo root crawls every nested project's +/// `node_modules`; each copy is judged by the project root that owns it, +/// so `ignorePaths` and the built-in `test/` default skip nested projects. +#[tokio::test] +#[serial] +async fn agent_mode_judges_nested_project_copies_by_their_own_root() { + let server = MockServer::start().await; + mount_api(&server, catalog()).await; + let repo = Repo::new(Some("version: 2\npatches:\n ignorePaths: [\"/services/legacy/\"]\n")); + // left-pad is installed in both web (admitted) and legacy (ignored). + write_npm_root(&repo.dir("services/legacy"), &["gamma", "left-pad"]); + let (code, doc) = scan_json(&repo.root, &server.uri(), &["--mode", "agent"], &[]); + assert_eq!(code, 0, "{doc:#}"); + let index = |rel: &str, name: &str| index_at(&repo, rel, name); + assert_eq!( + index("services/web", "alpha"), + patched_index("alpha"), + "{doc:#}" + ); + assert_eq!( + index("services/legacy", "gamma"), + orig_index("gamma"), + "ignored by patches.ignorePaths" + ); + assert_eq!( + index("services/test", "delta"), + orig_index("delta"), + "a discovered test/ root is a built-in ignore" + ); + let mut roots: Vec<(String, String)> = doc["policy"]["filtered"] + .as_array() + .unwrap() + .iter() + .filter(|f| f["purl"].is_null()) + .map(|f| { + ( + f["project"].as_str().unwrap().to_string(), + f["reason"].as_str().unwrap().to_string(), + ) + }) + .collect(); + roots.sort(); + assert_eq!( + roots, + [ + ( + "services/legacy".to_string(), + "policy_path_excluded".to_string() + ), + ( + "services/test".to_string(), + "policy_path_excluded".to_string() + ), + ], + "{:#}", + doc["policy"] + ); + // A package an admitted project also installs is patched per package + // version, so its copy in the ignored project is patched too: say so. + assert!( + warning_codes(&doc).contains(&"policy_shared_copy".to_string()), + "{doc:#}" + ); + let shared = doc["warnings"] + .as_array() + .unwrap() + .iter() + .find(|w| w["code"] == "policy_shared_copy") + .unwrap(); + let detail = shared["detail"].as_str().unwrap(); + assert!( + detail.contains("pkg:npm/left-pad@1.0.0") && detail.contains("services/legacy"), + "{detail}" + ); + + // includePaths: the docs' headline example selects nested projects from + // the repo root (which itself is not included). + let repo = Repo::new(Some("version: 2\npatches:\n includePaths: [\"/services/legacy/\"]\n")); + let (code, doc) = scan_json(&repo.root, &server.uri(), &["--mode", "agent"], &[]); + assert_eq!(code, 0, "{doc:#}"); + assert_eq!( + index_at(&repo, "services/legacy", "gamma"), + patched_index("gamma"), + "{doc:#}" + ); + assert_eq!( + index_at(&repo, "services/web", "alpha"), + orig_index("alpha"), + "{doc:#}" + ); + let web_root = doc["policy"]["filtered"] + .as_array() + .unwrap() + .iter() + .find(|f| f["purl"].is_null() && f["project"] == "services/web"); + assert_eq!( + web_root.map(|f| &f["reason"]), + Some(&json!("policy_path_not_included")), + "{:#}", + doc["policy"] + ); +} + +fn index_at(repo: &Repo, rel: &str, name: &str) -> String { + std::fs::read_to_string( + repo.dir(rel) + .join("node_modules") + .join(name) + .join("index.js"), + ) + .unwrap() +} + // --------------------------------------------------------------------------- // Vendored // --------------------------------------------------------------------------- diff --git a/crates/socket-patch-cli/tests/e2e_vendor_npm_build.rs b/crates/socket-patch-cli/tests/e2e_vendor_npm_build.rs index 6c141a4dc..7563f4e84 100644 --- a/crates/socket-patch-cli/tests/e2e_vendor_npm_build.rs +++ b/crates/socket-patch-cli/tests/e2e_vendor_npm_build.rs @@ -1058,8 +1058,9 @@ async fn npm_get_uuid_vendored_fresh_checkout_npm_ci() { /// removed the command and keeps a package-lock.json twin beside a committed /// shrinkwrap — and installs FROM the twin — so BOTH locks must be rewired /// (the dual-lock vendor fix) for the fresh `npm ci` to install the patched -/// bytes; the manifest-less tail then attests them, and revert restores -/// every lock byte-for-byte. +/// bytes; the manifest-less tail then attests them (a shrinkwrap-only +/// checkout only once its package-lock.json twin is committed: npm 12 never +/// reads the shrinkwrap, #899), and revert restores every lock byte-for-byte. #[test] fn npm_vendor_shrinkwrap_fresh_checkout_npm_ci_and_manifestless_vex() { let suite = "e2e_vendor_npm_build (shrinkwrap)"; diff --git a/crates/socket-patch-cli/tests/e2e_vex_lockfile/npm.rs b/crates/socket-patch-cli/tests/e2e_vex_lockfile/npm.rs index d072ef1af..bb6baaa04 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_lockfile/npm.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_lockfile/npm.rs @@ -21,8 +21,10 @@ //! pin); a uuid-shaped segment on a NON-Socket host is nothing at all (no //! ref, zero API requests); a record naming another package or another //! uuid is `record_mismatch`; a vendored artifact attests, and a tampered -//! member is `vendor_hash_mismatch`; and the npm 12 dual-lock hazard — one -//! lock wired, the other still on the registry — attests nothing. +//! member is `vendor_hash_mismatch`; the npm 12 dual-lock hazard — one +//! lock wired, the other still on the registry — attests nothing; and a +//! `shrinkwrap` with no package-lock.json twin attests nothing either (npm +//! 12 never reads it, #899), so the attesting cells cover the other shapes. use crate::vex_e2e_common; @@ -61,6 +63,11 @@ const SHAPES: [Shape; 5] = [ Shape::Dual, ]; +/// The shapes every npm major installs the wiring from: all but a lone +/// `shrinkwrap`, which npm 12 ignores +/// ([`shrinkwrap_without_a_package_lock_twin_attests_nothing`]). +const ATTESTABLE_SHAPES: [Shape; 4] = [Shape::V1, Shape::V2, Shape::V3, Shape::Dual]; + fn hosted_url(host: &str, uuid: &str) -> String { format!("https://{host}/patch/npm/{NAME}/{VERSION}/{TOKEN}/{uuid}/{NAME}-{VERSION}.tgz") } @@ -169,13 +176,13 @@ fn assert_no_manifest_or_ledgers(p: &Path) { } } -/// Hosted, nothing installed: every shape attests `(redirected)` from the +/// Hosted, nothing installed: every attestable shape attests `(redirected)` from the /// patched sha512 pin (the in-run `scan --mode hosted --vex` basis) with /// the API's record; `--offline` is `record_unavailable` with zero requests. #[test] fn hosted_uninstalled_attests_from_the_integrity_pin_in_every_shape() { let api = api_for(UUID, PURL); - for shape in SHAPES { + for shape in ATTESTABLE_SHAPES { let tmp = tempfile::tempdir().unwrap(); let p = tmp.path(); write_locks(p, shape, &hosted_url("patch.socket.dev", UUID), PIN); @@ -203,7 +210,7 @@ fn hosted_uninstalled_attests_from_the_integrity_pin_in_every_shape() { #[test] fn hosted_installed_tree_is_the_evidence_in_every_shape() { let api = api_for(UUID, PURL); - for shape in SHAPES { + for shape in ATTESTABLE_SHAPES { for (label, bytes, attested) in [ ("patched", PATCHED, true), ("pristine", PRISTINE, false), @@ -300,13 +307,13 @@ fn record_purl_or_uuid_mismatch_is_record_mismatch() { } } -/// Vendored (v2 / v3 / shrinkwrap / dual — vendoring refuses v1): the +/// Vendored (v2 / v3 / dual — vendoring refuses v1): the /// committed artifact is the evidence, nothing need be installed; a tampered /// artifact MEMBER is `vendor_hash_mismatch` even with a patched install. #[test] fn vendored_artifact_is_the_evidence_and_a_tampered_member_is_omitted() { let api = api_for(UUID, PURL); - for shape in [Shape::V2, Shape::V3, Shape::Shrinkwrap, Shape::Dual] { + for shape in [Shape::V2, Shape::V3, Shape::Dual] { let tmp = tempfile::tempdir().unwrap(); let p = tmp.path(); write_locks(p, shape, &format!("file:{}", vendored_rel(UUID)), PIN); @@ -361,6 +368,43 @@ fn dual_lock_with_one_lock_on_the_registry_attests_nothing() { } } +/// REGRESSION (#899): a shrinkwrap with NO package-lock.json twin attests +/// nothing. npm 12 never reads npm-shrinkwrap.json — it resolves the tree +/// from the registry and writes a fresh package-lock.json — so the wiring +/// reaches npm <= 11 only. Hosted (uninstalled, and installed patched under +/// an npm <= 11) and vendored: omitted `vex_npm_shrinkwrap_only`, and the +/// run says why. +#[test] +fn shrinkwrap_without_a_package_lock_twin_attests_nothing() { + let api = api_for(UUID, PURL); + for (mode, wired, installed) in [ + ("hosted", hosted_url("patch.socket.dev", UUID), false), + ( + "hosted installed", + hosted_url("patch.socket.dev", UUID), + true, + ), + ("vendored", format!("file:{}", vendored_rel(UUID)), false), + ] { + let tmp = tempfile::tempdir().unwrap(); + let p = tmp.path(); + write_locks(p, Shape::Shrinkwrap, &wired, PIN); + write_artifact(p, UUID, PATCHED); + if installed { + install(p, PATCHED); + } + let out = run_vex(&binary(), p, &VexRun::online(&api)); + assert_eq!(out.code, Some(1), "{mode}:\n{out}"); + assert_not_attested(&out.envelope, PURL, "vex_npm_shrinkwrap_only"); + assert_absent(out.doc.as_ref(), PURL); + let text = out.stdout.clone() + &out.stderr; + assert!( + text.contains("no package-lock.json") && text.contains("npm >= 12"), + "the shrinkwrap-only wiring must be explained:\n{out}" + ); + } +} + /// REGRESSION (#798): the same holds when the other lock has NO entry for /// the package (a stale twin an npm <= 11 teammate left behind): npm /// re-resolves the missing entry from the registry, so whichever npm major diff --git a/crates/socket-patch-cli/tests/e2e_vex_vendor.rs b/crates/socket-patch-cli/tests/e2e_vex_vendor.rs index b44d90c04..f67b4d4df 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_vendor.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_vendor.rs @@ -1608,6 +1608,248 @@ fn vendored_npm_patch_with_an_unwired_registry_copy_in_the_same_lock() { } } +/// REGRESSION (#879): npm 7-12 `npm install` on a vendored lockfileVersion +/// 2 lock (or shrinkwrap) re-saves the legacy `dependencies` mirror node +/// WITHOUT `resolved` (npm never writes one for a `file:` resolution +/// there) but with the patched `integrity`. npm 6 fails closed on that pin +/// and npm 7+ installs from the still-wired `packages` entry, so `vex` +/// attests and `vendor --check` passes. The same node pinned to the +/// registry tarball's integrity is what npm 6 installs unpatched: both +/// still fail (#432). +#[test] +fn vendored_npm_v2_mirror_without_resolved_keeps_the_patch_wired() { + let purl = "pkg:npm/lodash@4.17.21"; + let uuid = "0a0a0a0a-8790-4879-8879-0a0a0a0a0a0a"; + let patched = b"patched npm bytes\n"; + let after_hash = compute_git_sha256_from_bytes(patched); + for lock_name in ["package-lock.json", "npm-shrinkwrap.json"] { + for (label, mirror_pin, wired) in [ + ("patched pin", "sha512-cGF0Y2hlZA==", true), + ("registry pin", "sha512-T1JJR0lOQUw=", false), + ] { + let label = format!("{lock_name} {label}"); + let tmp = tempfile::tempdir().expect("create tempdir"); + let cwd = tmp.path(); + let rel = format!(".socket/vendor/npm/{uuid}/lodash-4.17.21.tgz"); + let sha256 = sha256_hex(&write_member_tgz( + &cwd.join(&rel), + "package/index.js", + patched, + )); + let record = make_record( + uuid, + "package/index.js", + &after_hash, + "GHSA-mirr-aaaa", + &["CVE-2026-879"], + ); + let mut wiring = write_matrix_wiring(cwd, "npm", uuid, &rel); + // What npm 8's `npm install` leaves: lockfileVersion 2 with a + // `resolved`-less mirror node. + let written = cwd.join("package-lock.json"); + let mut lock: Value = + serde_json::from_str(&std::fs::read_to_string(&written).unwrap()).unwrap(); + lock["lockfileVersion"] = serde_json::json!(2); + lock["dependencies"] = serde_json::json!({ + "lodash": { "version": "4.17.21", "integrity": mirror_pin } + }); + // The shrinkwrap shape keeps its package-lock.json twin: a + // shrinkwrap-only project is omitted from VEX on its own (#899). + std::fs::write(&written, lock.to_string()).unwrap(); + std::fs::write(cwd.join(lock_name), lock.to_string()).unwrap(); + wiring.file = lock_name.to_string(); + let mut state = VendorState::new(); + state.entries.insert( + purl.to_string(), + detached_matrix_entry("npm", purl, uuid, &rel, sha256, record, wiring), + ); + let dir = cwd.join(".socket/vendor"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("state.json"), + serde_json::to_string_pretty(&state).unwrap(), + ) + .unwrap(); + + let vex_path = cwd.join("out.vex.json"); + let out = cli() + .args([ + "vex", + "--cwd", + cwd.to_str().unwrap(), + "--json", + "--output", + vex_path.to_str().unwrap(), + "--product", + "pkg:npm/app@1.0.0", + ]) + .output() + .expect("invoke vex"); + let env = String::from_utf8_lossy(&out.stdout).into_owned(); + let check = cli() + .args([ + "vendor", + "--check", + "--cwd", + cwd.to_str().unwrap(), + "--json", + ]) + .output() + .expect("invoke vendor --check"); + let check_env = String::from_utf8_lossy(&check.stdout).into_owned(); + if wired { + assert!(out.status.success(), "{label}: {env}"); + let doc: Value = + serde_json::from_str(&std::fs::read_to_string(&vex_path).unwrap()).unwrap(); + assert_eq!( + doc["statements"].as_array().unwrap().len(), + 1, + "{label}: {doc}" + ); + assert!(check.status.success(), "{label}: {check_env}"); + } else { + assert_eq!(out.status.code(), Some(1), "{label}: {env}"); + assert!(!vex_path.exists(), "{label}: {env}"); + assert_eq!(check.status.code(), Some(1), "{label}: {check_env}"); + } + } + } +} + +/// REGRESSION (#753): `lodash@4.17.21` sits beneath a dependency that +/// ships its own npm-shrinkwrap.json (`"hasShrinkwrap": true`), and npm +/// 7–11 install that copy from the dependency's shrinkwrap, ignoring the +/// root lock. Neither a pre-fix lock whose ONLY copy (the nested one) was +/// rewired to the vendored tarball, nor a vendored hoisted copy beside an +/// unpatched nested one, may be attested by `vex`, and `vendor --check` +/// must fail naming the shrinkwrapped copy (re-vendoring cannot reach it, +/// so its generic "wiring missing; re-run vendor" advice would be wrong). +#[test] +fn vendored_npm_patch_with_a_copy_under_a_has_shrinkwrap_dependency() { + let purl = "pkg:npm/lodash@4.17.21"; + let uuid = "0a0a0a0a-7537-4537-8537-0a0a0a0a0a0a"; + let patched = b"patched npm bytes\n"; + let after_hash = compute_git_sha256_from_bytes(patched); + let nested = "node_modules/sw/node_modules/lodash"; + for (label, only_nested_wired) in [ + ("pre-fix lock, only the nested copy wired", true), + ("hoisted wired, nested registry copy", false), + ] { + let tmp = tempfile::tempdir().expect("create tempdir"); + let cwd = tmp.path(); + let rel = format!(".socket/vendor/npm/{uuid}/lodash-4.17.21.tgz"); + let sha256 = sha256_hex(&write_member_tgz( + &cwd.join(&rel), + "package/index.js", + patched, + )); + let record = make_record( + uuid, + "package/index.js", + &after_hash, + "GHSA-shrk-aaaa", + &["CVE-2026-753"], + ); + let wiring = write_matrix_wiring(cwd, "npm", uuid, &rel); + let lock_path = cwd.join("package-lock.json"); + let mut lock: Value = + serde_json::from_str(&std::fs::read_to_string(&lock_path).unwrap()).unwrap(); + let packages = lock["packages"].as_object_mut().unwrap(); + let hoisted = packages["node_modules/lodash"].clone(); + packages.insert( + "node_modules/sw".to_string(), + serde_json::json!({ + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-U1c=", + "hasShrinkwrap": true + }), + ); + if only_nested_wired { + packages.shift_remove("node_modules/lodash"); + packages.insert(nested.to_string(), hoisted); + } else { + packages.insert( + nested.to_string(), + serde_json::json!({ + "version": "4.17.21", + "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz", + "integrity": "sha512-T1JJR0lOQUw=" + }), + ); + } + std::fs::write(&lock_path, lock.to_string()).unwrap(); + let mut state = VendorState::new(); + state.entries.insert( + purl.to_string(), + detached_matrix_entry("npm", purl, uuid, &rel, sha256, record, wiring), + ); + let dir = cwd.join(".socket/vendor"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("state.json"), + serde_json::to_string_pretty(&state).unwrap(), + ) + .unwrap(); + + let vex_path = cwd.join("out.vex.json"); + let out = cli() + .args([ + "vex", + "--cwd", + cwd.to_str().unwrap(), + "--json", + "--output", + vex_path.to_str().unwrap(), + "--product", + "pkg:npm/app@1.0.0", + ]) + .output() + .expect("invoke vex"); + let env: Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "{label}: vex envelope JSON on stdout ({e}): {}", + String::from_utf8_lossy(&out.stdout) + ) + }); + assert_eq!(out.status.code(), Some(1), "{label}: {env}"); + assert!( + !vex_path.exists(), + "{label}: no VEX document may attest the purl: {env}" + ); + assert!( + env.to_string().contains(nested), + "{label}: the envelope names the shrinkwrapped copy: {env}" + ); + + let check = cli() + .args([ + "vendor", + "--check", + "--cwd", + cwd.to_str().unwrap(), + "--json", + ]) + .output() + .expect("invoke vendor --check"); + let check_env: Value = serde_json::from_slice(&check.stdout).unwrap_or_else(|e| { + panic!( + "{label}: vendor --check envelope JSON on stdout ({e}): {}", + String::from_utf8_lossy(&check.stdout) + ) + }); + assert_eq!(check.status.code(), Some(1), "{label}: {check_env}"); + let event = &check_env["events"][0]; + assert_eq!(event["errorCode"], "vendor_check_failed", "{check_env}"); + assert!( + event["reason"].as_str().is_some_and(|r| r.contains(nested) + && r.contains("hasShrinkwrap") + && !r.contains("re-run `socket-patch vendor`")), + "{label}: the check names the shrinkwrapped copy: {check_env}" + ); + } +} + // ────────────────────────────────────────────────────────────────────── // 8. an applied, byte-verified agent-mode patch attests whether or not its // ecosystem has an install hook (there is no setup-state filter). diff --git a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs index 6ec4c8e86..070480ecc 100644 --- a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs +++ b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs @@ -1439,6 +1439,42 @@ async fn npm_hosted_round_trip_manifest_less_vex() { }); } +/// REGRESSION (#828, yarn classic twin): a hosted registry block beside a +/// git-sourced block of the same `name@version` (the hosted rewriter skips +/// the git block, `redirect_yarn_classic_git_skipped`). Discovery withholds +/// the pin from VEX, but rollback must still restore its upstream entry — +/// it refused it as `hosted_wiring_contested` / `patched_ref_unattributable` +/// with a remedy (re-run the hosted scan) that only rewrote the same state. +/// The git block is left exactly as it was. +#[tokio::test] +#[serial] +async fn yarn_classic_hosted_pin_beside_a_git_copy_rolls_back() { + let server = MockServer::start().await; + mock_yarn_registry(&server, "left-pad", "1.2.3").await; + let git_block = "\"left-pad@git+https://github.com/stevemao/left-pad.git#v1.2.3\":\n \ + version \"1.2.3\"\n \ + resolved \"git+https://github.com/stevemao/left-pad.git#5e5f1a6e23f6fa2bd1e4a3d0c2bb1c0e1bb0f00a\""; + let tmp = tempfile::tempdir().unwrap(); + std::fs::write( + tmp.path().join("yarn.lock"), + yarn_lock_content(&format!("{git_block}\n\n{}", yarn_redirected_block())), + ) + .unwrap(); + + let (code, envelope) = run_rollback_subprocess_online(tmp.path(), &server, &[]); + assert_eq!(code, 0, "rollback must restore the pin: {envelope}"); + assert_eq!( + envelope["hosted"]["reverted"], + serde_json::json!([LP_PURL]), + "{envelope}" + ); + assert_eq!( + std::fs::read_to_string(tmp.path().join("yarn.lock")).unwrap(), + yarn_lock_content(&format!("{git_block}\n\n{}", yarn_original_block())), + "only the hosted block is restored" + ); +} + // --------------------------------------------------------------------------- // Agent record superseded by a hosted pin (#933) // --------------------------------------------------------------------------- diff --git a/crates/socket-patch-cli/tests/in_process_vendor.rs b/crates/socket-patch-cli/tests/in_process_vendor.rs index 26f566a13..93dbf0c24 100644 --- a/crates/socket-patch-cli/tests/in_process_vendor.rs +++ b/crates/socket-patch-cli/tests/in_process_vendor.rs @@ -775,6 +775,187 @@ async fn revendor_new_uuid_cleans_stale_artifact_and_still_reverts() { assert!(!fx.vendor_dir().exists(), "vendor tree fully pruned"); } +// ───────────────────────────────────────────────────────────────────── +// 8c'. a superseding patch the service has not built keeps the old one +// ───────────────────────────────────────────────────────────────────── + +/// #954: the package is vendored at `UUID`, then its patch moves to `UUID2` +/// whose prebuilt artifact the service has not built (`pending_build`) or +/// cannot serve (`build_failed`, `not_found`). The older vendoring is still +/// in force — nothing is touched — so the run keeps it and reports a skip +/// under the unserved code (exit 0), the way hosted mode keeps its pin, +/// instead of failing every re-run until the server builds the artifact. +#[tokio::test] +async fn superseding_patch_without_a_served_artifact_keeps_the_vendored_one() { + use wiremock::matchers::{method, path_regex}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + const UUID2: &str = "0a1b2c3d-4e5f-6a7b-8c9d-0e1f2a3b4c5d"; + for (status, code) in [ + ("pending_build", "vendor_prebuilt_pending"), + ("build_failed", "vendor_prebuilt_unavailable"), + ("not_found", "vendor_prebuilt_unavailable"), + ] { + let fx = npm_fixture(); + assert_eq!(vendor_run(vendor_args(fx.root())).await, 0); + let wired_lock = fx.lock_bytes(); + let state_before = std::fs::read(fx.state_path()).unwrap(); + + let mut manifest: Value = + serde_json::from_slice(&std::fs::read(fx.manifest_path()).unwrap()).unwrap(); + manifest["patches"][PURL]["uuid"] = json!(UUID2); + std::fs::write( + fx.manifest_path(), + serde_json::to_vec_pretty(&manifest).unwrap(), + ) + .unwrap(); + + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path_regex(r"/(patch|patches)/package$")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(json!({ "results": { UUID2: { "status": status } } })), + ) + .mount(&server) + .await; + let uri = server.uri(); + let (code_out, env) = vendor_cli( + fx.root(), + &[ + "--api-url", + &uri, + "--vendor-url", + &uri, + "--api-token", + "sktsec_placeholder_value_for_tests_api", + "--org", + "acme", + "--lock-timeout", + "5", + ], + ); + assert_eq!(code_out, 0, "{status}: the older patch is kept: {env:#}"); + let skipped = find_event(&env, "skipped", Some(code)); + assert_eq!(skipped["purl"], PURL); + assert!( + skipped.to_string().contains(UUID) && skipped.to_string().contains(UUID2), + "{status}: names the kept and the unserved patch: {skipped}" + ); + assert_eq!(env["summary"]["failed"], 0, "{status}: {env:#}"); + assert_eq!(fx.lock_bytes(), wired_lock, "{status}: wiring untouched"); + assert_eq!( + std::fs::read(fx.state_path()).unwrap(), + state_before, + "{status}: ledger untouched" + ); + assert!(fx.tgz_path().is_file(), "{status}: old artifact kept"); + } +} + +/// The #954 skip keeps only an older vendoring that is still wired: once a +/// relock dropped its `file:.socket/vendor/...` reference the package is +/// patched in neither mode, so the unserved upgrade still fails (the +/// `vendor --check` liveness verdict) instead of claiming the old patch. +#[tokio::test] +async fn superseding_patch_without_a_served_artifact_fails_when_the_old_one_is_unwired() { + use wiremock::matchers::{method, path_regex}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + const UUID2: &str = "0a1b2c3d-4e5f-6a7b-8c9d-0e1f2a3b4c5d"; + let fx = npm_fixture(); + assert_eq!(vendor_run(vendor_args(fx.root())).await, 0); + // A relock: the lock no longer references the vendored artifact. + std::fs::write(fx.lock_path(), &fx.original_lock).unwrap(); + + let mut manifest: Value = + serde_json::from_slice(&std::fs::read(fx.manifest_path()).unwrap()).unwrap(); + manifest["patches"][PURL]["uuid"] = json!(UUID2); + std::fs::write( + fx.manifest_path(), + serde_json::to_vec_pretty(&manifest).unwrap(), + ) + .unwrap(); + + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path_regex(r"/(patch|patches)/package$")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(json!({ "results": { UUID2: { "status": "pending_build" } } })), + ) + .mount(&server) + .await; + let uri = server.uri(); + let (code, env) = vendor_cli( + fx.root(), + &[ + "--api-url", + &uri, + "--vendor-url", + &uri, + "--api-token", + "sktsec_placeholder_value_for_tests_api", + "--org", + "acme", + "--lock-timeout", + "5", + ], + ); + assert_eq!(code, 1, "an unwired older patch is not kept: {env:#}"); + let failed = find_event(&env, "failed", None); + assert_eq!(failed["purl"], PURL); + assert!(failed.to_string().contains("still building"), "{failed}"); + assert!( + events(&env) + .iter() + .all(|e| e["errorCode"] != "vendor_prebuilt_pending"), + "no unserved marker leaks out of a real failure: {env:#}" + ); +} + +/// The #954 skip is only for a package vendored at ANOTHER patch: a first +/// vendor whose artifact is still building has nothing to keep and fails. +#[tokio::test] +async fn unserved_first_vendor_still_fails() { + use wiremock::matchers::{method, path_regex}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + let fx = npm_fixture(); + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path_regex(r"/(patch|patches)/package$")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(json!({ "results": { UUID: { "status": "pending_build" } } })), + ) + .mount(&server) + .await; + let uri = server.uri(); + let (code, env) = vendor_cli( + fx.root(), + &[ + "--api-url", + &uri, + "--vendor-url", + &uri, + "--api-token", + "sktsec_placeholder_value_for_tests_api", + "--org", + "acme", + "--lock-timeout", + "5", + ], + ); + assert_eq!(code, 1, "{env:#}"); + let failed = find_event(&env, "failed", None); + assert!(failed.to_string().contains("still building"), "{failed}"); + assert!( + events(&env) + .iter() + .all(|e| e["errorCode"] != "vendor_prebuilt_pending"), + "no unserved marker leaks out of a real failure: {env:#}" + ); + assert_eq!(fx.lock_bytes(), fx.original_lock); +} + // ───────────────────────────────────────────────────────────────────── // 8d. re-vendor under a new patch uuid — yarn berry // ───────────────────────────────────────────────────────────────────── @@ -3967,7 +4148,8 @@ snapshots: /// refuses the hosted tarball otherwise); the vendor takeover restores /// the last hosted pin to its upstream registry entry and, in the same transaction, deletes /// the `.npmrc` it created — vendored `file:` specs never need it (npm - /// gates them by `allow-file`, default `all`). + /// gates them by `allow-file`, default `all`; a refusing value is the + /// `vendor_npm_allow_file` advisory, #969). #[tokio::test] #[serial] async fn hosted_then_vendor_takeover_removes_the_npmrc_allow_remote_config() { @@ -4009,6 +4191,196 @@ snapshots: ); } + /// [`write_package_lock_project`] plus a parent package `bund` that + /// BUNDLES the same `name@version` (`inBundle: true`): the hosted run + /// rewires only the hoisted entry (`redirect_npm_bundled_instance_skipped`). + fn write_package_lock_project_with_bundled_copy(root: &Path) { + write_package_lock_project(root); + let path = root.join("package-lock.json"); + let mut lock: Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + let packages = lock["packages"].as_object_mut().unwrap(); + packages.insert( + "node_modules/bund".to_string(), + json!({ "version": "1.0.0", "resolved": "file:bund-1.0.0.tgz", "dependencies": { CONV_NAME: CONV_VERSION }, "bundleDependencies": [CONV_NAME] }), + ); + packages.insert( + format!("node_modules/bund/node_modules/{CONV_NAME}"), + json!({ + "version": CONV_VERSION, + "resolved": format!("https://registry.npmjs.org/{CONV_NAME}/-/{CONV_NAME}-{CONV_VERSION}.tgz"), + "integrity": UPSTREAM_SHA512, + "inBundle": true + }), + ); + let mut bytes = serde_json::to_vec_pretty(&lock).unwrap(); + bytes.push(b'\n'); + std::fs::write(&path, bytes).unwrap(); + } + + /// [`write_package_lock_project`] plus a parent package `@bh/sw` that + /// ships its own npm-shrinkwrap.json (`hasShrinkwrap: true`) with a + /// nested copy of the same `name@version`: npm 7–11 install that copy + /// from the parent's shrinkwrap, so the hosted run rewires only the + /// hoisted entry (`redirect_npm_shrinkwrapped_instance_skipped`, #753). + fn write_package_lock_project_with_shrinkwrapped_copy(root: &Path) { + write_package_lock_project(root); + let path = root.join("package-lock.json"); + let mut lock: Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + let packages = lock["packages"].as_object_mut().unwrap(); + packages.insert( + "node_modules/@bh/sw".to_string(), + json!({ + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-sw==", + "hasShrinkwrap": true, + "dependencies": { CONV_NAME: CONV_VERSION } + }), + ); + packages.insert( + format!("node_modules/@bh/sw/node_modules/{CONV_NAME}"), + json!({ + "version": CONV_VERSION, + "resolved": format!("https://registry.npmjs.org/{CONV_NAME}/-/{CONV_NAME}-{CONV_VERSION}.tgz"), + "integrity": UPSTREAM_SHA512 + }), + ); + let mut bytes = serde_json::to_vec_pretty(&lock).unwrap(); + bytes.push(b'\n'); + std::fs::write(&path, bytes).unwrap(); + } + + /// REGRESSION (#828): a hosted pin beside a BUNDLED copy of the same + /// `name@version` is withheld from VEX (the bundled copy stays + /// unpatched), but it is still the hosted run's own wiring of one + /// package version. `rollback` must restore it (it refused it as + /// `patched_ref_unattributable`, remedy "re-run scan --mode hosted", a + /// loop) and delete the `.npmrc` the hosted run created. + #[tokio::test] + #[serial] + async fn hosted_pin_beside_a_bundled_copy_rolls_back_to_upstream() { + hosted_pin_beside_an_unreachable_copy_rolls_back( + write_package_lock_project_with_bundled_copy, + ) + .await; + } + + /// REGRESSION (#828 over #753): the same for a copy beneath a + /// `hasShrinkwrap` package — discovery dropped the hoisted pin outright + /// instead of shadowing it, so rollback refused it as + /// `hosted_wiring_contested`. + #[tokio::test] + #[serial] + async fn hosted_pin_beside_a_shrinkwrapped_copy_rolls_back_to_upstream() { + hosted_pin_beside_an_unreachable_copy_rolls_back( + write_package_lock_project_with_shrinkwrapped_copy, + ) + .await; + } + + async fn hosted_pin_beside_an_unreachable_copy_rolls_back(write: fn(&Path)) { + let server = MockServer::start().await; + mock_hosted_api(&server).await; + let registry = mock_registry(&server).await; + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + write(root); + let pristine_lock = std::fs::read_to_string(root.join("package-lock.json")).unwrap(); + + assert_eq!(scan_run(hosted_args(root, server.uri())).await, 0); + assert!(std::fs::read_to_string(root.join("package-lock.json")) + .unwrap() + .contains(HOSTED_URL)); + assert!(root.join(".npmrc").exists()); + + let env_pairs = online_env(®istry, PATCH_ORIGIN); + let env_pairs: Vec<(&str, &str)> = + env_pairs.iter().map(|(k, v)| (*k, v.as_str())).collect(); + let (code, stdout, stderr) = run_cli( + root, + &[ + "rollback", + "--json", + "--yes", + "--cwd", + root.to_str().unwrap(), + ], + &env_pairs, + ); + assert_eq!(code, 0, "rollback: {stdout}\n{stderr}"); + let lock = std::fs::read_to_string(root.join("package-lock.json")).unwrap(); + let lock: Value = serde_json::from_str(&lock).unwrap(); + let pristine: Value = serde_json::from_str(&pristine_lock).unwrap(); + assert_eq!(lock, pristine, "rollback restores the upstream lock"); + assert!( + !root.join(".npmrc").exists(), + "the hosted run's allow-remote .npmrc goes with the last hosted pin" + ); + } + + /// REGRESSION (#828): the hosted → vendored takeover over a hosted pin + /// beside a bundled copy restores the upstream entry first, exactly as + /// without the bundled copy: `vendor_takeover_reverted_redirect`, the + /// `.npmrc` deleted, the vendor ledger's original the REGISTRY entry + /// (it recorded the grant-tokenized hosted URL), and `vendor --revert` + /// lands on upstream, never back on hosted. + #[tokio::test] + #[serial] + async fn hosted_pin_beside_a_bundled_copy_is_restored_by_the_vendor_takeover() { + hosted_pin_beside_an_unreachable_copy_is_restored_by_the_takeover( + write_package_lock_project_with_bundled_copy, + ) + .await; + } + + /// REGRESSION (#828 over #753): the same takeover beside a copy beneath + /// a `hasShrinkwrap` package. + #[tokio::test] + #[serial] + async fn hosted_pin_beside_a_shrinkwrapped_copy_is_restored_by_the_vendor_takeover() { + hosted_pin_beside_an_unreachable_copy_is_restored_by_the_takeover( + write_package_lock_project_with_shrinkwrapped_copy, + ) + .await; + } + + async fn hosted_pin_beside_an_unreachable_copy_is_restored_by_the_takeover(write: fn(&Path)) { + let server = MockServer::start().await; + mock_hosted_api(&server).await; + let registry = mock_registry(&server).await; + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + write(root); + + assert_eq!(scan_run(hosted_args(root, server.uri())).await, 0); + assert!(std::fs::read_to_string(root.join("package-lock.json")) + .unwrap() + .contains(HOSTED_URL)); + + seed_manifest_and_blob(root); + let (code, env) = vendor_online_cli(root, ®istry, PATCH_ORIGIN, &[]); + assert_eq!(code, 0, "vendor over the hosted lock must succeed: {env:#}"); + find_event(&env, "skipped", Some("vendor_takeover_reverted_redirect")); + assert!( + !root.join(".npmrc").exists(), + "the takeover must remove the .npmrc the hosted run created: {env:#}" + ); + let state = std::fs::read_to_string(root.join(".socket/vendor/state.json")).unwrap(); + assert!( + !state.contains("patch.socket.dev"), + "the ledger original is the registry entry, not the hosted pin: {state}" + ); + + let (code, env) = vendor_online_cli(root, ®istry, PATCH_ORIGIN, &["--revert"]); + assert_eq!(code, 0, "vendor --revert: {env:#}"); + let lock = std::fs::read_to_string(root.join("package-lock.json")).unwrap(); + assert!( + !lock.contains(HOSTED_URL) && !lock.contains(".socket/vendor/"), + "vendor --revert lands on upstream: {lock}" + ); + assert!(!root.join(".npmrc").exists()); + } + /// Hosted → vendored in a git project that ignores `.socket/` (#831): /// hosted mode writes nothing there, so the rule is common. The /// gitignore refusal comes BEFORE the takeover restores the upstream @@ -4308,6 +4680,32 @@ async fn vendor_check_fails_when_lock_no_longer_wires_artifact() { ); } +/// REGRESSION (#969): npm >= 11.14 refuses a vendored `file:` tarball +/// (EALLOWFILE) under a project `allow-file=none`, so every install of the +/// lock fails — but `vendor --check` verified it. It must fail the entry, +/// name the setting and the remedy, and pass again once it is lifted. +#[tokio::test] +async fn vendor_check_fails_when_npm_allow_file_refuses_the_tarball() { + let fx = npm_fixture(); + assert_eq!(vendor_run(vendor_args(fx.root())).await, 0, "vendor"); + std::fs::write(fx.root().join(".npmrc"), "allow-file=none\n").unwrap(); + + let (code, env) = vendor_cli(fx.root(), &["--check"]); + assert_eq!(code, 1, "{env:#}"); + let event = find_event(&env, "failed", Some("vendor_check_failed")); + let reason = event["reason"].as_str().unwrap_or_default(); + assert!( + reason.contains("EALLOWFILE") && reason.contains("`allow-file=none`"), + "{env:#}" + ); + assert!(reason.contains("npm ci --allow-file=all"), "{env:#}"); + + std::fs::write(fx.root().join(".npmrc"), "allow-file=all\n").unwrap(); + let (code, env) = vendor_cli(fx.root(), &["--check"]); + assert_eq!(code, 0, "{env:#}"); + find_event(&env, "verified", Some("vendor_check_ok")); +} + /// REGRESSION (#900, `npm uninstall` trigger): once the dependency leaves /// the lock, `vendor --check` said "no lockfile or config references …, /// so a fresh install gets the unpatched package; re-run `socket-patch diff --git a/crates/socket-patch-cli/tests/npm_e2e_common/manifestless.rs b/crates/socket-patch-cli/tests/npm_e2e_common/manifestless.rs index 6827c5601..e4fd0544a 100644 --- a/crates/socket-patch-cli/tests/npm_e2e_common/manifestless.rs +++ b/crates/socket-patch-cli/tests/npm_e2e_common/manifestless.rs @@ -133,6 +133,12 @@ fn run(case: &ManifestlessCase<'_>, run: VexRun) -> VexOutcome { /// all (the lock was the only hosted state). With the ledgers gone as /// well nothing names the patch either way. /// +/// A shrinkwrap-only checkout (npm <= 11's `npm shrinkwrap`) first runs the +/// `shrinkwrap-only` cell (#899): npm >= 12 never reads npm-shrinkwrap.json, +/// so nothing is attested (`vex_npm_shrinkwrap_only`). The cells above then +/// run after committing the package-lock.json twin npm 12 reads (the +/// documented remedy: a copy of the wired shrinkwrap). +/// /// Leaves the project reverted (locks at registry bytes, no ledgers). pub fn manifestless_vex_matrix(case: &ManifestlessCase<'_>) -> MatrixReport { let mut report = MatrixReport::default(); @@ -158,6 +164,19 @@ pub fn manifestless_vex_matrix(case: &ManifestlessCase<'_>) -> MatrixReport { .filter(|via| !(hosted && *via == VexVia::Vendor)) .collect(); + // npm >= 12 never reads a shrinkwrap without its package-lock.json twin. + let shrinkwrap = "npm-shrinkwrap.json"; + let twin = "package-lock.json"; + let shrinkwrap_only = p.join(shrinkwrap).is_file() && !p.join(twin).exists(); + if shrinkwrap_only { + let out = run(case, VexRun::online(case.api)); + assert_eq!(out.code, Some(1), "[{label}] shrinkwrap-only:\n{out}"); + assert_not_attested(&out.envelope, case.purl, "vex_npm_shrinkwrap_only"); + assert_absent(out.doc.as_ref(), case.purl); + std::fs::copy(p.join(shrinkwrap), p.join(twin)).unwrap(); + report.pass("shrinkwrap-only"); + } + // 0. a LEGACY vendored checkout: vendored mode is manifest-free now, // but a pre-5.0 run left the record in `.socket/manifest.json` // beside the ledger — same uuid, so it attests the same way. @@ -235,6 +254,9 @@ pub fn manifestless_vex_matrix(case: &ManifestlessCase<'_>) -> MatrixReport { } for (lock, bytes) in &case.registry_locks { std::fs::write(p.join(lock), bytes).unwrap(); + if shrinkwrap_only && *lock == shrinkwrap { + std::fs::write(p.join(twin), bytes).unwrap(); + } } for no_verify in [false, true] { let mut r = VexRun::online(case.api); diff --git a/crates/socket-patch-cli/tests/npm_e2e_common/mod.rs b/crates/socket-patch-cli/tests/npm_e2e_common/mod.rs index 03796292f..4fd053079 100644 --- a/crates/socket-patch-cli/tests/npm_e2e_common/mod.rs +++ b/crates/socket-patch-cli/tests/npm_e2e_common/mod.rs @@ -32,7 +32,7 @@ //! | 6 | lockfileVersion 1 | renames the lock | //! | 7, 8 | lockfileVersion 2 (+ v1 mirror)| renames the lock | //! | 9–11 | lockfileVersion 3 | renames the lock | -//! | 12 | lockfileVersion 3 | REMOVED; a committed shrinkwrap gets a package-lock.json twin on first install, and installs read the twin | +//! | 12 | lockfileVersion 3 | REMOVED; npm 12 never reads npm-shrinkwrap.json — installs read package-lock.json, resolved fresh from the registry when only the shrinkwrap is committed (#899) | //! //! npm 12 also defaults `allow-remote=none`, refusing (EALLOWREMOTE) every //! tarball not served from the registry origin — a hosted redirect's @@ -214,9 +214,10 @@ pub enum LockFlavor { /// package-lock.json only (every major's `npm install` default). PackageLock, /// A committed npm-shrinkwrap.json: npm <= 11's `npm shrinkwrap` - /// (renames the lock → shrinkwrap only); npm 12, which removed the - /// command, keeps a package-lock.json twin beside it (its default - /// dual-lock state, and the lock its installs read). + /// (renames the lock → shrinkwrap only, which npm 12 would ignore, so + /// VEX omits it until a package-lock.json twin exists — #899); npm 12, + /// which removed the command and never reads the shrinkwrap, is given + /// the package-lock.json twin it installs from (the dual-lock state). Shrinkwrap, } diff --git a/crates/socket-patch-cli/tests/remove/covgap_commands_remove.rs b/crates/socket-patch-cli/tests/remove/covgap_commands_remove.rs index a005b7b94..eed5d5dd8 100644 --- a/crates/socket-patch-cli/tests/remove/covgap_commands_remove.rs +++ b/crates/socket-patch-cli/tests/remove/covgap_commands_remove.rs @@ -803,6 +803,99 @@ fn remove_hosted_preserve_state_notes_no_preservable_state() { ); } +/// Manifest-backed JSON twin: the note rides the envelope's `warnings[]` +/// as `hosted_state_not_preservable` (the code `rollback +/// --preserve-state` reports), not only human stderr. +#[test] +fn remove_hosted_preserve_state_json_warns_no_preservable_state() { + let tmp = tempfile::tempdir().expect("tempdir"); + std::fs::write(tmp.path().join("package-lock.json"), redirected_lock_text()).unwrap(); + write_manifest_files_empty(tmp.path(), NPM_PURL, NPM_UUID); + let registry = NpmRegistry::start(true); + + let (code, stdout, stderr) = run_remove_online( + tmp.path(), + &[NPM_PURL, "--json", "--yes", "--preserve-state"], + ®istry, + ); + assert_eq!(code, 0, "stdout=\n{stdout}\nstderr=\n{stderr}"); + let v = parse_envelope(&stdout); + let warnings = v["warnings"].as_array().expect("warnings array"); + assert!( + warnings + .iter() + .any(|w| w["code"] == "hosted_state_not_preservable"), + "expected a hosted_state_not_preservable warning; envelope={v}" + ); +} + +/// Hosted-only twin (#433): with no manifest — the default v5 shape after +/// a bare `scan` — `remove --preserve-state` still restores the pin, and +/// must say so: the `Note: …` line on stderr in human mode, the +/// `hosted_state_not_preservable` warning in `--json`. +#[test] +fn remove_hosted_only_preserve_state_notes_no_preservable_state() { + let tmp = tempfile::tempdir().expect("tempdir"); + let lock_path = tmp.path().join("package-lock.json"); + std::fs::write(&lock_path, redirected_lock_text()).unwrap(); + let registry = NpmRegistry::start(true); + + let (code, stdout, stderr) = run_remove_online( + tmp.path(), + &[NPM_PURL, "--yes", "--preserve-state"], + ®istry, + ); + assert_eq!(code, 0, "stdout=\n{stdout}\nstderr=\n{stderr}"); + assert!( + stderr.contains("Note: hosted wiring has no preservable local state"), + "the preserve-state hosted note must reach stderr; got:\n{stderr}" + ); + assert_eq!( + std::fs::read_to_string(&lock_path).unwrap(), + expected_reverted_lock_text(), + "the lock must hold exactly the upstream registry entry" + ); + + // JSON: re-pin, then the warning must ride the envelope. + std::fs::write(&lock_path, redirected_lock_text()).unwrap(); + let (code, stdout, stderr) = run_remove_online( + tmp.path(), + &[NPM_PURL, "--json", "--yes", "--preserve-state"], + ®istry, + ); + assert_eq!(code, 0, "stdout=\n{stdout}\nstderr=\n{stderr}"); + assert!( + !stderr.contains("no preservable local state"), + "--json keeps the note off stderr; got:\n{stderr}" + ); + let v = parse_envelope(&stdout); + let warnings = v["warnings"].as_array().expect("warnings array"); + assert!( + warnings + .iter() + .any(|w| w["code"] == "hosted_state_not_preservable"), + "expected a hosted_state_not_preservable warning; envelope={v}" + ); + assert_eq!( + std::fs::read_to_string(&lock_path).unwrap(), + expected_reverted_lock_text(), + "the JSON run restores the pin too" + ); + + // Without --preserve-state there is nothing to note. + std::fs::write(&lock_path, redirected_lock_text()).unwrap(); + let (code, stdout, stderr) = + run_remove_online(tmp.path(), &[NPM_PURL, "--json", "--yes"], ®istry); + assert_eq!(code, 0, "stdout=\n{stdout}\nstderr=\n{stderr}"); + let v = parse_envelope(&stdout); + assert!( + !v["warnings"].as_array().is_some_and(|ws| ws + .iter() + .any(|w| w["code"] == "hosted_state_not_preservable")), + "a plain remove must not warn about preserve-state; envelope={v}" + ); +} + // --------------------------------------------------------------------------- // 7. Corrupt pre-v5 hosted ledger: ignored // --------------------------------------------------------------------------- diff --git a/crates/socket-patch-cli/tests/scan_rollout_e2e.rs b/crates/socket-patch-cli/tests/scan_rollout_e2e.rs index 638ce21c5..58a042c85 100644 --- a/crates/socket-patch-cli/tests/scan_rollout_e2e.rs +++ b/crates/socket-patch-cli/tests/scan_rollout_e2e.rs @@ -1061,3 +1061,116 @@ async fn a_failed_detail_lookup_for_an_applied_package_does_not_freeze_new_rows( "the pin stays" ); } + +/// #954: a vendored package gains a superseding patch whose prebuilt +/// artifact the service has not built (`pending_build`) or cannot serve +/// (`not_found`). `scan --mode vendored` keeps the vendored patch and +/// reports the upgrade as a skip, exit 0, the way hosted mode keeps its pin: +/// before the fix every re-run failed (`partial_failure`, exit 1) until the +/// server built the artifact. +#[tokio::test] +async fn vendored_upgrade_without_a_served_artifact_keeps_the_vendored_patch() { + let newer = "0000000e-2222-4222-8222-00000000000e"; + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + for (status, code) in [ + ("pending_build", "vendor_prebuilt_pending"), + ("not_found", "vendor_prebuilt_unavailable"), + ] { + let first = MockServer::start().await; + mount_api(&first, |_| Grant::Granted).await; + for (name, severities) in PACKAGES { + let view = json!({ + "uuid": uuid(name), "purl": purl(name), "publishedAt": "2026-01-01T00:00:00Z", + "files": { "package/index.js": { "beforeHash": git_sha256(&before(name)), "afterHash": git_sha256(&after(name)), "blobContent": b64(&after(name)) } }, + "vulnerabilities": vulns(name, severities), "description": name, "license": "MIT", "tier": "free" + }); + prebuilt_common::mount_view(&first, &view, None).await; + } + let tmp = tempfile::tempdir().unwrap(); + write_project(tmp.path(), &names9); + run_json(tmp.path(), &first, &["--mode", "vendored"]); + let state_path = tmp.path().join(".socket/vendor/state.json"); + let state_before = std::fs::read(&state_path).unwrap(); + let lock_before = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + + // A fresh API: roll-e's newer patch is offered and viewable, but + // the service has no artifact for it. + let second = MockServer::start().await; + mount_api(&second, |_| Grant::Granted).await; + let sevs = PACKAGES.iter().find(|(n, _)| *n == "roll-e").unwrap().1; + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*roll-e(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "patches": [ + { "uuid": newer, "purl": purl("roll-e"), "publishedAt": "2026-06-01T00:00:00Z", + "description": "newer", "license": "MIT", "tier": "free", + "vulnerabilities": vulns("roll-e", sevs) }, + { "uuid": uuid("roll-e"), "purl": purl("roll-e"), "publishedAt": "2026-01-01T00:00:00Z", + "description": "roll-e", "license": "MIT", "tier": "free", + "vulnerabilities": vulns("roll-e", sevs) } + ], + "canAccessPaidPatches": false, + }))) + .with_priority(1) + .mount(&second) + .await; + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/view/{newer}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "uuid": newer, "purl": purl("roll-e"), + "publishedAt": "2026-06-01T00:00:00Z", + "files": { "package/index.js": { + "beforeHash": git_sha256(&before("roll-e")), + "afterHash": git_sha256(&after("roll-e")), + "blobContent": b64(&after("roll-e")), + }}, + "vulnerabilities": vulns("roll-e", sevs), + "description": "newer", "license": "MIT", "tier": "free", + }))) + .mount(&second) + .await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(json!({ "results": { newer: { "status": status } } })), + ) + .with_priority(1) + .mount(&second) + .await; + + for _ in 0..2 { + let (exit, stdout, stderr) = + run(tmp.path(), &second, &["--json", "--mode", "vendored"]); + assert_eq!(exit, 0, "{status}: stdout={stdout}\nstderr={stderr}"); + let v: Value = serde_json::from_str(stdout.trim()).unwrap(); + assert_eq!(v["status"], "success", "{status}: {v:#}"); + assert_eq!(v["vendor"]["summary"]["failed"], 0, "{status}: {v:#}"); + let vendor = v["vendor"].to_string(); + assert!( + vendor.contains(code) && vendor.contains(newer), + "{status}: the upgrade is a `{code}` skip: {v:#}" + ); + assert_eq!( + std::fs::read(&state_path).unwrap(), + state_before, + "{status}" + ); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + lock_before, + "{status}: the vendored wiring stays" + ); + } + + // Human mode: no "Failed to vendor", exit 0. + let (exit, stdout, stderr) = run(tmp.path(), &second, &["--mode", "vendored"]); + assert_eq!(exit, 0, "{status}: stdout={stdout}\nstderr={stderr}"); + assert!( + !stderr.contains("Failed to vendor") && !stdout.contains("failed"), + "{status}: stdout={stdout}\nstderr={stderr}" + ); + } +} diff --git a/crates/socket-patch-cli/tests/vendor/redirect_npm_allow_remote.rs b/crates/socket-patch-cli/tests/vendor/redirect_npm_allow_remote.rs index 6314f1687..96ce71ac7 100644 --- a/crates/socket-patch-cli/tests/vendor/redirect_npm_allow_remote.rs +++ b/crates/socket-patch-cli/tests/vendor/redirect_npm_allow_remote.rs @@ -44,6 +44,7 @@ const PATCHED_SHA512: &str = "sha512-PATCHEDpatchedPATCHEDpatched0123456789=="; const UPSTREAM_SHA512: &str = "sha512-UPSTREAMupstream=="; const CODE: &str = "redirect_npm_allow_remote"; const LEFT: &str = "npm_allow_remote_left"; +const REPLACE_HOST: &str = "redirect_npm_replace_registry_host"; fn hosted_url() -> String { format!( @@ -188,6 +189,8 @@ fn npm_isolation(root: &Path) -> Vec<(String, String)> { ("PREFIX".into(), absent(".absent-prefix")), ("NPM_CONFIG_ALLOW_REMOTE".into(), String::new()), ("npm_config_allow_remote".into(), String::new()), + ("NPM_CONFIG_REPLACE_REGISTRY_HOST".into(), String::new()), + ("npm_config_replace_registry_host".into(), String::new()), ] } @@ -789,6 +792,119 @@ async fn outer_npm_config_layers_are_respected() { ); } +fn replace_host_warning(doc: &Value) -> Option<&str> { + doc["redirect"]["warnings"] + .as_array() + .into_iter() + .flatten() + .find(|w| w["code"] == REPLACE_HOST) + .and_then(|w| w["detail"].as_str()) +} + +/// #812: npm >= 8's `replace-registry-host=always` (or the pinned host +/// itself) makes npm rewrite every hosted pin to the configured registry, +/// so each later `npm ci` / `npm install` fails E404. The hosted run +/// detected nothing and reported success. It now warns +/// `redirect_npm_replace_registry_host` (JSON + human), naming the layer +/// that sets it — project file, user config or env var — on the first run +/// and on the "already hosted" re-run; a value that does not match the +/// pinned host stays quiet. +#[tokio::test] +async fn replace_registry_host_rewriting_the_pin_is_warned() { + let server = MockServer::start().await; + mock_api(&server).await; + + // Project `.npmrc`: first run and the idempotent re-run both warn. + let tmp = tempfile::tempdir().unwrap(); + write_npm_project(tmp.path(), "package-lock.json"); + std::fs::write(tmp.path().join(".npmrc"), "replace-registry-host=always\n").unwrap(); + for run in 0..2 { + let (code, doc, _) = scan_hosted(tmp.path(), &server.uri(), &["--json"]); + assert_eq!(code, 0, "{doc:#}"); + let detail = replace_host_warning(&doc) + .unwrap_or_else(|| panic!("run {run}: no {REPLACE_HOST}: {doc:#}")); + assert!( + detail.contains("patch.test") + && detail.contains("the project .npmrc sets `replace-registry-host=always`") + && detail.contains("E404") + && detail.contains("replace-registry-host=npmjs") + && detail.contains("--mode vendored"), + "{detail}" + ); + } + let (code, _, stderr) = scan_hosted(tmp.path(), &server.uri(), &[]); + assert_eq!(code, 0, "{stderr}"); + assert!( + stderr.contains("Warning: ") && stderr.contains("`replace-registry-host=always`"), + "{stderr}" + ); + + // The pinned hostname itself rewrites it too; another host does not. + for (value, warns) in [ + ("patch.test", true), + ("registry.example", false), + ("never", false), + ] { + let tmp = tempfile::tempdir().unwrap(); + write_npm_project(tmp.path(), "package-lock.json"); + std::fs::write( + tmp.path().join(".npmrc"), + format!("replace-registry-host={value}\n"), + ) + .unwrap(); + let (code, doc, _) = scan_hosted(tmp.path(), &server.uri(), &["--json"]); + assert_eq!(code, 0, "{doc:#}"); + assert_eq!( + replace_host_warning(&doc).is_some(), + warns, + "{value}: {doc:#}" + ); + assert!(allow_remote_warning(&doc).is_some(), "{doc:#}"); + } + + // User config (relocated the way npm allows). + let tmp = tempfile::tempdir().unwrap(); + let cfg = tempfile::tempdir().unwrap(); + write_npm_project(tmp.path(), "package-lock.json"); + let user = cfg.path().join("user.npmrc"); + std::fs::write(&user, "replace-registry-host=always\n").unwrap(); + let user_s = user.to_str().unwrap(); + let (code, doc, _) = scan_hosted_env( + tmp.path(), + &server.uri(), + &["--json"], + &[ + ("NPM_CONFIG_USERCONFIG", user_s), + ("npm_config_userconfig", user_s), + ], + ); + assert_eq!(code, 0, "{doc:#}"); + let detail = replace_host_warning(&doc).unwrap_or_else(|| panic!("no {REPLACE_HOST}: {doc:#}")); + assert!( + detail.contains("the user npm config") && detail.contains(user_s), + "{detail}" + ); + + // The env var beats a project `npmjs`. + let tmp = tempfile::tempdir().unwrap(); + write_npm_project(tmp.path(), "package-lock.json"); + std::fs::write(tmp.path().join(".npmrc"), "replace-registry-host=npmjs\n").unwrap(); + let (code, doc, _) = scan_hosted_env( + tmp.path(), + &server.uri(), + &["--json"], + &[("npm_config_replace_registry_host", "always")], + ); + assert_eq!(code, 0, "{doc:#}"); + let detail = replace_host_warning(&doc).unwrap_or_else(|| panic!("no {REPLACE_HOST}: {doc:#}")); + assert!( + detail + .to_ascii_lowercase() + .contains("npm_config_replace_registry_host sets `replace-registry-host=always`"), + "{detail}" + ); +} + /// Review finding: `remove` dropped the hosted leg's warnings. A /// redirect-created `.npmrc` the user has since added to is no longer the /// scaffold, so the restore keeps it byte-for-byte and warns that the diff --git a/crates/socket-patch-cli/tests/vendor_eject_fresh_checkout.rs b/crates/socket-patch-cli/tests/vendor_eject_fresh_checkout.rs index 2e029af81..00c32cb2b 100644 --- a/crates/socket-patch-cli/tests/vendor_eject_fresh_checkout.rs +++ b/crates/socket-patch-cli/tests/vendor_eject_fresh_checkout.rs @@ -85,22 +85,39 @@ fn run_json_with( args: &[&str], extra: &[(&str, String)], ) -> (i32, Value) { - let cargo_home = root.join("../cargo-home"); - std::fs::create_dir_all(&cargo_home).unwrap(); - let mut cmd = Command::new(env!("CARGO_BIN_EXE_socket-patch")); - cmd.args(args) + let mut cmd = command(root, server, extra); + let out = cmd + .args(args) .arg("--json") .arg("--cwd") .arg(root) - .current_dir(root); + .output() + .expect("spawn socket-patch"); + let stdout = String::from_utf8_lossy(&out.stdout); + let env = serde_json::from_str(&stdout).unwrap_or_else(|e| { + panic!( + "--json must emit an envelope: {e}\nstdout:\n{stdout}\nstderr:\n{}", + String::from_utf8_lossy(&out.stderr) + ) + }); + (out.status.code().unwrap_or(-1), env) +} + +/// The binary in `root` (the caller adds the args, then `--cwd`), with +/// every ambient `SOCKET_*` var scrubbed, an empty `CARGO_HOME`, and the +/// API / registries / patch origin on `server`. +fn command(root: &Path, server: &MockServer, extra: &[(&str, String)]) -> Command { + let cargo_home = root.join("../cargo-home"); + std::fs::create_dir_all(&cargo_home).unwrap(); + let mut cmd = Command::new(env!("CARGO_BIN_EXE_socket-patch")); + cmd.current_dir(root); for (key, _) in std::env::vars() { if key.starts_with("SOCKET_") { cmd.env_remove(key); } } let uri = server.uri(); - let out = cmd - .env("SOCKET_TELEMETRY_DISABLED", "1") + cmd.env("SOCKET_TELEMETRY_DISABLED", "1") .env("SOCKET_API_URL", &uri) .env("SOCKET_API_TOKEN", "fake-token") .env("SOCKET_ORG_SLUG", ORG) @@ -110,17 +127,8 @@ fn run_json_with( .env("SOCKET_PATCH_SERVER_URL", &uri) .env("SOCKET_VENDOR_SOURCE", "service") .env("CARGO_HOME", &cargo_home) - .envs(extra.iter().map(|(k, v)| (*k, v.as_str()))) - .output() - .expect("spawn socket-patch"); - let stdout = String::from_utf8_lossy(&out.stdout); - let env = serde_json::from_str(&stdout).unwrap_or_else(|e| { - panic!( - "--json must emit an envelope: {e}\nstdout:\n{stdout}\nstderr:\n{}", - String::from_utf8_lossy(&out.stderr) - ) - }); - (out.status.code().unwrap_or(-1), env) + .envs(extra.iter().map(|(k, v)| (*k, v.as_str()))); + cmd } fn applied(env: &Value, purl: &str) -> bool { @@ -402,3 +410,198 @@ async fn pypi_eject_needs_no_virtualenv() { "the requirement is wired to the vendored wheel: {reqs}" ); } + +/// #1005: an eject where one package vendors and another fails is rolled +/// back whole, and the report says so. The human run prints the +/// `eject_rolled_back` warning, never "Vendored 1 package" or the advice to +/// commit `.socket/vendor/`; the JSON envelope does not count the +/// rolled-back package as applied. +#[tokio::test] +async fn rolled_back_eject_reports_nothing_vendored() { + const LEFT_PAD_UUID: &str = "9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f"; + const MS_UUID: &str = "7a1e3c5b-2d4f-4a6b-9c8d-0e1f2a3b4c5d"; + const LEFT_PAD: &str = "pkg:npm/left-pad@1.3.0"; + const MS: &str = "pkg:npm/ms@2.1.3"; + const ORIG: &[u8] = b"module.exports = () => 'orig';\n"; + const PATCHED: &[u8] = b"module.exports = () => 'patched';\n"; + let server = MockServer::start().await; + let uri = server.uri(); + + // left-pad vendors: packument, pristine tarball and service artifact. + let tarball = tgz( + "package", + &[ + ("package.json", br#"{"name":"left-pad","version":"1.3.0"}"#), + ("index.js", ORIG), + ], + ); + let integrity = format!( + "sha512-{}", + base64::engine::general_purpose::STANDARD.encode(Sha512::digest(&tarball)) + ); + Mock::given(method("GET")) + .and(path("/left-pad/1.3.0")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "name": "left-pad", + "version": "1.3.0", + "dist": { "tarball": format!("{uri}/left-pad/-/left-pad-1.3.0.tgz"), "integrity": integrity } + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/left-pad/-/left-pad-1.3.0.tgz")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(tarball)) + .mount(&server) + .await; + mock_view( + &server, + LEFT_PAD_UUID, + LEFT_PAD, + "package/index.js", + ORIG, + PATCHED, + ) + .await; + + // ms restores upstream (its packument resolves) but cannot vendor: its + // record has no service artifact and its pristine tarball is a 500. + Mock::given(method("GET")) + .and(path("/ms/2.1.3")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "name": "ms", + "version": "2.1.3", + "dist": { + "tarball": format!("{uri}/ms/-/ms-2.1.3.tgz"), + "integrity": "sha512-msUPSTREAMmsUPSTREAM==" + } + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/ms/-/ms-2.1.3.tgz")) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/view/{MS_UUID}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "uuid": MS_UUID, + "purl": MS, + "publishedAt": "2026-01-01T00:00:00Z", + "files": { + "package/index.js": { + "beforeHash": compute_git_sha256_from_bytes(ORIG), + "afterHash": compute_git_sha256_from_bytes(PATCHED) + } + }, + "vulnerabilities": {}, + "description": "eject fixture", + "license": "MIT", + "tier": "free" + }))) + .mount(&server) + .await; + + let hosted = |name: &str, uuid: &str, version: &str| { + format!( + "{uri}/patch/npm/{name}/{version}/55555555-5555-4555-8555-555555555555/{uuid}/{name}-{version}.tgz" + ) + }; + let lock = serde_json::to_string_pretty(&json!({ + "name": "fixture", "version": "1.0.0", "lockfileVersion": 3, "requires": true, + "packages": { + "": { + "name": "fixture", "version": "1.0.0", + "dependencies": { "left-pad": "1.3.0", "ms": "2.1.3" } + }, + "node_modules/left-pad": { + "version": "1.3.0", "resolved": hosted("left-pad", LEFT_PAD_UUID, "1.3.0"), + "integrity": "sha512-HOSTEDpatchedHOSTEDpatched==", "license": "WTFPL" + }, + "node_modules/ms": { + "version": "2.1.3", "resolved": hosted("ms", MS_UUID, "2.1.3"), + "integrity": "sha512-HOSTEDmsHOSTEDms==", "license": "MIT" + } + } + })) + .unwrap() + + "\n"; + let tmp = tempfile::tempdir().unwrap(); + let fresh = |name: &str| { + let root = tmp.path().join(name); + std::fs::create_dir_all(&root).unwrap(); + std::fs::write( + root.join("package.json"), + br#"{"name":"fixture","version":"1.0.0","private":true,"dependencies":{"left-pad":"1.3.0","ms":"2.1.3"}}"#, + ) + .unwrap(); + std::fs::write(root.join("package-lock.json"), &lock).unwrap(); + root + }; + let still_hosted = |root: &Path| { + assert_eq!( + std::fs::read_to_string(root.join("package-lock.json")).unwrap(), + lock, + "the eject is rolled back" + ); + assert!( + std::fs::read_dir(root.join(".socket/vendor")) + .map(|mut d| d.next().is_none()) + .unwrap_or(true), + "no vendored residue" + ); + }; + + let root = fresh("json"); + let (code, env) = run_json(&root, &server, &["vendor"]); + assert_eq!(code, 1, "{env:#}"); + still_hosted(&root); + assert!( + env["warnings"] + .as_array() + .is_some_and(|w| w.iter().any(|w| w["code"] == "eject_rolled_back")), + "{env:#}" + ); + assert!( + !applied(&env, LEFT_PAD), + "rolled back, not applied: {env:#}" + ); + assert_eq!(env["summary"]["applied"], 0, "{env:#}"); + assert!( + env["events"] + .as_array() + .unwrap() + .iter() + .any(|e| e["purl"] == LEFT_PAD + && e["action"] == "skipped" + && e["errorCode"] == "eject_rolled_back"), + "{env:#}" + ); + assert!( + env["events"] + .as_array() + .unwrap() + .iter() + .any(|e| e["purl"] == MS && e["action"] == "failed"), + "{env:#}" + ); + + let root = fresh("human"); + let out = command(&root, &server, &[]) + .args(["vendor", "--cwd"]) + .arg(&root) + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert_eq!(out.status.code(), Some(1), "{stdout}\n{stderr}"); + still_hosted(&root); + assert!( + stderr.contains("the project is still hosted, exactly as before"), + "the rollback is announced: {stdout}\n{stderr}" + ); + assert!( + !stdout.contains("Vendored") && !stdout.contains("Next steps"), + "nothing was vendored: {stdout}\n{stderr}" + ); +} diff --git a/crates/socket-patch-core/src/crawlers/npm_crawler.rs b/crates/socket-patch-core/src/crawlers/npm_crawler.rs index 7c9cffad2..a37d75250 100644 --- a/crates/socket-patch-core/src/crawlers/npm_crawler.rs +++ b/crates/socket-patch-core/src/crawlers/npm_crawler.rs @@ -1667,7 +1667,8 @@ impl NpmCrawler { // the alias, its real dir `node_modules/` (#852), so those // entries are searched for alias copies like an importer tree. if !store_entry || is_npm_linked_store_entry(&nm_path) { - matched.extend(Self::alias_copies(&nm_path, &listing, pending)); + let aliases = Self::alias_copies(&nm_path, &listing, pending, &matched); + matched.extend(aliases); } let gvs_member = !store_entry && may_be_gvs_workspace_member(&listing); let mut nested = Self::collect_nested_node_modules(&nm_path, listing); @@ -1698,12 +1699,17 @@ impl NpmCrawler { /// Only real package dirs count (links are dependency edges into a /// store or into first-party source, never copies of their own), and /// a dir whose name is its package's own name is the direct probe's - /// job, so it is skipped here: that keeps one physical dir from being - /// recorded twice through a case-insensitive lookup. + /// job, so it is skipped here. A dir whose name differs only by case + /// (`node_modules/Left-Pad` holding `left-pad`) is an alias on a + /// case-sensitive file system, where the probe misses it; it is skipped + /// only when it IS the dir the probe already returned (`probed`, a + /// case-insensitive file system folding the probe's path onto it), so + /// one physical dir is never recorded twice (#856). fn alias_copies( nm_path: &Path, listing: &Listing, pending: &[Target], + probed: &[(usize, PathBuf)], ) -> Vec<(usize, PathBuf)> { let mut by_identity: HashMap<(&str, &str), Vec> = HashMap::new(); for (index, target) in pending.iter().enumerate() { @@ -1742,11 +1748,21 @@ impl NpmCrawler { else { continue; }; - if name.eq_ignore_ascii_case(&dir_key) { + if name == dir_key { continue; } + let case_only = name.eq_ignore_ascii_case(&dir_key); if let Some(indices) = by_identity.get(&(name.as_str(), version.as_str())) { - found.extend(indices.iter().map(|&index| (index, pkg_path.clone()))); + for &index in indices { + let already_probed = case_only + && probed.iter().any(|(probed_index, probed_path)| { + *probed_index == index + && same_file::is_same_file(probed_path, &pkg_path).unwrap_or(false) + }); + if !already_probed { + found.push((index, pkg_path.clone())); + } + } } } found @@ -3716,6 +3732,48 @@ mod tests { assert_eq!(copy.name, "pkg"); } + /// #856: a key differing from the package's name only by case + /// (`node_modules/Left-Pad` holding `left-pad@1.3.0`, a legacy-valid + /// npm name) is an alias copy. On a case-sensitive file system the + /// direct probe of `node_modules/left-pad` misses it, so the alias pass + /// must return it; where the file system folds case the probe already + /// returned that physical dir, and it is reported exactly once. + #[tokio::test] + async fn find_by_purls_resolves_a_case_only_alias_once() { + let tmp = tempfile::tempdir().unwrap(); + let nm = tmp.path().join("node_modules"); + write_pkg(&nm.join("Left-Pad"), "left-pad", "1.3.0"); + write_pkg(&nm.join("mm"), "minimist", "1.2.2"); + let case_folding = nm.join("left-pad").exists(); + + let pad = "pkg:npm/left-pad@1.3.0".to_string(); + let mm = "pkg:npm/minimist@1.2.2".to_string(); + let found = NpmCrawler::new() + .find_by_purls(&nm, &[pad.clone(), mm.clone()]) + .await + .unwrap(); + let pad_copies = copy_paths(&found, &pad); + assert_eq!(pad_copies.len(), 1, "{pad_copies:?}"); + if !case_folding { + assert_eq!(pad_copies, vec![nm.join("Left-Pad")]); + } + assert_eq!(copy_paths(&found, &mm), vec![nm.join("mm")]); + + // Beside a plain copy (case-sensitive file systems only: a folding + // one cannot hold both names), both dirs are copies. + if !case_folding { + write_pkg(&nm.join("left-pad"), "left-pad", "1.3.0"); + let found = NpmCrawler::new() + .find_by_purls(&nm, std::slice::from_ref(&pad)) + .await + .unwrap(); + assert_eq!( + copy_paths(&found, &pad), + vec![nm.join("left-pad"), nm.join("Left-Pad")] + ); + } + } + /// A link is a dependency edge (into a store, a workspace member or an /// `npm link` target), never an alias copy of its own; pnpm's isolated /// alias link resolves through the store entry instead. diff --git a/crates/socket-patch-core/src/crawlers/npm_crawler/oracle.rs b/crates/socket-patch-core/src/crawlers/npm_crawler/oracle.rs index 37bf7ad02..2d26df0b9 100644 --- a/crates/socket-patch-core/src/crawlers/npm_crawler/oracle.rs +++ b/crates/socket-patch-core/src/crawlers/npm_crawler/oracle.rs @@ -230,11 +230,19 @@ impl LegacyNpmCrawler { // Alias installs (`"lp": "npm:left-pad@1.3.0"`): a real // importer-tree package dir whose own package.json names a // pending target under a different dir name is a copy too. + // A dir named after its package only up to case is an alias + // too (#856), unless it IS the dir this visit's probe just + // recorded (a case-folding file system resolving the probe's + // spelling onto it), so one physical dir is recorded once. if !store_entry { - for (index, pkg_path) in Self::alias_copies(&nm_path, &pending).await { + for (index, pkg_path, case_only) in Self::alias_copies(&nm_path, &pending).await { let target = &pending[index]; let copies = result.entry(target.purl.clone()).or_default(); - if !copies.iter().any(|c| c.path == pkg_path) { + let probe_path = nm_path.join(&target.dir_key); + let already_probed = case_only + && copies.iter().any(|c| c.path == probe_path) + && same_file::is_same_file(&probe_path, &pkg_path).unwrap_or(false); + if !already_probed && !copies.iter().any(|c| c.path == pkg_path) { copies.push(CrawledPackage { name: target.name.clone(), version: target.version.clone(), @@ -268,8 +276,9 @@ impl LegacyNpmCrawler { /// The alias installs directly below `nm_path` that are copies of a /// pending target, as `(target index, path)` in listing order: real /// package dirs (scoped ones one level down) whose package.json - /// `name@version` is a target's while the dir is named otherwise. - async fn alias_copies(nm_path: &Path, pending: &[Target]) -> Vec<(usize, PathBuf)> { + /// `name@version` is a target's while the dir is named otherwise, each + /// flagged when the names differ only by case. + async fn alias_copies(nm_path: &Path, pending: &[Target]) -> Vec<(usize, PathBuf, bool)> { async fn package_dirs(dir: &Path) -> Vec<(String, PathBuf)> { let mut out = Vec::new(); for entry in crate::utils::fs::list_dir_entries(dir).await { @@ -303,12 +312,13 @@ impl LegacyNpmCrawler { else { continue; }; - if name.eq_ignore_ascii_case(&dir_key) { + if name == dir_key { continue; } + let case_only = name.eq_ignore_ascii_case(&dir_key); for (index, target) in pending.iter().enumerate() { if target.dir_key == name && target.version == version { - found.push((index, pkg_path.clone())); + found.push((index, pkg_path.clone(), case_only)); } } } @@ -1811,7 +1821,9 @@ mod tests { ); // A dir whose spelling differs from its package's name only by case // (resolves under the lowercase name on case-insensitive volumes), - // and a scope spelled likewise. + // and a scope spelled likewise. On a case-sensitive file system + // both are alias copies of `casedir` / `@cs/x` (#856); where the + // file system folds case each is reported once, via the probe. write(&nm.join("CaseDir"), "casedir", "1.0.0"); write(&nm.join("@Cs").join("x"), "@cs/x", "1.0.0"); // Broken / BOM'd package.json. diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs index cc2e07164..d64d8312d 100644 --- a/crates/socket-patch-core/src/hosted/engine.rs +++ b/crates/socket-patch-core/src/hosted/engine.rs @@ -28,8 +28,8 @@ use crate::constants::npm_family::{ RUSH_COMMON_LOCK_REL, RUSH_SUBSPACES_DIR, VLT_HIDDEN_LOCK_REL, VLT_LOCK, }; use crate::patch::redirect::npmrc::{ - plan_npmrc_allow_remote_with, NpmrcPlan, OuterAllowRemote, NPMRC_ALLOW_REMOTE_EDIT_KIND, - NPMRC_REL, + effective_replace_registry_host, plan_npmrc_allow_remote_with, replace_registry_host_rewrites, + NpmrcPlan, OuterAllowRemote, NPMRC_ALLOW_REMOTE_EDIT_KIND, NPMRC_REL, }; use crate::patch::redirect::presence::groups_present; use crate::patch::redirect::yarnrc::OuterYarnMirror; @@ -45,13 +45,13 @@ use super::guidance::{ npm_allow_remote_already_detail, npm_allow_remote_configured_detail, npm_allow_remote_env_set_detail, npm_allow_remote_manual_detail, npm_allow_remote_outer_set_detail, npm_allow_remote_unreadable_detail, - npm_allow_remote_user_set_detail, npm_lock_url_needles, plan_workspace_trust, pnpm_heal_root, - pnpm_is_shrinkwrap_lock, pnpm_lock_may_need_store_flag, pnpm_lock_version_major, - pnpm_trust_configured_detail, pnpm_trust_legacy_detail, pnpm_trust_manual_guidance, - pnpm_trust_policy_preamble, pnpm_trust_workspace_unreadable_detail, + npm_allow_remote_user_set_detail, npm_lock_url_needles, npm_replace_registry_host_detail, + plan_workspace_trust, pnpm_heal_root, pnpm_is_shrinkwrap_lock, pnpm_lock_may_need_store_flag, + pnpm_lock_version_major, pnpm_trust_configured_detail, pnpm_trust_legacy_detail, + pnpm_trust_manual_guidance, pnpm_trust_policy_preamble, pnpm_trust_workspace_unreadable_detail, pnpm_trust_workspace_unsupported_detail, read_npmrc_for_allow_remote, read_workspace_for_trust, - url_host, TrustPlan, NPM_LOCKS, PNPM_TRUST_TRADEOFF_AND_CAUTION, PNPM_WORKSPACE_REL, - REDIRECT_PNPM_WORKSPACE_TRUST_EDIT_KIND, + url_host, TrustPlan, NPM_LOCKS, NPM_REPLACE_REGISTRY_HOST_CODE, + PNPM_TRUST_TRADEOFF_AND_CAUTION, PNPM_WORKSPACE_REL, REDIRECT_PNPM_WORKSPACE_TRUST_EDIT_KIND, }; use super::vlt::bun_lockb_present; @@ -2004,7 +2004,9 @@ fn governing_workspace(view: &ProjectView<'_>) -> Option { /// opts out entirely; every variant still WARNS /// (`redirect_npm_allow_remote`) with the whole-tree tradeoff. Vendored /// mode is unaffected: its `file:.socket/vendor/…` specs are npm `file` -/// specs, gated by `allow-file` (default `all`), not `allow-remote`. +/// specs, gated by `allow-file` (default `all`), not `allow-remote` — an +/// explicit refusing `allow-file` is the vendored flow's own advisory +/// (`vendor_npm_allow_file`, #969). fn npm_allow_remote( view: &ProjectView<'_>, files: &BTreeMap, @@ -2046,34 +2048,52 @@ fn npm_allow_remote( original: None, new: Some(serde_json::json!("all")), }; - let detail = match read_npmrc(view) { + let npmrc = read_npmrc(view); + let outer = (options.npm_outer)(); + let detail = match &npmrc { // Opt-out still reports an explicit / already-set value truthfully; // only the WRITE is suppressed. - Ok(existing) => { - match plan_npmrc_allow_remote_with(existing.as_deref(), &(options.npm_outer)()) { - NpmrcPlan::AlreadyAll => npm_allow_remote_already_detail(&npm_hosts), - NpmrcPlan::UserSet(value) => npm_allow_remote_user_set_detail(&npm_hosts, &value), - NpmrcPlan::EnvSet { var, value } => { - npm_allow_remote_env_set_detail(&npm_hosts, &var, &value) - } - NpmrcPlan::OuterSet { layer, path, value } => { - npm_allow_remote_outer_set_detail(&npm_hosts, layer, &path, &value) - } - NpmrcPlan::Unsupported(why) => npm_allow_remote_unreadable_detail(&npm_hosts, &why), - _ if !options.npm_allow_remote_config => npm_allow_remote_manual_detail(&npm_hosts), - NpmrcPlan::Create(text) => { - npmrc_config_write = Some((text, edit("created"))); - npm_allow_remote_configured_detail(&npm_hosts, true, options.dry_run) - } - NpmrcPlan::Append(text) => { - npmrc_config_write = Some((text, edit("added"))); - npm_allow_remote_configured_detail(&npm_hosts, false, options.dry_run) - } + Ok(existing) => match plan_npmrc_allow_remote_with(existing.as_deref(), &outer) { + NpmrcPlan::AlreadyAll => npm_allow_remote_already_detail(&npm_hosts), + NpmrcPlan::UserSet(value) => npm_allow_remote_user_set_detail(&npm_hosts, &value), + NpmrcPlan::EnvSet { var, value } => { + npm_allow_remote_env_set_detail(&npm_hosts, &var, &value) } - } - Err(why) => npm_allow_remote_unreadable_detail(&npm_hosts, &why), + NpmrcPlan::OuterSet { layer, path, value } => { + npm_allow_remote_outer_set_detail(&npm_hosts, layer, &path, &value) + } + NpmrcPlan::Unsupported(why) => npm_allow_remote_unreadable_detail(&npm_hosts, &why), + _ if !options.npm_allow_remote_config => npm_allow_remote_manual_detail(&npm_hosts), + NpmrcPlan::Create(text) => { + npmrc_config_write = Some((text, edit("created"))); + npm_allow_remote_configured_detail(&npm_hosts, true, options.dry_run) + } + NpmrcPlan::Append(text) => { + npmrc_config_write = Some((text, edit("added"))); + npm_allow_remote_configured_detail(&npm_hosts, false, options.dry_run) + } + }, + Err(why) => npm_allow_remote_unreadable_detail(&npm_hosts, why), }; npm_warnings.push(warning("redirect_npm_allow_remote", detail)); + // #812: npm >= 8's `replace-registry-host` (`always`, or the pinned + // host itself) rewrites the hosted pins to the configured registry, + // so every install 404s. The setting is the user's, so it is reported + // (from whichever layer sets it), never overridden. + let project = npmrc.as_ref().ok().and_then(|t| t.as_deref()); + if let Some((value, source)) = effective_replace_registry_host(project, &outer) { + let blocked: Vec<&str> = npm_hosts + .iter() + .copied() + .filter(|host| replace_registry_host_rewrites(&value, host)) + .collect(); + if !blocked.is_empty() { + npm_warnings.push(warning( + NPM_REPLACE_REGISTRY_HOST_CODE, + npm_replace_registry_host_detail(&blocked, &value, &source), + )); + } + } (npm_warnings, npmrc_config_write) } @@ -2145,7 +2165,9 @@ fn confirm( let uuid = c.dep.patch_uuid.as_str(); // vlt decides before the binary-bun rule, so `bun.lockb` beside // a vlt-driven `vlt-lock.json` never confirms an npm purl. - if rewrite.refused_vlt_uuids.contains(uuid) || rewrite.refused_bun_uuids.contains(uuid) + if rewrite.refused_vlt_uuids.contains(uuid) + || rewrite.refused_bun_uuids.contains(uuid) + || rewrite.refused_npm_uuids.contains(uuid) { return ProbeStep::Decided(false); } diff --git a/crates/socket-patch-core/src/hosted/guidance.rs b/crates/socket-patch-core/src/hosted/guidance.rs index 4d430290c..db8d82d9a 100644 --- a/crates/socket-patch-core/src/hosted/guidance.rs +++ b/crates/socket-patch-core/src/hosted/guidance.rs @@ -406,6 +406,54 @@ pub fn npm_allow_remote_unreadable_detail(hosts: &[&str], why: &str) -> String { ) } +/// Warning code for a hosted npm pin that npm's +/// `replace-registry-host` setting rewrites to the configured registry +/// (#812). +pub const NPM_REPLACE_REGISTRY_HOST_CODE: &str = "redirect_npm_replace_registry_host"; + +/// npm (>= 8) `replace-registry-host` rewrites the hosted pins' origin to +/// the configured registry, so every install fetches `/patch/…` +/// and fails E404 (closed: never unpatched bytes). Nothing is written to +/// override it — like an explicit `allow-remote`, the setting is the +/// user's — so the warning names where it is set and both remedies. +pub fn npm_replace_registry_host_detail( + hosts: &[&str], + value: &str, + source: &crate::patch::redirect::npmrc::SettingSource, +) -> String { + use crate::patch::redirect::npmrc::SettingSource; + let (where_, fix) = match source { + SettingSource::Env(var) => ( + format!("the environment variable {var} sets"), + format!( + "unset {var} (npm's environment layer overrides every .npmrc) or set it to \ + `npmjs`" + ), + ), + SettingSource::Project => ( + "the project .npmrc sets".to_string(), + "change it to `replace-registry-host=npmjs` (npm's default) or remove it".to_string(), + ), + SettingSource::File { layer, path } => ( + format!("the {layer} npm config ({}) sets", path.display()), + format!( + "set `replace-registry-host=npmjs` (npm's default) in the project .npmrc (it \ + outranks the {layer} config) or change the {layer} config" + ), + ), + }; + format!( + "the npm lockfile now resolves patched dependencies from the hosted patch server ({}), \ + but {where_} `replace-registry-host={value}`, which makes npm >=8 rewrite those \ + `resolved` URLs to the configured registry: every `npm ci` / `npm install` then \ + fails E404 (a warm npm cache can hide this locally; a fresh checkout or CI runner \ + fails). To install the hosted patches, {fix}; or switch this project to vendored \ + patches (`socket-patch scan --mode vendored`), whose `file:` resolutions npm never \ + rewrites", + hosts.join(", "), + ) +} + /// The project `.npmrc` read, classified for the allow-remote auto-config: /// `Ok(Some(text))` — a regular file read fine; `Ok(None)` — ABSENT (the /// only state where planning a Create is safe); `Err(why)` — present but diff --git a/crates/socket-patch-core/src/patch/redirect/lock_index_equivalence_tests.rs b/crates/socket-patch-core/src/patch/redirect/lock_index_equivalence_tests.rs index 2e61248f0..79225d6e8 100644 --- a/crates/socket-patch-core/src/patch/redirect/lock_index_equivalence_tests.rs +++ b/crates/socket-patch-core/src/patch/redirect/lock_index_equivalence_tests.rs @@ -163,7 +163,14 @@ fn indexed_npm_lock_rewrite_matches_golden() { let refs: Vec<&DepOverride> = deps.iter().collect(); for lockfile in ["package-lock.json", "npm-shrinkwrap.json"] { let mut got = RewriteResult::default(); - rewrite_one_npm_lock(&text, lockfile, &refs, &NpmOverrides::default(), &mut got); + rewrite_one_npm_lock( + &text, + parse_json_text(&text).ok(), + lockfile, + &refs, + &NpmOverrides::default(), + &mut got, + ); record(&(&text, lockfile, &deps), &got); edits += got.edits.len(); codes.extend(got.warnings.iter().map(|w| w.code.clone())); diff --git a/crates/socket-patch-core/src/patch/redirect/mod.rs b/crates/socket-patch-core/src/patch/redirect/mod.rs index cf21a0c54..8ecbd8a84 100644 --- a/crates/socket-patch-core/src/patch/redirect/mod.rs +++ b/crates/socket-patch-core/src/patch/redirect/mod.rs @@ -28,8 +28,10 @@ use crate::utils::digest::is_hex64_lower; #[cfg(test)] use crate::utils::line_endings::LineEndings; use crate::vendor::common::{parse_json_text, JsonLayout}; -use crate::vendor::lock_inventory::npm_legacy_identity; -use crate::vendor::npm_origin::{legacy_packages_key, npm_non_registry_entries, NpmOverrides}; +use crate::vendor::lock_inventory::{npm_lock_entries, NpmLockEntry, NpmLockSection}; +use crate::vendor::npm_origin::{ + npm_non_registry_entries, npm_shrinkwrapped_entries, NpmOverrides, +}; mod bun_binary; pub use bun_binary::{preflight_bun_binary, rewrite_bun_binary}; @@ -283,6 +285,16 @@ pub struct RewriteResult { serde(skip_serializing_if = "std::collections::BTreeSet::is_empty") )] pub refused_bun_uuids: std::collections::BTreeSet, + /// Patch uuids the npm lock rewriter left on their registry entry + /// because the project patches that package itself with npm 12's + /// native `npm patch` (#711). Never confirmed: npm applies the user's + /// diff on top of whatever tarball the lock names, so a hosted pin + /// either fails `EPATCHFAILED` or installs bytes VEX can't attest. + #[cfg_attr( + test, + serde(skip_serializing_if = "std::collections::BTreeSet::is_empty") + )] + pub refused_npm_uuids: std::collections::BTreeSet, /// Patch uuids whose package version a yarn berry `yarn.lock` locks, so /// the berry rewriter alone decides them: the hosted pin is the /// URL-keyed lock entry AND the root `package.json` `resolutions` @@ -790,6 +802,7 @@ fn merge_group_delta(result: &mut RewriteResult, delta: RewriteResult) { refused_pdm_uuids, refused_pnpm_uuids, refused_bun_uuids, + refused_npm_uuids, yarn_berry_uuids, confirmed_yarn_berry_uuids, refused_yarn_classic_uuids, @@ -825,6 +838,7 @@ fn merge_group_delta(result: &mut RewriteResult, delta: RewriteResult) { result.refused_pdm_uuids.extend(refused_pdm_uuids); result.refused_pnpm_uuids.extend(refused_pnpm_uuids); result.refused_bun_uuids.extend(refused_bun_uuids); + result.refused_npm_uuids.extend(refused_npm_uuids); result.yarn_berry_uuids.extend(yarn_berry_uuids); result .confirmed_yarn_berry_uuids @@ -1052,14 +1066,17 @@ fn rewrite_npm_lock( return; } // BOTH npm locks can legitimately co-exist and BOTH must be rewritten. - // `npm shrinkwrap` was removed in npm 12, which now auto-creates a - // `package-lock.json` beside any committed `npm-shrinkwrap.json` on first - // install and reifies the install FROM `package-lock.json`. So the + // `npm shrinkwrap` was removed in npm 12, which never reads a committed + // `npm-shrinkwrap.json`: its first install resolves from the registry, + // writes a `package-lock.json` beside it and reifies FROM that. So the // dual-lock state is the DEFAULT for a shrinkwrap repo under npm 12. // Rewriting only the first present lock would patch the file npm doesn't // install from — a silent FALSE SUCCESS. Rewrite EVERY present npm lock so - // a fresh `npm install`/`npm ci` from EITHER is redirected (shrinkwrap-only - // repos on npm <= 6 keep working: only that one file is present). + // a fresh `npm install`/`npm ci` from EITHER is redirected. A + // shrinkwrap-ONLY repo is still rewritten (npm <= 11 installs from it), + // but npm 12 never reads npm-shrinkwrap.json — it resolves from the + // registry and writes its own package-lock.json — so that is SAID + // (#899, `redirect_npm_shrinkwrap_only`). let present: Vec<&str> = crate::constants::npm_family::NPM_LOCKS .into_iter() .filter(|f| files.contains_key(*f)) @@ -1108,15 +1125,202 @@ fn rewrite_npm_lock( .get("package.json") .map(|text| NpmOverrides::from_manifest_text(text)) .unwrap_or_default(); - for lockfile in present { + // A package the project patches itself with npm 12's native `npm patch` + // is left on its registry entries in EVERY present lock (#711): the root + // `patchedDependencies` key, or the `"patched": {integrity, path}` + // record npm writes on the lock entry (lockfileVersion 4). npm applies + // that diff to every install of the entry, so rewriting a sibling lock + // would still stack the user's diff on the hosted bytes. Each present + // lock is parsed ONCE, here, and the parse is handed to its rewrite; + // only a lock that spells `"patched"` at all is walked for the gate. + let user_patched = crate::vendor::bun_lock_text::patched_dependency_keys( + files.get("package.json").map(String::as_str), + None, + ); + let parsed: Vec<(&str, Option)> = present + .iter() + .map(|lockfile| (*lockfile, parse_json_text(&files[*lockfile]).ok())) + .collect(); + let lock_patched: Vec<(String, String, &str, String)> = parsed + .iter() + .filter(|(lockfile, _)| files[*lockfile].contains("\"patched\"")) + .filter_map(|(lockfile, lock)| Some((*lockfile, lock.as_ref()?))) + .flat_map(|(lockfile, lock)| { + npm_lock_entries(lock) + .into_iter() + .filter(|e| e.section == NpmLockSection::Packages && e.is_dependency()) + .filter(|e| e.value.get("patched").is_some_and(|p| !p.is_null())) + .filter_map(|e| { + let version = e.node.version?.to_string(); + Some(( + e.node.name.to_string(), + version, + lockfile, + e.key.to_string(), + )) + }) + .collect::>() + }) + .collect(); + let npm: Vec<&DepOverride> = npm + .into_iter() + .filter(|dep| { + let fname = full_name(dep); + let source = if let Some(key) = crate::vendor::bun_lock_text::patched_dependency_key( + &user_patched, + &fname, + &dep.version, + ) { + format!("package.json `patchedDependencies` has `{key}`") + } else if let Some((_, _, lockfile, key)) = lock_patched + .iter() + .find(|(name, version, _, _)| *name == fname && *version == dep.version) + { + format!("{lockfile} entry `{key}` carries npm's `patched` record") + } else { + return true; + }; + // A lock an earlier (pre-#711) hosted run already pinned keeps + // that pin: say so, or the warning's "left unchanged" would + // read as healthy while every install still fails. + let pinned_in: Vec<&str> = present + .iter() + .copied() + .filter(|lockfile| files[*lockfile].contains(dep.artifact_url.as_str())) + .collect(); + skip_npm_user_patched(&source, &fname, dep, &pinned_in, result); + false + }) + .collect(); + for (lockfile, lock) in parsed { rewrite_one_npm_lock( &files[lockfile], + lock, lockfile, &npm, &manifest_overrides, result, ); } + if let [SHRINKWRAP] = present.as_slice() { + warn_npm_shrinkwrap_only(files, &npm, result); + } +} + +const SHRINKWRAP: &str = crate::constants::npm_family::NPM_LOCKS[0]; +const PACKAGE_LOCK: &str = crate::constants::npm_family::NPM_LOCKS[1]; + +/// #899: the root `npm-shrinkwrap.json` is the ONLY npm lock and it carries +/// a hosted redirect (spliced this run or by an earlier one — the warning +/// repeats until the project gains the twin). npm 12 ignores the shrinkwrap, +/// so its installs fetch the unpatched registry bytes. +fn warn_npm_shrinkwrap_only( + files: &BTreeMap, + npm: &[&DepOverride], + result: &mut RewriteResult, +) { + let lock = result + .files + .get(SHRINKWRAP) + .or_else(|| files.get(SHRINKWRAP)); + let wired: Vec = npm + .iter() + .filter(|dep| lock.is_some_and(|text| text.contains(dep.artifact_url.as_str()))) + .map(|dep| format!("{}@{}", full_name(dep), dep.version)) + .collect(); + if wired.is_empty() { + return; + } + result.warnings.push(RewriteWarning { + code: "redirect_npm_shrinkwrap_only".into(), + detail: npm_shrinkwrap_only_detail(&wired, "redirected"), + }); +} + +/// The shared detail of the hosted (`redirect_npm_shrinkwrap_only`) and +/// vendored (`vendor_npm_shrinkwrap_only`) shrinkwrap-only warnings. +pub fn npm_shrinkwrap_only_detail(packages: &[String], how: &str) -> String { + format!( + "{} {how} in {SHRINKWRAP} only — there is no {PACKAGE_LOCK}, and npm >= 12 never \ + reads {SHRINKWRAP}: it resolves the tree from the registry and writes a fresh \ + {PACKAGE_LOCK}, so npm >= 12 installs stay UNPATCHED (npm <= 11 installs from the \ + shrinkwrap and is patched); rename the lock to {PACKAGE_LOCK} (or commit a copy \ + under that name) and re-run", + packages.join(", ") + ) +} + +/// Leave `dep` on its registry entry because the project patches it with +/// npm 12's native `npm patch` (#711). npm extracts whatever tarball the +/// lock names and then applies the user's diff, failing the whole install +/// `EPATCHFAILED` when the diff no longer applies: a hosted pin of the +/// same lines breaks every later `npm ci` / `npm install`, and one that +/// does apply installs bytes that are neither the original nor the Socket +/// patch, which VEX can never attest. Warns, keeps the in-run VEX from +/// assuming the uuid patched, and keeps any other lock from confirming it. +fn skip_npm_user_patched( + source: &str, + name: &str, + dep: &DepOverride, + pinned_in: &[&str], + result: &mut RewriteResult, +) { + result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); + result.refused_npm_uuids.insert(dep.patch_uuid.clone()); + let state = if pinned_in.is_empty() { + "It is left unchanged and stays without the Socket patch".to_string() + } else { + format!( + "An earlier hosted run already pinned it in {}, which keeps breaking installs \ + until `socket-patch rollback {}` restores the registry entry", + pinned_in.join(" and "), + dep.patch_uuid + ) + }; + result.warnings.push(RewriteWarning { + code: "redirect_npm_patched_dependency_skipped".into(), + detail: format!( + "{source}, a patch the project applies with npm's native `npm patch`; npm applies \ + it on top of whatever tarball the lock names, so pinning {name}@{} to the hosted \ + patch would fail every install `EPATCHFAILED` (or install bytes that match \ + neither the original nor the Socket patch). {state}. To get the Socket fix, fold \ + it into your own patch, or remove the `patchedDependencies` entry and re-run", + dep.version + ), + }); +} + +/// An installed npm lock entry ([`npm_lock_entries`]), detached from the +/// lock so the rewrite can edit the document through `pointer`. +struct NpmLockTarget { + section: NpmLockSection, + /// The `packages` key, or the legacy dependency name: what warnings + /// name and the ledger records as the edit key. + key: String, + pointer: String, + packages_key: String, + name: String, + version: Option, + link: bool, + bundled: bool, + /// A legacy alias node (#432). + alias: bool, +} + +impl From> for NpmLockTarget { + fn from(entry: NpmLockEntry<'_>) -> Self { + NpmLockTarget { + section: entry.section, + key: entry.key.to_string(), + alias: entry.is_legacy_alias(), + pointer: entry.pointer, + packages_key: entry.packages_key.into_owned(), + name: entry.node.name.to_string(), + version: entry.node.version.map(str::to_string), + link: entry.link, + bundled: entry.bundled, + } + } } /// Rewrite a single npm lockfile (`package-lock.json` or `npm-shrinkwrap.json`) @@ -1124,13 +1328,15 @@ fn rewrite_npm_lock( /// the identical override rewrite (see the dual-lock note there). fn rewrite_one_npm_lock( content: &str, + lock: Option, lockfile: &str, npm: &[&DepOverride], manifest_overrides: &NpmOverrides, result: &mut RewriteResult, ) { - // npm reads past a leading UTF-8 BOM; so do we. - let Ok(mut lock) = parse_json_text(content) else { + // `lock` is `content` parsed (npm reads past a leading UTF-8 BOM; so + // does that parse), `None` when it is not JSON. + let Some(mut lock) = lock else { // A corrupt lockfile is strictly worse than a missing one (which // warns in the caller) — never skip the whole npm redirect silently. result.warnings.push(RewriteWarning { @@ -1139,43 +1345,25 @@ fn rewrite_one_npm_lock( }); return; }; - // The (package, version) each `packages` entry stands for, by map - // position, computed once: the per-dep scan below compares against it - // instead of re-deriving it for every entry for every dep. Sound - // because a rewrite only ever touches an entry's `resolved`/`integrity` - // (never a key, `name` or `version`), so positions and identities hold. - let package_ids: Vec)>> = lock - .get("packages") - .and_then(Value::as_object) - .map(|packages| { - packages - .iter() - .map(|(key, entry)| { - // Only `node_modules/` keys are installable dependencies: - // "" is the project root and other bare keys are workspace - // members — SOURCE dirs a resolved/integrity insert would - // corrupt. - let (_, key_name) = key.rsplit_once("node_modules/")?; - // The package a lock entry stands for: the explicit `name` - // field when present (npm writes it for aliases — `npm i - // alias@npm:real` keys the entry by the ALIAS), else the - // key's trailing path. Mirrors `vendor::npm_lock`'s - // `entry_name`, so an alias install of the patched package - // redirects and an entry that merely SHARES the key name - // (`npm i @npm:other`) is never hijacked. - let entry_nm = entry - .get("name") - .and_then(Value::as_str) - .unwrap_or(key_name); - let version = entry.get("version").and_then(Value::as_str); - Some((entry_nm.to_string(), version.map(str::to_string))) - }) - .collect() - }) - .unwrap_or_default(); + // Every installed entry, detached from the lock and computed once: the + // per-dep scan below compares against it instead of re-walking the lock + // for every dep. Sound because a rewrite only ever touches an entry's + // `resolved`/`integrity` (never a key, `name`, `version` or flag), so + // addresses and identities hold. + let targets: Vec = npm_lock_entries(&lock) + .into_iter() + .filter(NpmLockEntry::is_dependency) + .map(NpmLockTarget::from) + .collect(); // Entries npm installs from a git / url / `file:` spec: see // `vendor::npm_origin` (#326). let non_registry = npm_non_registry_entries(&lock, manifest_overrides); + // Entries npm 7–11 install from a dependency's own shrinkwrap (#753). + let shrinkwrapped = npm_shrinkwrapped_entries(&lock); + // The legacy mirror of an entry the `packages` scan skips (and warns + // about) is never rewired either. + let mut mirror_skipped = non_registry.clone(); + mirror_skipped.extend(shrinkwrapped.clone()); // npm 7+ reads `packages` when it exists; the legacy `dependencies` // mirror must not suppress an attestation for that install tree. // Match the shared npm lock inventory's object-valued-map precedence. @@ -1193,91 +1381,125 @@ fn rewrite_one_npm_lock( continue; }; let mut matched_any = false; - if let Some(packages) = lock.get_mut("packages").and_then(Value::as_object_mut) { - for ((key, entry), id) in packages.iter_mut().zip(&package_ids) { - let Some((entry_nm, version)) = id else { - continue; - }; - if *entry_nm != fname || version.as_deref() != Some(dep.version.as_str()) { - continue; - } - if entry.get("link").and_then(Value::as_bool) == Some(true) { - matched_any = true; - result.warnings.push(RewriteWarning { - code: "redirect_npm_link_entry_skipped".into(), - detail: format!( - "lock entry `{key}` is a link (npm workspaces/file: dir); skipped" - ), - }); - continue; - } - // npm reify extracts a bundled copy from its PARENT's tarball - // and ignores the entry's resolved/integrity, so a rewrite - // here would put the hosted URL in the lockfile (confirming - // and VEX-attesting the patch) while the unpatched bundled - // bytes keep installing. Mirrors the vendored backend's - // `vendor_bundled_instance_skipped` refusal. The uuid is - // recorded so the in-run `--vex` verifies instead of - // assuming the patch applied (#325, as Bun's #469). - if entry.get("inBundle").and_then(Value::as_bool) == Some(true) { - matched_any = true; - result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); - result.warnings.push(RewriteWarning { - code: "redirect_npm_bundled_instance_skipped".into(), - detail: format!( - "lock entry `{key}` is bundled inside its parent's tarball and \ - CANNOT be redirected — that copy stays UNPATCHED; vendor or \ - update the bundling parent to cover it" - ), - }); - continue; + for target in &targets { + if target.name != fname || target.version.as_deref() != Some(dep.version.as_str()) { + continue; + } + let key = target.key.as_str(); + matched_any = true; + let (kind, edit_key) = match target.section { + NpmLockSection::Packages => { + if target.link { + result.warnings.push(RewriteWarning { + code: "redirect_npm_link_entry_skipped".into(), + detail: format!( + "lock entry `{key}` is a link (npm workspaces/file: dir); skipped" + ), + }); + continue; + } + // npm reify extracts a bundled copy from its PARENT's + // tarball and ignores the entry's resolved/integrity, so a + // rewrite here would put the hosted URL in the lockfile + // (confirming and VEX-attesting the patch) while the + // unpatched bundled bytes keep installing. Mirrors the + // vendored backend's `vendor_bundled_instance_skipped` + // refusal. The uuid is recorded so the in-run `--vex` + // verifies instead of assuming the patch applied (#325, as + // Bun's #469). + if target.bundled { + result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); + result.warnings.push(RewriteWarning { + code: "redirect_npm_bundled_instance_skipped".into(), + detail: format!( + "lock entry `{key}` is bundled inside its parent's tarball \ + and CANNOT be redirected — that copy stays UNPATCHED; vendor \ + or update the bundling parent to cover it" + ), + }); + continue; + } + // npm 7–11 install everything beneath a `hasShrinkwrap` + // package from that package's own npm-shrinkwrap.json and + // ignore the root lock's entry, so a rewrite here would + // confirm (and VEX-attest) a patch that never installs + // there (#753). Recorded like a bundled copy so the in-run + // `--vex` verifies instead of assuming. + if let Some(ancestor) = shrinkwrapped.get(key) { + result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); + result.warnings.push(RewriteWarning { + code: "redirect_npm_shrinkwrapped_instance_skipped".into(), + detail: format!( + "lock entry `{key}` is installed from `{ancestor}`'s own \ + npm-shrinkwrap.json (hasShrinkwrap), which npm 7–11 read \ + instead of this lock, so it CANNOT be redirected — that copy \ + stays UNPATCHED; vendor or update `{ancestor}` to cover it" + ), + }); + continue; + } + // npm installs a git / url / `file:` dependency from the + // dependent's spec and ignores `resolved`, so a rewrite + // here would confirm (and VEX-attest) a patch that never + // installs. + if let Some(reason) = non_registry.get(key) { + result.warnings.push(RewriteWarning { + code: "redirect_npm_non_registry_entry_skipped".into(), + detail: format!( + "lock entry `{key}` is not installed from the registry \ + ({reason}) and CANNOT be redirected — npm installs it from \ + that spec, so that copy stays UNPATCHED; depend on the \ + registry release to patch it" + ), + }); + continue; + } + ("redirect_npm_lock_entry", key) } - // npm installs a git / url / `file:` dependency from the - // dependent's spec and ignores `resolved`, so a rewrite here - // would confirm (and VEX-attest) a patch that never installs. - if let Some(reason) = non_registry.get(key.as_str()) { - matched_any = true; - result.warnings.push(RewriteWarning { - code: "redirect_npm_non_registry_entry_skipped".into(), - detail: format!( - "lock entry `{key}` is not installed from the registry ({reason}) \ - and CANNOT be redirected — npm installs it from that spec, so \ - that copy stays UNPATCHED; depend on the registry release to \ - patch it" - ), - }); - continue; + // The legacy `dependencies` tree (keyed by name): an alias + // node (`"lp": {"version": "npm:left-pad@1.3.0"}`) is an + // install of its target, like the `packages` twin's `name` + // field. + NpmLockSection::Legacy => { + // Legacy spelling of `inBundle`: same + // npm-ignores-the-rewrite fail-open as the `packages` + // guard above. + if target.bundled { + if legacy_is_install_tree { + result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); + } + result.warnings.push(RewriteWarning { + code: "redirect_npm_bundled_instance_skipped".into(), + detail: format!( + "legacy dependencies entry `{key}` is bundled inside its \ + parent's tarball and CANNOT be redirected — that copy stays \ + UNPATCHED; vendor or update the bundling parent to cover it" + ), + }); + continue; + } + // The mirror of a `packages` entry npm installs from a git + // / url / `file:` spec, or from a dependency's own + // shrinkwrap (#753): that twin was skipped (and warned + // about) above, so rewriting this copy would only record + // an edit for bytes that never install. + if mirror_skipped.contains_key(&target.packages_key) { + continue; + } + ("redirect_npm_lock_dep", key) } - matched_any = true; - if let Some(edit) = rewrite_npm_entry( - entry, - dep, - &sha512, - lockfile, - "redirect_npm_lock_entry", - key, - ) { - result.edits.push(edit); - changed = true; + }; + let Some(entry) = lock.pointer_mut(&target.pointer) else { + continue; + }; + if let Some(edit) = rewrite_npm_entry(entry, dep, &sha512, lockfile, kind, edit_key) { + result.edits.push(edit); + changed = true; + if target.alias { + aliased.push(target.key.clone()); } } } - // v2 legacy `dependencies` tree (keyed by name), recursive. - if let Some(deps) = lock.get_mut("dependencies").and_then(Value::as_object_mut) { - changed = rewrite_npm_v2_deps( - deps, - "", - &non_registry, - &fname, - dep, - &sha512, - lockfile, - legacy_is_install_tree, - result, - &mut matched_any, - &mut aliased, - ) || changed; - } // Parity with the pnpm/berry/uv rewriters: a granted dep the // lockfile cannot pin must be SAID, not silently dropped from the // redirected count. @@ -1370,82 +1592,6 @@ fn rewrite_npm_entry( }) } -#[allow(clippy::too_many_arguments)] -fn rewrite_npm_v2_deps( - deps: &mut serde_json::Map, - parent_key: &str, - non_registry: &BTreeMap, - fname: &str, - dep: &DepOverride, - sha512: &str, - lockfile: &str, - legacy_is_install_tree: bool, - result: &mut RewriteResult, - matched_any: &mut bool, - aliased: &mut Vec, -) -> bool { - let mut changed = false; - for (name, entry) in deps.iter_mut() { - let packages_key = legacy_packages_key(parent_key, name); - // An alias node (`"lp": {"version": "npm:left-pad@1.3.0"}`) is an - // install of its target, like the `packages` twin's `name` field. - let (node_name, node_version) = - npm_legacy_identity(name, entry.get("version").and_then(Value::as_str)); - let is_alias = node_name != name.as_str(); - if node_name == fname && node_version == Some(dep.version.as_str()) { - // Legacy spelling of `inBundle`: same npm-ignores-the-rewrite - // fail-open as the `packages` guard above. - if entry.get("bundled").and_then(Value::as_bool) == Some(true) { - *matched_any = true; - if legacy_is_install_tree { - result.bundled_skipped_uuids.insert(dep.patch_uuid.clone()); - } - result.warnings.push(RewriteWarning { - code: "redirect_npm_bundled_instance_skipped".into(), - detail: format!( - "legacy dependencies entry `{name}` is bundled inside its parent's \ - tarball and CANNOT be redirected — that copy stays UNPATCHED; vendor \ - or update the bundling parent to cover it" - ), - }); - } else if non_registry.contains_key(&packages_key) { - // The mirror of a `packages` entry npm installs from a git / - // url / `file:` spec: that twin was skipped (and warned about) - // above, so rewriting this copy would only record an edit for - // bytes that never install. - *matched_any = true; - } else { - *matched_any = true; - if let Some(edit) = - rewrite_npm_entry(entry, dep, sha512, lockfile, "redirect_npm_lock_dep", name) - { - result.edits.push(edit); - changed = true; - if is_alias { - aliased.push(name.clone()); - } - } - } - } - if let Some(nested) = entry.get_mut("dependencies").and_then(Value::as_object_mut) { - changed = rewrite_npm_v2_deps( - nested, - &packages_key, - non_registry, - fname, - dep, - sha512, - lockfile, - legacy_is_install_tree, - result, - matched_any, - aliased, - ) || changed; - } - } - changed -} - // ── cargo (Cargo.toml + .cargo/config.toml + Cargo.lock) ───────────────────── // // TRANSACTIONAL per dependency: a dep is redirected ONLY if its Cargo.toml pin @@ -16445,12 +16591,15 @@ mod tests { ); } - /// A shrinkwrap-ONLY project (the npm <= 6 world, where `npm shrinkwrap` - /// wrote the sole lock and no `package-lock.json` was auto-created) must - /// still be rewritten with zero warnings — the dual-lock handling must - /// not perturb the single-lock path. + /// A shrinkwrap-ONLY project (what `npm shrinkwrap` leaves: no + /// `package-lock.json`) is still rewritten — npm <= 11 installs from it — + /// and NO package-lock.json is invented. REGRESSION (#899): npm 12 never + /// reads the shrinkwrap (it resolves from the registry and writes a + /// fresh package-lock.json), so the run warns + /// `redirect_npm_shrinkwrap_only` — again on an in-sync re-run — instead + /// of reporting a clean success. #[test] - fn npm_shrinkwrap_only_still_rewritten_no_warnings() { + fn npm_shrinkwrap_only_rewritten_and_warns_npm12_ignores_it() { let ovr = npm_override( "left-pad", "1.3.0", @@ -16492,10 +16641,196 @@ mod tests { ); assert_eq!( warning_codes(&r), - Vec::<&str>::new(), - "a clean shrinkwrap-only success must emit NO warnings: {:?}", + vec!["redirect_npm_shrinkwrap_only"], + "{:?}", r.warnings ); + let detail = &r.warnings[0].detail; + for needle in [ + "left-pad@1.3.0", + "npm >= 12", + "no package-lock.json", + "UNPATCHED", + ] { + assert!(detail.contains(needle), "{needle}: {detail}"); + } + + // The in-sync re-run writes nothing and still warns. + files.insert("npm-shrinkwrap.json".to_string(), out.clone()); + let again = rewrite_registry_redirect(&files, std::slice::from_ref(&ovr)); + assert!(again.files.is_empty(), "{:?}", again.files.keys()); + assert_eq!(warning_codes(&again), vec!["redirect_npm_shrinkwrap_only"]); + + // The twin npm 12 reads: both rewritten, no shrinkwrap-only warning. + files.insert("package-lock.json".to_string(), out.clone()); + let twin = rewrite_registry_redirect(&files, std::slice::from_ref(&ovr)); + assert_eq!( + warning_codes(&twin), + Vec::<&str>::new(), + "{:?}", + twin.warnings + ); + + // A shrinkwrap without any of the deps carries no redirect: no warning. + let mut other = BTreeMap::new(); + other.insert( + "npm-shrinkwrap.json".to_string(), + r#"{"lockfileVersion":3,"packages":{"":{"name":"app"}}}"#.to_string(), + ); + let none = rewrite_registry_redirect(&other, std::slice::from_ref(&ovr)); + assert!( + !warning_codes(&none).contains(&"redirect_npm_shrinkwrap_only"), + "{:?}", + none.warnings + ); + } + + /// REGRESSION (#711): npm 12.1+ `npm patch` records the project's own + /// diff in the root `patchedDependencies` and on the lock entry + /// (`"patched"`, lockfileVersion 4), then applies it on top of whatever + /// tarball the lock names. Pinning that entry to the hosted patch made + /// every later `npm ci` fail `EPATCHFAILED` while the scan reported a + /// clean switch. The entry keeps its registry tuple in EVERY present + /// lock, the run says why, and the in-run VEX never assumes it patched; + /// another granted package in the same lock is still rewired. + #[test] + fn npm_lock_user_patched_dependency_is_left_alone_loudly() { + let ovr = npm_override( + "left-pad", + "1.3.0", + "http://p.test/lp.tgz", + "sha512-PATCHED==", + ); + let mut other = npm_override( + "is-number", + "7.0.0", + "http://p.test/isn.tgz", + "sha512-PATCHED==", + ); + other.patch_uuid = "22222222-2222-4222-8222-222222222222".into(); + let lock = |patched: bool| { + let record = if patched { + ",\n \"patched\": {\n \"integrity\": \"sha512-USER==\",\n \ + \"path\": \"patches/left-pad@1.3.0.patch\"\n }" + } else { + "" + }; + format!( + "{{\n \"name\": \"app\",\n \"lockfileVersion\": {},\n \"packages\": {{\n \ + \"\": {{\n \"name\": \"app\"\n }},\n \"node_modules/is-number\": {{\n \ + \"version\": \"7.0.0\",\n \ + \"resolved\": \"https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz\",\n \ + \"integrity\": \"sha512-UPSTREAM==\"\n }},\n \"node_modules/left-pad\": {{\n \ + \"version\": \"1.3.0\",\n \ + \"resolved\": \"https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz\",\n \ + \"integrity\": \"sha512-UPSTREAM==\"{record}\n }}\n }}\n}}\n", + if patched { 4 } else { 3 } + ) + }; + let manifest = r#"{"name":"app","dependencies":{"left-pad":"1.3.0","is-number":"7.0.0"},"patchedDependencies":{"left-pad@1.3.0":"patches/left-pad@1.3.0.patch"}}"#; + + // Each signal alone gates the dep: the manifest key (over a lock + // that does not record it), or the lock's `patched` record (with no + // manifest read), and both together. A shrinkwrap beside a + // `patched` package-lock is left alone for that dep too. + for (with_manifest, with_record, shrinkwrap) in [ + (true, false, false), + (false, true, false), + (true, true, false), + (false, true, true), + ] { + let case = format!("manifest={with_manifest} record={with_record} sw={shrinkwrap}"); + let mut files = BTreeMap::new(); + files.insert("package-lock.json".to_string(), lock(with_record)); + if shrinkwrap { + files.insert("npm-shrinkwrap.json".to_string(), lock(false)); + } + if with_manifest { + files.insert("package.json".to_string(), manifest.to_string()); + } + let r = rewrite_registry_redirect(&files, &[ovr.clone(), other.clone()]); + assert_eq!(r.edits.len(), if shrinkwrap { 2 } else { 1 }, "{case}"); + assert!( + r.edits + .iter() + .all(|e| e.key.as_deref() == Some("node_modules/is-number")), + "{case}: {:?}", + r.edits + ); + for out in r.files.values() { + assert!( + out.contains("https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz") + && !out.contains("http://p.test/lp.tgz"), + "{case}: the user-patched entry keeps its registry tuple: {out}" + ); + } + assert_eq!( + warning_codes(&r), + vec!["redirect_npm_patched_dependency_skipped"], + "{case}: {:?}", + r.warnings + ); + let detail = &r.warnings[0].detail; + assert!( + detail.contains("left-pad@1.3.0") + && detail.contains("npm patch") + && detail.contains("EPATCHFAILED"), + "{case}: {detail}" + ); + if !with_manifest { + assert!( + detail.contains("package-lock.json entry `node_modules/left-pad`"), + "{case}: {detail}" + ); + } + assert!(r.bundled_skipped_uuids.contains(&ovr.patch_uuid), "{case}"); + assert!(r.refused_npm_uuids.contains(&ovr.patch_uuid), "{case}"); + assert!(!r.refused_npm_uuids.contains(&other.patch_uuid), "{case}"); + } + + // A patch for ANOTHER version, or a null `patched`, gates nothing. + let mut files = BTreeMap::new(); + files.insert( + "package-lock.json".to_string(), + lock(false).replace( + "\"integrity\": \"sha512-UPSTREAM==\"\n }\n }", + "\"integrity\": \"sha512-UPSTREAM==\",\n \"patched\": null\n }\n }", + ), + ); + files.insert( + "package.json".to_string(), + manifest.replace("left-pad@1.3.0\":", "left-pad@1.2.0\":"), + ); + let r = rewrite_registry_redirect(&files, &[ovr.clone(), other.clone()]); + assert_eq!(r.edits.len(), 2, "{:?} {:?}", r.edits, r.warnings); + assert!(r.warnings.is_empty(), "{:?}", r.warnings); + assert!(r.refused_npm_uuids.is_empty()); + + // A lock an earlier (pre-fix) hosted run already pinned keeps that + // pin: the warning must say so and name the rollback, never claim + // the entry is unchanged while every install still fails. + let mut files = BTreeMap::new(); + files.insert( + "package-lock.json".to_string(), + lock(true).replace( + "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "http://p.test/lp.tgz", + ), + ); + let r = rewrite_registry_redirect(&files, std::slice::from_ref(&ovr)); + assert!(r.edits.is_empty(), "{:?}", r.edits); + assert_eq!( + warning_codes(&r), + vec!["redirect_npm_patched_dependency_skipped"] + ); + let detail = &r.warnings[0].detail; + assert!( + detail.contains("already pinned it in package-lock.json") + && detail.contains(&format!("socket-patch rollback {}", ovr.patch_uuid)) + && !detail.contains("left unchanged"), + "{detail}" + ); + assert!(r.refused_npm_uuids.contains(&ovr.patch_uuid)); } /// #324: the hosted npm rewrite changes only the rewired values and keeps @@ -16633,6 +16968,106 @@ mod tests { ); } + /// REGRESSION (#753): npm 7–11 install everything beneath a + /// `hasShrinkwrap` package from that package's own npm-shrinkwrap.json, + /// ignoring the root lock's entry. Rewriting the nested entry (or its + /// v2 legacy mirror) would confirm a patch npm never installs, so it is + /// skipped loudly, while a hoisted copy of the same version elsewhere + /// is still redirected. + #[test] + fn npm_entry_under_has_shrinkwrap_parent_is_skipped_with_loud_warning() { + let mut files = BTreeMap::new(); + files.insert( + "package-lock.json".to_string(), + r#"{ + "name": "app", + "lockfileVersion": 2, + "packages": { + "": { "name": "app", "version": "0.0.0" }, + "node_modules/@bh/sw": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-SW==", + "hasShrinkwrap": true + }, + "node_modules/@bh/sw/node_modules/left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-UPSTREAM==" + } + }, + "dependencies": { + "@bh/sw": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-SW==", + "dependencies": { + "left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-UPSTREAM==" + } + } + } + } +} +"# + .to_string(), + ); + let overrides = vec![npm_override( + "left-pad", + "1.3.0", + "http://patch.test/lp.tgz", + "sha512-PATCHED==", + )]; + let r = rewrite_registry_redirect(&files, &overrides); + assert!( + r.files.is_empty() && r.edits.is_empty(), + "a shrinkwrapped-only dep must change nothing: files={:?} edits={:?}", + r.files.keys(), + r.edits + ); + let skipped = r + .warnings + .iter() + .find(|w| w.code == "redirect_npm_shrinkwrapped_instance_skipped") + .unwrap_or_else(|| panic!("shrinkwrapped skip must warn: {:?}", r.warnings)); + assert!( + skipped.detail.contains("UNPATCHED") + && skipped + .detail + .contains("node_modules/@bh/sw/node_modules/left-pad") + && skipped.detail.contains("`node_modules/@bh/sw`"), + "the warning must name the entry and its shrinkwrap owner: {}", + skipped.detail + ); + assert!( + !warning_codes(&r).contains(&"redirect_npm_entry_not_found"), + "a shrinkwrapped skip is a MATCH: {:?}", + r.warnings + ); + assert!( + r.bundled_skipped_uuids.contains(&overrides[0].patch_uuid), + "the skipped copy must keep the patch out of the in-run VEX" + ); + + // A hoisted copy outside the shrinkwrapped subtree still redirects. + let lock = files["package-lock.json"].replace( + r#""node_modules/@bh/sw": {"#, + r#""node_modules/left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-UPSTREAM==" + }, + "node_modules/@bh/sw": {"#, + ); + files.insert("package-lock.json".to_string(), lock); + let r = rewrite_registry_redirect(&files, &overrides); + let keys: Vec<_> = r.edits.iter().filter_map(|e| e.key.as_deref()).collect(); + assert_eq!(keys, ["node_modules/left-pad"], "edits: {:?}", r.edits); + assert!(warning_codes(&r).contains(&"redirect_npm_shrinkwrapped_instance_skipped")); + } + /// #326: npm installs a git, remote-tarball or `file:` dependency from /// the dependent's spec and ignores the lock's `resolved`, so rewiring /// that entry would report (and VEX-attest) a patch `npm ci` never @@ -17107,8 +17542,8 @@ mod tests { /// An alias install (`npm i my-alias@npm:left-pad@1.3.0`) keys the lock /// entry by the ALIAS with the real package in `name`. Discovery is /// alias-aware (the crawler reads the installed package.json name), so - /// the rewriter must be too — matching on the entry's `name`, mirroring - /// `vendor::npm_lock::entry_name`. + /// the rewriter must be too — matching on the entry's `name`, the + /// shared `vendor::lock_inventory::npm_lock_entries` identity rule. #[test] fn npm_alias_entry_is_redirected() { let mut files = BTreeMap::new(); diff --git a/crates/socket-patch-core/src/patch/redirect/npmrc.rs b/crates/socket-patch-core/src/patch/redirect/npmrc.rs index 9d275a8e9..b2bf194d3 100644 --- a/crates/socket-patch-core/src/patch/redirect/npmrc.rs +++ b/crates/socket-patch-core/src/patch/redirect/npmrc.rs @@ -277,6 +277,20 @@ pub struct OuterAllowRemote { /// The highest-precedence explicit value among the user, global and /// builtin config files (all below the project `.npmrc`). pub file: Option, + /// `replace-registry-host` in the same outer layers (see + /// [`effective_replace_registry_host`]). + pub replace_registry_host: OuterSetting, +} + +/// One npm config key as the layers OUTSIDE the project `.npmrc` set it. +#[derive(Debug, Default, Clone, PartialEq)] +pub struct OuterSetting { + /// `(variable, value)` of an `npm_config_*` env var (beats every + /// `.npmrc`). + pub env: Option<(String, String)>, + /// The highest-precedence value among the user, global and builtin + /// config files. + pub file: Option, } /// One explicit `allow-remote` assignment in a non-project npm config file. @@ -430,9 +444,9 @@ impl NpmConfigEnv { /// An `npm_config_*` variable, matched the way npm's `loadEnv` does: /// prefix case-insensitive, then non-leading `_` → `-` and lowercased. /// Empty values are ignored (npm skips them). When several spellings - /// are set a non-`all` one wins (process env order is unspecified, so - /// the conservative reading is reported). - fn npm_config(&self, key: &str) -> Option<(String, String)> { + /// are set a non-`benign` one wins (process env order is unspecified, + /// so the conservative reading is reported). + fn npm_config(&self, key: &str, benign: &str) -> Option<(String, String)> { let mut found: Option<(String, String)> = None; for (k, v) in &self.vars { let Some(rest) = k @@ -453,7 +467,7 @@ impl NpmConfigEnv { c.to_ascii_lowercase() }); } - if norm == key && found.as_ref().is_none_or(|(_, prev)| prev == "all") { + if norm == key && found.as_ref().is_none_or(|(_, prev)| prev == benign) { found = Some((k.clone(), v.clone())); } } @@ -492,6 +506,43 @@ pub fn resolve_outer_allow_remote( env: &NpmConfigEnv, read: impl Fn(&std::path::Path) -> Option, ) -> OuterAllowRemote { + let layers = outer_config_layers(env, read); + OuterAllowRemote { + env: env.npm_config("allow-remote", "all"), + file: outer_file_value(&layers, "allow-remote"), + replace_registry_host: OuterSetting { + env: env.npm_config(REPLACE_REGISTRY_HOST, "npmjs"), + file: outer_file_value(&layers, REPLACE_REGISTRY_HOST), + }, + } +} + +/// [`resolve_outer_allow_remote`] for any npm config `key` (e.g. the +/// vendored flow's `allow-file`): the same layers, located the same way. +/// `benign` is the key's permissive value, used to pick among several env +/// spellings (a non-`benign` one wins). +pub fn resolve_outer_npm_setting( + env: &NpmConfigEnv, + read: impl Fn(&std::path::Path) -> Option, + key: &str, + benign: &str, +) -> OuterSetting { + let layers = outer_config_layers(env, read); + OuterSetting { + env: env.npm_config(key, benign), + file: outer_file_value(&layers, key), + } +} + +/// One non-project npm config file: `(layer, path, text)`. +type OuterLayer = (&'static str, Option, Option); + +/// The user, global and builtin config files (highest precedence first), +/// located as [`resolve_outer_allow_remote`] documents. +fn outer_config_layers( + env: &NpmConfigEnv, + read: impl Fn(&std::path::Path) -> Option, +) -> [OuterLayer; 3] { use std::path::{Path, PathBuf}; let home = env.home.as_deref(); // The directory holding node (Windows) / its `bin` parent (Unix). @@ -517,7 +568,7 @@ pub fn resolve_outer_allow_remote( } }); let builtin_text = builtin_path.as_deref().and_then(&read); - let env_path = |key: &str| env.npm_config(key).map(|(_, v)| env.config_path(&v)); + let env_path = |key: &str| env.npm_config(key, "all").map(|(_, v)| env.config_path(&v)); let file_value = |text: &Option, key: &str| { text.as_deref() .and_then(|t| npmrc_top_level_value(t, key)) @@ -538,24 +589,100 @@ pub fn resolve_outer_allow_remote( .map(|prefix| prefix.join("etc").join("npmrc")) }); let global_text = global_path.as_deref().and_then(&read); - let file = [ + [ ("user", user_path, user_text), ("global", global_path, global_text), ("builtin", builtin_path, builtin_text), ] - .into_iter() - .find_map(|(layer, path, text)| { - let value = npmrc_allow_remote(text.as_deref()?)?; +} + +/// The highest-precedence top-level `key` assignment among `layers`. +fn outer_file_value(layers: &[OuterLayer], key: &str) -> Option { + layers.iter().find_map(|(layer, path, text)| { + let value = npmrc_top_level_value(text.as_deref()?, key)?; Some(OuterFileValue { layer, - path: path?, + path: path.clone()?, value, }) - }); - OuterAllowRemote { - env: env.npm_config("allow-remote"), - file, + }) +} + +/// npm's (>= 8) `replace-registry-host` config key. npm rewrites the origin +/// of a lock's `resolved` URL to the configured registry when the URL's +/// hostname equals this value (`npmjs`, the default, means +/// `registry.npmjs.org`; `never` matches nothing) — or for EVERY origin +/// under `always` (`@npmcli/arborist` `#registryResolved`). A hosted pin +/// rewritten that way is fetched from the registry, which 404s. +pub const REPLACE_REGISTRY_HOST: &str = "replace-registry-host"; + +/// Where the effective `replace-registry-host` value comes from. +#[derive(Debug, Clone, PartialEq)] +pub enum SettingSource { + /// An `npm_config_*` environment variable (named). + Env(String), + /// The project `.npmrc`. + Project, + /// A user / global / builtin config file. + File { + layer: &'static str, + path: std::path::PathBuf, + }, +} + +/// The `replace-registry-host` value npm would use, following its layer +/// order (env > project > user > global > builtin), or `None` when no +/// layer sets it (npm's default `npmjs`). +pub fn effective_replace_registry_host( + project: Option<&str>, + outer: &OuterAllowRemote, +) -> Option<(String, SettingSource)> { + let outer = &outer.replace_registry_host; + if let Some((var, value)) = &outer.env { + return Some((value.clone(), SettingSource::Env(var.clone()))); + } + if let Some(value) = project.and_then(|t| npmrc_top_level_value(t, REPLACE_REGISTRY_HOST)) { + return Some((value, SettingSource::Project)); + } + outer.file.as_ref().map(|f| { + ( + f.value.clone(), + SettingSource::File { + layer: f.layer, + path: f.path.clone(), + }, + ) + }) +} + +/// Whether npm's `replace-registry-host=` rewrites a `resolved` URL +/// whose authority is `host` (`host[:port]`, as [`url_host`] returns it): +/// `always`, or the URL's hostname (port dropped) equal to the value +/// (`npmjs` standing for `registry.npmjs.org`). npm compares the raw value +/// against the URL's lowercased hostname, so an uppercase value matches +/// nothing. +/// +/// [`url_host`]: crate::hosted::guidance::url_host +pub fn replace_registry_host_rewrites(value: &str, host: &str) -> bool { + let value = value.trim_matches(is_js_ws); + if value == "always" { + return true; } + // Drop a `:port` suffix (never inside an `[ipv6]` literal). + let hostname = match host.rfind(':') { + Some(i) + if !host[i..].contains(']') && host[i + 1..].bytes().all(|b| b.is_ascii_digit()) => + { + &host[..i] + } + _ => host, + }; + let target = if value == "npmjs" { + "registry.npmjs.org" + } else { + value + }; + hostname.to_ascii_lowercase() == target } /// Decide how to ensure `allow-remote=all` in the project `.npmrc`, @@ -1135,4 +1262,78 @@ mod tests { ); } } + + /// #812: `replace-registry-host` is read from every npm config layer in + /// npm's order (env > project > user > global > builtin), and matched + /// against a pinned origin the way `@npmcli/arborist` does. + #[test] + fn replace_registry_host_layers_and_matching() { + use std::collections::HashMap; + use std::path::{Path, PathBuf}; + let files: HashMap = HashMap::from([ + ( + PathBuf::from("/home/u/.npmrc"), + "replace-registry-host=always\n", + ), + ( + PathBuf::from("/opt/node/etc/npmrc"), + "replace-registry-host=never\n", + ), + ]); + let read = |p: &Path| files.get(p).map(|s| s.to_string()); + + // Nothing set anywhere: npm's default (`npmjs`) applies. + let outer = resolve_outer_allow_remote(&cfg_env(&[]), |_| None); + assert_eq!(effective_replace_registry_host(None, &outer), None); + + // The user file beats the global one. + let outer = resolve_outer_allow_remote(&cfg_env(&[]), read); + assert_eq!( + effective_replace_registry_host(Some("fund=false\n"), &outer), + Some(( + "always".into(), + SettingSource::File { + layer: "user", + path: "/home/u/.npmrc".into() + } + )) + ); + // ...the project file beats both... + assert_eq!( + effective_replace_registry_host(Some("replace-registry-host=npmjs\n"), &outer), + Some(("npmjs".into(), SettingSource::Project)) + ); + // ...and the env beats every file, in any spelling npm normalizes. + for var in [ + "npm_config_replace_registry_host", + "NPM_CONFIG_REPLACE_REGISTRY_HOST", + ] { + let outer = resolve_outer_allow_remote(&cfg_env(&[(var, "always")]), read); + assert_eq!( + effective_replace_registry_host(Some("replace-registry-host=never\n"), &outer), + Some(("always".into(), SettingSource::Env(var.into()))) + ); + } + // allow-remote resolution is unaffected by the new key. + assert_eq!((outer.env, outer.file), (None, None)); + + let host = "patch.socket.dev"; + assert!(replace_registry_host_rewrites("always", host)); + assert!(replace_registry_host_rewrites(host, host)); + assert!(replace_registry_host_rewrites( + "127.0.0.1", + "127.0.0.1:8765" + )); + assert!(replace_registry_host_rewrites( + "npmjs", + "registry.npmjs.org" + )); + assert!(!replace_registry_host_rewrites("npmjs", host)); + assert!(!replace_registry_host_rewrites("never", host)); + assert!(!replace_registry_host_rewrites("registry.example", host)); + assert!(!replace_registry_host_rewrites("Always", host)); + assert!(!replace_registry_host_rewrites("PATCH.SOCKET.DEV", host)); + assert!(replace_registry_host_rewrites("[::1]", "[::1]:80")); + assert!(replace_registry_host_rewrites("[::1]", "[::1]")); + } } diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/mod.rs b/crates/socket-patch-core/src/patch/redirect/upstream/mod.rs index 9dd2ad713..a6bf9b577 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/mod.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/mod.rs @@ -82,9 +82,12 @@ impl HostedPin { .collect() } - /// Every hosted pin a discovery holds. + /// Every hosted pin a discovery holds: its refs, plus the refs it + /// withholds from attestation only because an unreachable unpatched + /// copy installs beside them ([`Discovery::shadowed`], #828) — that + /// wiring is still one package version's pin, so it is restorable. pub fn all(discovery: &Discovery) -> Vec { - Self::from_refs(&discovery.refs) + Self::from_refs(discovery.refs.iter().chain(&discovery.shadowed)) } /// THE "is this patch pinned" answer: the attributable hosted pins @@ -181,6 +184,7 @@ impl HostedInventory { let urls = discovery .refs .iter() + .chain(&discovery.shadowed) .filter(|r| r.mode == WiringMode::Hosted) .filter_map(|r| Some((r.url.as_deref()?, r.uuid.as_str()))) .chain( diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs index 4b95c7c07..fae12f96a 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs @@ -14,7 +14,7 @@ use serde_json::Value; use super::client::NpmDist; use super::{Ctx, FormatResult, HostedPin, View}; -use crate::vendor::lock_inventory::npm_legacy_identity; +use crate::vendor::lock_inventory::{npm_lock_entries, NpmLockEntry}; /// The pins by uuid. pub(super) fn by_uuid<'p>(pins: &[&'p HostedPin]) -> BTreeMap<&'p str, &'p HostedPin> { @@ -140,86 +140,25 @@ struct NpmHit { version: String, } +/// Every hosted entry of an npm lock: the installed `packages` entries +/// (never the root or a workspace member) and every node of the legacy +/// `dependencies` tree, an alias node restoring its target's registry dist +/// (#432). fn npm_lock_hits(lock: &Value, ctx: &Ctx<'_>) -> Vec { - let mut hits = Vec::new(); - if let Some(packages) = lock.get("packages").and_then(Value::as_object) { - for (key, entry) in packages { - let Some((_, key_name)) = key.rsplit_once("node_modules/") else { - continue; - }; - let Some(uuid) = entry - .get("resolved") - .and_then(Value::as_str) - .and_then(|u| ctx.hosted_uuid(u)) - else { - continue; - }; - let name = entry - .get("name") - .and_then(Value::as_str) - .unwrap_or(key_name) - .to_string(); - let Some(version) = entry.get("version").and_then(Value::as_str) else { - continue; - }; - hits.push(NpmHit { - pointer: format!("/packages/{}", json_pointer_escape(key)), - uuid, - name, - version: version.to_string(), - }); - } - } - if let Some(deps) = lock.get("dependencies").and_then(Value::as_object) { - v2_hits(deps, "/dependencies", ctx, &mut hits, 0); - } - hits -} - -fn v2_hits( - deps: &serde_json::Map, - prefix: &str, - ctx: &Ctx<'_>, - hits: &mut Vec, - depth: usize, -) { - if depth > 64 { - return; - } - for (name, entry) in deps { - let pointer = format!("{prefix}/{}", json_pointer_escape(name)); - // An alias node (`"lp": {"version": "npm:left-pad@1.3.0"}`) restores - // its target's registry dist (#432). - let (node_name, node_version) = - npm_legacy_identity(name, entry.get("version").and_then(Value::as_str)); - if let (Some(uuid), Some(version)) = ( - entry - .get("resolved") - .and_then(Value::as_str) - .and_then(|u| ctx.hosted_uuid(u)), - node_version, - ) { - hits.push(NpmHit { - pointer: pointer.clone(), + npm_lock_entries(lock) + .into_iter() + .filter(NpmLockEntry::is_dependency) + .filter_map(|entry| { + let uuid = entry.node.resolved.and_then(|u| ctx.hosted_uuid(u))?; + let version = entry.node.version?; + Some(NpmHit { + pointer: entry.pointer, uuid, - name: node_name.to_string(), + name: entry.node.name.to_string(), version: version.to_string(), - }); - } - if let Some(nested) = entry.get("dependencies").and_then(Value::as_object) { - v2_hits( - nested, - &format!("{pointer}/dependencies"), - ctx, - hits, - depth + 1, - ); - } - } -} - -fn json_pointer_escape(key: &str) -> String { - key.replace('~', "~0").replace('/', "~1") + }) + }) + .collect() } pub(crate) async fn restore_npm_locks( diff --git a/crates/socket-patch-core/src/vendor/bun_binary.rs b/crates/socket-patch-core/src/vendor/bun_binary.rs index 4674b3b80..236e1bf56 100644 --- a/crates/socket-patch-core/src/vendor/bun_binary.rs +++ b/crates/socket-patch-core/src/vendor/bun_binary.rs @@ -1,18 +1,16 @@ //! Native binary Bun vendoring. Package records are edited without re-resolving //! dependencies or requiring a Bun executable. use super::bun_lockb::{BinaryPackage, BunLockb}; -use super::common::{already_patched_result, refused}; +use super::common::refused; use super::npm_common::{ - done_failure_unstage, gate_packages, guard_coordinates, guard_revert_uuid_dir, refusal_code, - stage_patch_pack, tgz_rel_leaf, NpmCoords, + gate_packages, guard_revert_uuid_dir, refusal_code, tgz_rel_leaf, NpmCommit, NpmCoords, + NpmLockBackend, NpmStagedPack, WireCx, }; use super::path::parse_vendor_path; -use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, -}; +use super::state::{VendorEntry, WiringAction, WiringRecord}; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; use crate::manifest::schema::PatchRecord; +#[cfg(test)] use crate::patch::apply::PatchSources; use crate::utils::fs::{atomic_write_bytes_preserving_mode, read_regular_to_bytes_sync}; use std::path::{Path, PathBuf}; @@ -27,238 +25,203 @@ fn is_ours(package: &BinaryPackage, name: &str, leaf: &str) -> bool { && package.resolution.ends_with(&format!("/{leaf}")) } +/// [`BunBinaryBackend`] through the shared driver, under the signature the +/// suite below calls it by. +#[cfg(test)] #[allow(clippy::too_many_arguments)] pub(crate) async fn vendor( purl: &str, - installed_dir: PackageSource<'_>, + installed_dir: super::source::PackageSource<'_>, root: &Path, record: &PatchRecord, - sources: &PatchSources<'_>, + sources: &crate::patch::apply::PatchSources<'_>, vendored_at: &str, dry_run: bool, force: bool, service: Option<&super::VendorServiceConfig>, ) -> VendorOutcome { - let coords = match guard_coordinates(purl, record) { - Ok(v) => v, - Err(o) => return *o, - }; - let project = match read_project(root).await { - Ok(v) => v, - Err(o) => return *o, - }; - let leaf = tgz_rel_leaf(&coords.name, &coords.version); - let BinaryTargets { - matches, - mirrors, - bundled, - } = match preflight_package(&project, root, &coords, &leaf) { - Ok(v) => v, - Err(o) => return *o, - }; - let BinaryProject { mut lock, .. } = project; - let mut warnings = Vec::new(); - for package in bundled { - // LOUD: this copy ships inside its PARENT's tarball, which we do not - // repack — it stays the unpatched bytes after vendor (#469). - warnings.push(super::VendorWarning::new( - "vendor_bundled_instance_skipped", - format!( - "{LOCK} package #{} ({}@{}) is {}bundled inside its parent's tarball and \ - CANNOT be rewritten there — that copy stays UNPATCHED; vendor or update the \ - bundling parent to cover it", - package.id, - coords.name, - coords.version, - if package.bundled_only { "" } else { "also " }, - ), - )); - } - let preexisted = root.join(&coords.uuid_dir_rel).exists(); - let (staged, result) = match stage_patch_pack( - purl, - installed_dir, - root, - record, - sources, - dry_run, - force, - &mut warnings, - service, + super::npm_common::vendor_npm_family( + &BunBinaryBackend, + super::npm_common::NpmVendorRequest { + purl, + installed_dir, + project_root: root, + record, + sources, + vendored_at, + dry_run, + force, + service, + }, ) .await - { - Ok(v) => v, - Err(o) => return *o, - }; - let Some(staged) = staged else { - return VendorOutcome::Done { - result, - entry: None, - warnings, - }; - }; - let mut wiring = Vec::new(); - for package in matches { - if package.resolution == staged.rel_tgz - && package.integrity.as_deref() == Some(&staged.packed.integrity) - { - continue; +} + +/// The `bun.lockb` half of [`super::bun_lock::vendor_bun`], for a project +/// whose installs the binary lock drives (driven by +/// [`super::npm_common::vendor_npm_family`]). +pub(super) struct BunBinaryBackend; + +/// [`BunBinaryBackend`]'s pre-flight product: the parsed lock and the +/// records and workspace mirrors to rewrite. +pub(super) struct BunBinaryPlan { + lock: BunLockb, + leaf: String, + matches: Vec, + mirrors: Vec<(String, String)>, +} + +impl NpmLockBackend for BunBinaryBackend { + type Plan = BunBinaryPlan; + + fn flavor(&self) -> Option<&'static str> { + Some("bun") + } + + async fn preflight( + &self, + root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result> { + let project = read_project(root).await?; + let leaf = tgz_rel_leaf(&coords.name, &coords.version); + let BinaryTargets { + matches, + mirrors, + bundled, + } = preflight_package(&project, root, coords, &leaf)?; + for package in bundled { + // LOUD: this copy ships inside its PARENT's tarball, which we do + // not repack — it stays the unpatched bytes after vendor (#469). + warnings.push(VendorWarning::new( + "vendor_bundled_instance_skipped", + format!( + "{LOCK} package #{} ({}@{}) is {}bundled inside its parent's tarball and \ + CANNOT be rewritten there — that copy stays UNPATCHED; vendor or update the \ + bundling parent to cover it", + package.id, + coords.name, + coords.version, + if package.bundled_only { "" } else { "also " }, + ), + )); } - let mutation = (|| { + let BinaryProject { lock, .. } = project; + Ok(BunBinaryPlan { + lock, + leaf, + matches, + mirrors, + }) + } + + async fn wire( + &self, + plan: BunBinaryPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + _warnings: &mut Vec, + ) -> Result, String> { + let BunBinaryPlan { + mut lock, + leaf, + matches, + mirrors, + } = plan; + let (root, coords) = (cx.project_root, cx.coords); + let mut wiring = Vec::new(); + for package in matches { + if package.resolution == staged.rel_tgz + && package.integrity.as_deref() == Some(&staged.packed.integrity) + { + continue; + } let original = lock.snapshot(package.id)?; lock.set_package(package.id, &staged.rel_tgz, &staged.packed.integrity)?; - Ok::<_, String>((original, lock.snapshot(package.id)?)) - })(); - let (original, mut new) = match mutation { - Ok(v) => v, - Err(e) => { - return done_failure_unstage(purl, e, root, &coords.uuid_dir_rel, preexisted).await + let mut new = lock.snapshot(package.id)?; + if is_ours(&package, &coords.name, &leaf) { + // Bun may renumber packages on re-save. Preserve the semantic + // predecessor so ledger carry-forward can recover the correct + // pristine original even after the numeric key has changed. + new["previous"] = serde_json::json!({ + "name": original["name"], "version": original["version"], + "resolution": original["resolution"], "integrity": original["integrity"], + }); } - }; - if is_ours(&package, &coords.name, &leaf) { - // Bun may renumber packages on re-save. Preserve the semantic - // predecessor so ledger carry-forward can recover the correct - // pristine original even after the numeric key has changed. - new["previous"] = serde_json::json!({ - "name": original["name"], "version": original["version"], - "resolution": original["resolution"], "integrity": original["integrity"], + wiring.push(WiringRecord { + file: LOCK.into(), + kind: KIND.into(), + action: WiringAction::Rewritten, + key: Some(package.id.to_string()), + original: if is_ours(&package, &coords.name, &leaf) { + None + } else { + Some(original) + }, + new: Some(new), }); } - wiring.push(WiringRecord { - file: LOCK.into(), - kind: KIND.into(), - action: WiringAction::Rewritten, - key: Some(package.id.to_string()), - original: if is_ours(&package, &coords.name, &leaf) { - None - } else { - Some(original) - }, - new: Some(new), - }); - } - // Bun 0.5.9–1.3 resolves workspace local tarballs relative to the - // declaring member. Preserve the same portable resolution for every - // consumer by committing identical tarballs at those member-relative - // locations as well as the canonical root artifact. - let artifact = match read_regular_to_bytes_sync(&root.join(&staged.rel_tgz)) { - Ok(bytes) => bytes, - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot read staged tarball: {e}"), - root, - &coords.uuid_dir_rel, - preexisted, - ) - .await - } - }; - let lock_changed = !wiring.is_empty(); - let mut mirror_backups: Vec<(PathBuf, Option>)> = Vec::new(); - for (workspace, rel) in &mirrors { - let path = root.join(rel); - let before = match read_regular_to_bytes_sync(&path) { - Ok(bytes) => Some(bytes), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, - Err(e) => { + // Bun 0.5.9–1.3 resolves workspace local tarballs relative to the + // declaring member. Preserve the same portable resolution for every + // consumer by committing identical tarballs at those member-relative + // locations as well as the canonical root artifact. + let artifact = read_regular_to_bytes_sync(&root.join(&staged.rel_tgz)) + .map_err(|e| format!("cannot read staged tarball: {e}"))?; + let lock_changed = !wiring.is_empty(); + let mut mirror_backups: Vec<(PathBuf, Option>)> = Vec::new(); + for (workspace, rel) in &mirrors { + let path = root.join(rel); + let before = match read_regular_to_bytes_sync(&path) { + Ok(bytes) => Some(bytes), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => { + undo_mirrors(&mirror_backups).await; + return Err(format!("cannot read workspace tarball {rel}: {e}")); + } + }; + wiring.push(WiringRecord { + file: rel.clone(), + kind: MIRROR_KIND.into(), + action: WiringAction::Added, + key: Some(workspace.clone()), + original: None, + new: Some(serde_json::Value::String(staged.packed.sha256_hex.clone())), + }); + if before.as_deref() == Some(artifact.as_slice()) { + continue; + } + if let Err(e) = tokio::fs::create_dir_all(path.parent().expect("mirror parent")).await { undo_mirrors(&mirror_backups).await; - return done_failure_unstage( - purl, - format!("cannot read workspace tarball {rel}: {e}"), - root, - &coords.uuid_dir_rel, - preexisted, - ) - .await; + return Err(format!("cannot create workspace vendor directory: {e}")); } - }; - wiring.push(WiringRecord { - file: rel.clone(), - kind: MIRROR_KIND.into(), - action: WiringAction::Added, - key: Some(workspace.clone()), - original: None, - new: Some(serde_json::Value::String(staged.packed.sha256_hex.clone())), - }); - if before.as_deref() == Some(artifact.as_slice()) { - continue; + if let Err(e) = atomic_write_bytes_preserving_mode(&path, &artifact).await { + undo_mirrors(&mirror_backups).await; + return Err(format!("cannot write workspace tarball {rel}: {e}")); + } + mirror_backups.push((path, before)); } - if let Err(e) = tokio::fs::create_dir_all(path.parent().expect("mirror parent")).await { - undo_mirrors(&mirror_backups).await; - return done_failure_unstage( - purl, - format!("cannot create workspace vendor directory: {e}"), - root, - &coords.uuid_dir_rel, - preexisted, - ) - .await; + if !lock_changed && mirror_backups.is_empty() { + return Ok(None); } - if let Err(e) = atomic_write_bytes_preserving_mode(&path, &artifact).await { + if let Err(e) = atomic_write_bytes_preserving_mode(&root.join(LOCK), &lock.bytes()).await { undo_mirrors(&mirror_backups).await; - return done_failure_unstage( - purl, - format!("cannot write workspace tarball {rel}: {e}"), - root, - &coords.uuid_dir_rel, - preexisted, - ) - .await; + return Err(format!("cannot write {LOCK}: {e}")); } - mirror_backups.push((path, before)); - } - if !lock_changed && mirror_backups.is_empty() { - return VendorOutcome::Done { - result: already_patched_result(purl, &root.join(&staged.rel_tgz), &record.files), - entry: None, - warnings, - }; - } - if staged.staged_pkg_json.is_some() { - warnings.push(VendorWarning::new("vendor_dep_manifest_stale", format!("the patch changes package.json; {LOCK} dependency edges were preserved — run bun install if dependency ranges changed"))); + Ok(Some(NpmCommit { + wiring, + ..NpmCommit::default() + })) } - if let Err(e) = atomic_write_bytes_preserving_mode(&root.join(LOCK), &lock.bytes()).await { - undo_mirrors(&mirror_backups).await; - return done_failure_unstage( - purl, - format!("cannot write {LOCK}: {e}"), - root, - &coords.uuid_dir_rel, - preexisted, + + fn manifest_warning(&self, _name: &str, _version: &str) -> VendorWarning { + VendorWarning::new( + "vendor_dep_manifest_stale", + format!( + "the patch changes package.json; {LOCK} dependency edges were preserved — run \ + bun install if dependency ranges changed" + ), ) - .await; - } - let marker = VendorMarker::new("npm", &coords.base_purl, record, vendored_at); - write_marker_or_warn(&root.join(&coords.uuid_dir_rel), &marker, &mut warnings).await; - VendorOutcome::Done { - result, - warnings, - entry: Some(VendorEntry { - ecosystem: "npm".into(), - base_purl: coords.base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: staged.rel_tgz, - sha256: staged.packed.sha256_hex, - size: Some(staged.packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some("bun".into()), - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }), } } @@ -1158,4 +1121,49 @@ mod rebuild_tests { assert!(!root.join(&mirror.file).exists()); } } + + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched (and the run that wires says it once). + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = flip_fixture().await; + let patched: &[u8] = br#"{"name":"minimist","version":"1.2.2","sideEffects":false}"#; + let after_hash = compute_git_sha256_from_bytes(patched); + std::fs::write(fx.root().join(".socket/blobs").join(&after_hash), patched).unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + crate::manifest::schema::PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(PACKAGE), + after_hash, + }, + ); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + let VendorOutcome::Done { + result, + entry: Some(entry), + warnings, + } = flip_run(&fx, None).await + else { + panic!("the first run wires"); + }; + assert!(result.success, "{result:?}"); + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + ts::persist(fx.root(), PURL, entry).await; + + let VendorOutcome::Done { + result, + entry, + warnings, + } = flip_run(&fx, None).await + else { + panic!("expected Done"); + }; + assert!(result.success && entry.is_none(), "{result:?}"); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } } diff --git a/crates/socket-patch-core/src/vendor/bun_lock.rs b/crates/socket-patch-core/src/vendor/bun_lock.rs index 713ee8142..7a13741b0 100644 --- a/crates/socket-patch-core/src/vendor/bun_lock.rs +++ b/crates/socket-patch-core/src/vendor/bun_lock.rs @@ -46,16 +46,14 @@ use crate::vendor::bun_lock_text::{ split_name_spec, BunEntry, }; -use super::common::{already_patched_result, refused}; +use super::common::refused; use super::npm_common::{ - done_failure_unstage, gate_packages, guard_coordinates, guard_revert_uuid_dir, refusal_code, - stage_patch_pack, tgz_rel_leaf, + gate_packages, guard_revert_uuid_dir, refusal_code, tgz_rel_leaf, vendor_npm_family, NpmCommit, + NpmCoords, NpmLockBackend, NpmStagedPack, NpmVendorRequest, WireCx, }; use super::path::parse_vendor_path; use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, -}; +use super::state::{VendorEntry, WiringAction, WiringRecord}; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; const BUN_LOCK: &str = "bun.lock"; @@ -312,7 +310,9 @@ pub async fn binary_vendor_paths(project_root: &Path) -> Result, Str /// Vendor one installed npm package into a bun project (see the module doc). /// Same contract as `npm_lock::vendor_npm`: refuse-early / wire-last, /// `entry` present iff `result.success` and not a dry run, and an in-sync -/// re-run synthesizes AlreadyPatched with no entry. +/// re-run synthesizes AlreadyPatched with no entry. The flow is +/// [`vendor_npm_family`]'s; [`BunTextBackend`] is the text-lock grammar +/// (a project driven by `bun.lockb` routes to [`super::bun_binary`]). #[allow(clippy::too_many_arguments)] pub(crate) async fn vendor_bun<'a>( purl: &str, @@ -325,325 +325,284 @@ pub(crate) async fn vendor_bun<'a>( force: bool, service: Option<&super::VendorServiceConfig>, ) -> VendorOutcome { - let installed_dir = installed_dir.into(); - if binary_lock_drives(project_root) { - return super::bun_binary::vendor( - purl, - installed_dir, - project_root, - record, - sources, - vendored_at, - dry_run, - force, - service, - ) - .await; - } - let mut warnings: Vec = Vec::new(); - - // ── 1. Coordinates (shared fail-closed guard) ───────────────────────── - let coords = match guard_coordinates(purl, record) { - Ok(coords) => coords, - Err(outcome) => return *outcome, - }; - let (name, version) = (coords.name.as_str(), coords.version.as_str()); - - // ── 2. Read + strictly parse the lock (refuse before any write) ────── - let project = match read_project(project_root).await { - Ok(project) => project, - Err(outcome) => return *outcome, - }; - - // ── 3. Pre-flight: at least one rewritable instance ────────────────── - let (target_spec, target_leaf) = match preflight_package(&project, name, version) { - Ok(target) => target, - Err(outcome) => return *outcome, - }; - for key in bundled_matches(&project.entries, &target_spec, name, &target_leaf) { - // LOUD: this copy ships inside its PARENT's tarball, which we do not - // repack — it stays the unpatched bytes after vendor (#469). - warnings.push(VendorWarning::new( - "vendor_bundled_instance_skipped", - format!( - "{BUN_LOCK} entry `{key}` is bundled inside its parent's tarball and CANNOT be \ - rewritten — that copy stays UNPATCHED; vendor or update the bundling parent \ - to cover it" - ), - )); - } - let BunProject { - mut lines, entries, .. - } = project; - - // BN3 spelling: BARE project-relative path, no `file:`/`./` prefix (the - // shared pipeline's `prepare_tgz_dest` builds the identical string). - let rel_tgz = format!("{}/{}", coords.uuid_dir_rel, target_leaf); - // The sha512 of the artifact already sitting at the target path, if - // any — the one witness a digest-less in-sync tuple (see `classify`) - // still has of the digest Bun dropped: the lock line was written from - // these bytes. With a ledger anchor the shared pipeline reuses exactly - // these bytes (so this equals the staged integrity); without one it - // keeps today's behavior. Read BEFORE staging, which may overwrite the - // file; a missing or non-regular path (a `repair` rebuild after - // deletion, a FIFO) yields `None`, which the in-sync check below treats - // as "not provably the same bytes". Only a digest-less 2-tuple of OURS at this - // path can consume it, so every other re-run skips the read + hash. - let has_digestless_own_tuple = entries.iter().any(|e| { - e.elems.len() == 2 - && matches!( - classify_rewritable(e, &target_spec, name, &target_leaf), - Some(TupleShape::Ours { path }) if path == rel_tgz - ) - }); - let prior_artifact_integrity: Option = if has_digestless_own_tuple { - let abs = project_root.join(&coords.uuid_dir_rel).join(&target_leaf); - match tokio::fs::metadata(&abs).await { - Ok(meta) if meta.is_file() => tokio::fs::read(&abs) - .await - .ok() - .map(|bytes| crate::utils::digest::sha512_sri_of(&bytes)), - _ => None, - } - } else { - None - }; - - // ── 4. Stage → patch → pack (shared flavor-agnostic pipeline) ──────── - let (staged, result) = match stage_patch_pack( + let req = NpmVendorRequest { purl, - installed_dir, + installed_dir: installed_dir.into(), project_root, record, sources, + vendored_at, dry_run, force, - &mut warnings, service, - ) - .await - { - Ok(pair) => pair, - Err(outcome) => return *outcome, - }; - let Some(staged) = staged else { - // Failed patch or dry run: wiring never ran, project byte-untouched. - return VendorOutcome::Done { - result, - entry: None, - warnings, - }; }; - let uuid_dir_preexisted = staged.uuid_dir_preexisted; - debug_assert_eq!(staged.rel_tgz, rel_tgz); - let packed = staged.packed; - if staged.staged_pkg_json.is_some() { - // The tuple's deps object mirrors the package's own manifest; the - // spike has no fixture for a manifest-rewriting patch, so it is - // preserved verbatim rather than recomputed (fail-safe + loud). - warnings.push(VendorWarning::new( - "vendor_dep_manifest_stale", - format!( - "the patch rewrites {name}@{version}'s package.json; its {BUN_LOCK} tuple's \ - dependency object was preserved verbatim — if the patch changed dependency \ - ranges, run `bun install` to re-resolve them" - ), - )); + if binary_lock_drives(project_root) { + return vendor_npm_family(&super::bun_binary::BunBinaryBackend, req).await; } + vendor_npm_family(&BunTextBackend, req).await +} - // ── 5. Rewrite every matching instance (in-memory) ──────────────────── - let mut wiring: Vec = Vec::new(); - let mut changed = false; - // In-sync instances whose digest Bun dropped (see `classify`): re-pinned - // on disk WITHOUT a wiring record — the ledger already holds this - // instance's pristine original and `revert_one_record` recognises both - // spellings — so the run stays an AlreadyPatched no-op for the ledger - // while ≥ 1.3.10 consumers of the committed lock regain verification. - let mut healed = false; - for entry in &entries { - let Some(shape) = classify_rewritable(entry, &target_spec, name, &target_leaf) else { - continue; - }; - let original_line = lines[entry.line_idx].clone(); - // Lines come from a bare `split('\n')`, so a CRLF lock's lines carry - // a trailing `\r` (the grammar trims it away when parsing). Re-emit - // it verbatim: the surgery must never mix line endings. - let cr = if original_line.ends_with('\r') { - "\r" +/// The text-lock (`bun.lock`) half of [`vendor_bun`]. +struct BunTextBackend; + +/// [`BunTextBackend`]'s pre-flight product: the lock's lines and parsed +/// entries, the target, and the digest witness read before staging. +struct BunTextPlan { + lines: Vec, + entries: Vec, + target_spec: String, + target_leaf: String, + /// The sha512 of the artifact at the target path before staging (see + /// [`BunTextBackend::preflight`]). + prior_artifact_integrity: Option, +} + +impl NpmLockBackend for BunTextBackend { + type Plan = BunTextPlan; + + fn flavor(&self) -> Option<&'static str> { + Some("bun") + } + + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result> { + let (name, version) = (coords.name.as_str(), coords.version.as_str()); + + // ── 2. Read + strictly parse the lock (refuse before any write) ── + let project = read_project(project_root).await?; + + // ── 3. Pre-flight: at least one rewritable instance ────────────── + let (target_spec, target_leaf) = preflight_package(&project, name, version)?; + for key in bundled_matches(&project.entries, &target_spec, name, &target_leaf) { + // LOUD: this copy ships inside its PARENT's tarball, which we do + // not repack — it stays the unpatched bytes after vendor (#469). + warnings.push(VendorWarning::new( + "vendor_bundled_instance_skipped", + format!( + "{BUN_LOCK} entry `{key}` is bundled inside its parent's tarball and CANNOT \ + be rewritten — that copy stays UNPATCHED; vendor or update the bundling \ + parent to cover it" + ), + )); + } + let BunProject { lines, entries, .. } = project; + + // BN3 spelling: BARE project-relative path, no `file:`/`./` prefix + // (the shared pipeline's `prepare_tgz_dest` builds the identical + // string). + let rel_tgz = format!("{}/{}", coords.uuid_dir_rel, target_leaf); + // The sha512 of the artifact already sitting at the target path, if + // any — the one witness a digest-less in-sync tuple (see `classify`) + // still has of the digest Bun dropped: the lock line was written from + // these bytes. With a ledger anchor the shared pipeline reuses + // exactly these bytes (so this equals the staged integrity); without + // one it keeps today's behavior. Read BEFORE staging, which may + // overwrite the file; a missing or non-regular path (a `repair` + // rebuild after deletion, a FIFO) yields `None`, which the in-sync + // check treats as "not provably the same bytes". Only a digest-less + // 2-tuple of OURS at this path can consume it, so every other re-run + // skips the read + hash. + let has_digestless_own_tuple = entries.iter().any(|e| { + e.elems.len() == 2 + && matches!( + classify_rewritable(e, &target_spec, name, &target_leaf), + Some(TupleShape::Ours { path }) if path == rel_tgz + ) + }); + let prior_artifact_integrity: Option = if has_digestless_own_tuple { + let abs = project_root.join(&coords.uuid_dir_rel).join(&target_leaf); + match tokio::fs::metadata(&abs).await { + Ok(meta) if meta.is_file() => tokio::fs::read(&abs) + .await + .ok() + .map(|bytes| crate::utils::digest::sha512_sri_of(&bytes)), + _ => None, + } } else { - "" - }; - let local_tuple_line = |deps: &str| { - format!( - "{indent}{key}: [\"{name}@{rel_tgz}\", {deps}, \"{integrity}\"]{comma}{cr}", - indent = entry.indent, - key = entry.key_raw, - integrity = packed.integrity, - comma = if entry.trailing_comma { "," } else { "" }, - ) + None }; - let (deps_verbatim, was_ours) = match shape { - TupleShape::Registry => (entry.elems[2].clone(), false), - TupleShape::Ours { path } => { - if path == rel_tgz { - match entry.elems.get(2) { - // Idempotency: an instance already carrying this exact - // path and integrity needs no edit and no wiring record. - Some(integrity) if *integrity == format!("\"{}\"", packed.integrity) => { - continue; - } - // Digest-less re-save of THIS wiring (Bun 1.1.39–1.3.9) - // over the SAME bytes the lock was written from (the - // artifact found at the path before this run re-staged - // it equals the staged one): heal the line in place, - // record nothing — the ledger's fingerprint still holds. - None if prior_artifact_integrity.as_deref() - == Some(packed.integrity.as_str()) => - { - lines[entry.line_idx] = local_tuple_line(&entry.elems[1]); - healed = true; - continue; + Ok(BunTextPlan { + lines, + entries, + target_spec, + target_leaf, + prior_artifact_integrity, + }) + } + + async fn wire( + &self, + plan: BunTextPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + _warnings: &mut Vec, + ) -> Result, String> { + let BunTextPlan { + mut lines, + entries, + target_spec, + target_leaf, + prior_artifact_integrity, + } = plan; + let name = cx.coords.name.as_str(); + let project_root = cx.project_root; + let rel_tgz = staged.rel_tgz.as_str(); + let packed = &staged.packed; + + // ── 5. Rewrite every matching instance (in-memory) ──────────────── + let mut wiring: Vec = Vec::new(); + let mut changed = false; + // In-sync instances whose digest Bun dropped (see `classify`): + // re-pinned on disk WITHOUT a wiring record — the ledger already + // holds this instance's pristine original and `revert_one_record` + // recognises both spellings — so the run stays an AlreadyPatched + // no-op for the ledger while ≥ 1.3.10 consumers of the committed + // lock regain verification. + let mut healed = false; + for entry in &entries { + let Some(shape) = classify_rewritable(entry, &target_spec, name, &target_leaf) else { + continue; + }; + let original_line = lines[entry.line_idx].clone(); + // Lines come from a bare `split('\n')`, so a CRLF lock's lines + // carry a trailing `\r` (the grammar trims it away when + // parsing). Re-emit it verbatim: the surgery must never mix line + // endings. + let cr = if original_line.ends_with('\r') { + "\r" + } else { + "" + }; + let local_tuple_line = |deps: &str| { + format!( + "{indent}{key}: [\"{name}@{rel_tgz}\", {deps}, \"{integrity}\"]{comma}{cr}", + indent = entry.indent, + key = entry.key_raw, + integrity = packed.integrity, + comma = if entry.trailing_comma { "," } else { "" }, + ) + }; + let (deps_verbatim, was_ours) = match shape { + TupleShape::Registry => (entry.elems[2].clone(), false), + TupleShape::Ours { path } => { + if path == rel_tgz { + match entry.elems.get(2) { + // Idempotency: an instance already carrying this + // exact path and integrity needs no edit and no + // wiring record. + Some(integrity) + if *integrity == format!("\"{}\"", packed.integrity) => + { + continue; + } + // Digest-less re-save of THIS wiring (Bun + // 1.1.39–1.3.9) over the SAME bytes the lock was + // written from (the artifact found at the path + // before this run re-staged it equals the staged + // one): heal the line in place, record nothing — + // the ledger's fingerprint still holds. + None if prior_artifact_integrity.as_deref() + == Some(packed.integrity.as_str()) => + { + lines[entry.line_idx] = local_tuple_line(&entry.elems[1]); + healed = true; + continue; + } + // Same path, different digest — or a digest-less + // line whose artifact was missing or differed + // before staging. A source flip does not reach + // here when the ledger verifies the committed + // tarball (it is reused, so the digests agree); + // only a missing, corrupt or unanchored artifact + // (a `repair` rebuild, a lost state.json) is + // re-pinned below like any stale tuple of ours, + // so the returned entry carries the rebuilt + // artifact's fingerprint (`carry_forward_wiring` + // refills the pristine original from the entry it + // replaces). + _ => {} } - // Same path, different digest — or a digest-less line - // whose artifact was missing or differed before staging. - // A source flip does not reach here when the ledger - // verifies the committed tarball (it is reused, so the - // digests agree); only a missing, corrupt or unanchored - // artifact (a `repair` rebuild, a lost state.json) is - // re-pinned below like any stale tuple of ours, so the - // returned entry carries the rebuilt artifact's - // fingerprint (`carry_forward_wiring` refills the - // pristine original from the entry it replaces). - _ => {} } + (entry.elems[1].clone(), true) } - (entry.elems[1].clone(), true) - } - }; - let new_line = local_tuple_line(&deps_verbatim); - lines[entry.line_idx] = new_line.clone(); - wiring.push(WiringRecord { - file: BUN_LOCK.to_string(), - kind: KIND_LOCK_PACKAGE.to_string(), - action: WiringAction::Rewritten, - key: Some(entry.key.clone()), - // Never record one of our own (stale) edits as the "original" — - // revert must restore the pre-vendor registry tuple, not a - // dangling `.socket/vendor/` pointer from an earlier uuid. - original: if was_ours { - None + }; + let new_line = local_tuple_line(&deps_verbatim); + lines[entry.line_idx] = new_line.clone(); + wiring.push(WiringRecord { + file: BUN_LOCK.to_string(), + kind: KIND_LOCK_PACKAGE.to_string(), + action: WiringAction::Rewritten, + key: Some(entry.key.clone()), + // Never record one of our own (stale) edits as the + // "original" — revert must restore the pre-vendor registry + // tuple, not a dangling `.socket/vendor/` pointer from an + // earlier uuid. + original: if was_ours { + None + } else { + Some(Value::String(original_line)) + }, + new: Some(Value::String(new_line)), + }); + changed = true; + } + + // A workspace lock Bun 1.4 migrated from a vendored `bun.lockb` keeps + // the member paths the binary normalization wrote as inter-workspace + // literals, so Bun re-resolves the workspace and drops the vendored + // tuples (#803). Restore each manifest's `workspace:` literal (Bun's + // own spelling) like the digest heal above: in place, with no wiring + // record, since a revert has no reason to put the path back. + let literal_heals = super::bun_lock_text::heal_workspace_literals(&mut lines, |dir| { + let rel = if dir.is_empty() { + "package.json".to_string() } else { - Some(Value::String(original_line)) - }, - new: Some(Value::String(new_line)), + format!("{dir}/package.json") + }; + crate::utils::fs::read_regular_to_bytes_sync(&project_root.join(rel)) + .ok() + .and_then(|bytes| String::from_utf8(bytes).ok()) }); - changed = true; - } - - // A workspace lock Bun 1.4 migrated from a vendored `bun.lockb` keeps - // the member paths the binary normalization wrote as inter-workspace - // literals, so Bun re-resolves the workspace and drops the vendored - // tuples (#803). Restore each manifest's `workspace:` literal (Bun's own - // spelling) like the digest heal above: in place, with no wiring - // record, since a revert has no reason to put the path back. - let literal_heals = super::bun_lock_text::heal_workspace_literals(&mut lines, |dir| { - let rel = if dir.is_empty() { - "package.json".to_string() - } else { - format!("{dir}/package.json") - }; - crate::utils::fs::read_regular_to_bytes_sync(&project_root.join(rel)) - .ok() - .and_then(|bytes| String::from_utf8(bytes).ok()) - }); - healed |= !literal_heals.is_empty(); - - if !changed { - // Every instance already points at this uuid with the packed - // integrity (or with the digest Bun dropped, now re-pinned): in - // sync. The integrity is that of the reused committed tarball (or of - // a fresh acquisition that reproduced it when reuse missed); - // synthesize AlreadyPatched and record nothing. The - // heal is the one write of an in-sync run, and a failed write - // leaves the still-installable digest-less lock — reported as the - // failure it is, like every other lock write below. - if healed { - if let Err(e) = atomic_write_bytes_preserving_mode( + healed |= !literal_heals.is_empty(); + + // In sync (every instance already points at this uuid with the + // packed integrity, or with the digest Bun dropped, now re-pinned): + // the heal is the one write of such a run, and a failed write leaves + // the still-installable digest-less lock — reported as the failure + // it is, like every other lock write. + if changed || healed { + atomic_write_bytes_preserving_mode( &project_root.join(BUN_LOCK), lines.join("\n").as_bytes(), ) .await - { - return done_failure_unstage( - purl, - format!("cannot write {BUN_LOCK}: {e}"), - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, - ) - .await; - } + .map_err(|e| format!("cannot write {BUN_LOCK}: {e}"))?; } - return VendorOutcome::Done { - result: already_patched_result(purl, &project_root.join(&rel_tgz), &record.files), - entry: None, - warnings, - }; + if !changed { + return Ok(None); + } + Ok(Some(NpmCommit { + wiring, + ..NpmCommit::default() + })) } - if let Err(e) = atomic_write_bytes_preserving_mode( - &project_root.join(BUN_LOCK), - lines.join("\n").as_bytes(), - ) - .await - { - return done_failure_unstage( - purl, - format!("cannot write {BUN_LOCK}: {e}"), - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning { + // The tuple's deps object mirrors the package's own manifest; the + // spike has no fixture for a manifest-rewriting patch, so it is + // preserved verbatim rather than recomputed (fail-safe + loud). + VendorWarning::new( + "vendor_dep_manifest_stale", + format!( + "the patch rewrites {name}@{version}'s package.json; its {BUN_LOCK} tuple's \ + dependency object was preserved verbatim — if the patch changed dependency \ + ranges, run `bun install` to re-resolve them" + ), ) - .await; - } - - // ── 6. Marker + ledger entry ────────────────────────────────────────── - let marker = VendorMarker::new("npm", &coords.base_purl, record, vendored_at); - write_marker_or_warn( - &project_root.join(&coords.uuid_dir_rel), - &marker, - &mut warnings, - ) - .await; - - let entry = VendorEntry { - ecosystem: "npm".to_string(), - base_purl: coords.base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: rel_tgz, - sha256: packed.sha256_hex, - size: Some(packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some("bun".to_string()), - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }; - VendorOutcome::Done { - result, - entry: Some(entry), - warnings, } } @@ -2368,6 +2327,39 @@ mod tests { ); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture_with(BN3_BEFORE_LOCK, "node_modules/left-pad").await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = br#"{"name":"left-pad","version":"1.3.0","sideEffects":false}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + // ── lockfileVersion 0 + workspace locks: real per-version grammar ───── // // Provenance (real `bun install --save-text-lockfile` output on a root + diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs b/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs index e5d88861a..5abd61dee 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs @@ -70,8 +70,8 @@ pub(crate) mod wired; pub(crate) mod yarn; pub(crate) use self::npm::{ - npm_legacy_identity, npm_lock_bundled_nodes, npm_lock_legacy_mirror_nodes, - npm_lock_located_nodes, NpmLockNode, + npm_lock_bundled_nodes, npm_lock_entries, npm_lock_legacy_mirror_nodes, npm_lock_located_nodes, + NpmLockEntry, NpmLockNode, NpmLockSection, }; #[cfg(test)] pub(crate) use self::npm_family::inventory_npm_lock; diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/npm.rs b/crates/socket-patch-core/src/vendor/lock_inventory/npm.rs index 0867fb42a..61f281365 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/npm.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/npm.rs @@ -1,13 +1,17 @@ -//! `package-lock.json` / `npm-shrinkwrap.json`: the shared entry walk +//! `package-lock.json` / `npm-shrinkwrap.json`: the one addressed entry +//! walk ([`npm_lock_entries`]), the install view built on it //! ([`npm_lock_nodes`]) and its registry view. #[cfg(test)] use std::path::Path; +use std::borrow::Cow; + use serde_json::Value; use crate::constants::npm_family::NPM_LOCKS; use crate::utils::digest::is_sri_pin; +use crate::vendor::npm_origin::legacy_packages_key; use crate::vendor::path::parse_vendor_path; use super::view::ProjectView; @@ -28,26 +32,206 @@ pub(crate) struct NpmLockNode<'a> { } /// Bound on the legacy `dependencies` tree depth — the lock is tamper-able -/// input and the walk recurses (real trees are a handful of levels deep). +/// input and the walk recurses (real trees are a handful of levels deep; +/// serde_json's 128-level parse limit already caps a parsed lock near 62). const MAX_LEGACY_NPM_DEPTH: usize = 64; -/// Every install entry of a parsed npm lock, the ONE walk the inventory and -/// lockfile discovery (`vex::discover::npm`) share: +/// The `packages` key segment that marks an installed dependency. +const NODE_MODULES_SEG: &str = "node_modules/"; + +/// Which half of an npm lock an [`NpmLockEntry`] lives in. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum NpmLockSection { + /// The lockfileVersion 2/3 `packages` map. + Packages, + /// The legacy `dependencies` tree: the install tree of a + /// lockfileVersion 1 lock, the npm 6 mirror of a v2 one. + Legacy, +} + +/// One entry of a parsed `package-lock.json` / `npm-shrinkwrap.json`, with +/// its address in the document. [`npm_lock_entries`] yields every entry, +/// unfiltered; each reader and rewriter applies its own skip policy over +/// the flags and edits the entry through `Value::pointer_mut(pointer)`. +pub(crate) struct NpmLockEntry<'a> { + pub(crate) section: NpmLockSection, + /// The map key: the `packages` key, or the legacy dependency name + /// (the ALIAS for an alias node). + pub(crate) key: &'a str, + /// RFC 6901 pointer to the entry (`/packages/`, or + /// `/dependencies//dependencies/` for a legacy node). + pub(crate) pointer: String, + /// The `packages` key npm gives the install: the key itself, or the + /// legacy node's derived one ([`legacy_packages_key`]). + pub(crate) packages_key: Cow<'a, str>, + /// Where diagnostics place the entry: the `packages` key, or the + /// `>`-joined chain of legacy dependency names. + pub(crate) location: Cow<'a, str>, + /// The entry's JSON value (any type; a non-object has no fields). + pub(crate) value: &'a Value, + /// The package and version the entry installs. + pub(crate) node: NpmLockNode<'a>, + /// `link: true` (a `packages` entry npm links from a source dir). + pub(crate) link: bool, + /// `inBundle: true` in `packages`, `bundled: true` in the legacy tree: + /// npm unpacks it from the parent package's own tarball. + pub(crate) bundled: bool, +} + +impl NpmLockEntry<'_> { + /// An installed dependency rather than the project's own source: every + /// legacy node, and a `packages` key with a `node_modules/` segment + /// (`""` is the root, other bare keys are workspace members / `file:` + /// directories). + pub(crate) fn is_dependency(&self) -> bool { + self.section == NpmLockSection::Legacy || self.key.contains(NODE_MODULES_SEG) + } + + /// A legacy node npm 6 installs through an npm alias (its key is not + /// the package it stands for). + pub(crate) fn is_legacy_alias(&self) -> bool { + self.section == NpmLockSection::Legacy && self.node.name != self.key + } +} + +/// Every entry of a parsed npm lock, the ONE walk the inventory, lockfile +/// discovery, the vendored and hosted rewriters and the upstream restore +/// share: the `packages` map in document order (when it is an object), +/// then the legacy `dependencies` tree depth-first, parent before its +/// nested `dependencies` (when it is an object, whether or not `packages` +/// exists), bounded at [`MAX_LEGACY_NPM_DEPTH`]. +/// +/// Identity is one rule: a `packages` entry is its `name` field when +/// present (npm writes it for aliases — the key is then the ALIAS), else +/// the path after the LAST `node_modules/` (nesting and scopes), else the +/// key's basename (workspace-member keys, for classification only); a +/// legacy node decodes an alias spec ([`npm_legacy_identity`]). +pub(crate) fn npm_lock_entries(doc: &Value) -> Vec> { + let mut out = Vec::new(); + walk_npm_packages(doc, &mut out); + walk_npm_legacy(doc, &mut out); + out +} + +/// The section npm reads to install: `packages` whenever it is an object +/// (npm >= 7 never reads the legacy mirror then), else `dependencies`. +fn npm_lock_install_section(doc: &Value) -> NpmLockSection { + match doc.get("packages").and_then(Value::as_object) { + Some(_) => NpmLockSection::Packages, + None => NpmLockSection::Legacy, + } +} + +fn walk_npm_packages<'a>(doc: &'a Value, out: &mut Vec>) { + let Some(packages) = doc.get("packages").and_then(Value::as_object) else { + return; + }; + for (key, value) in packages { + let name = match value.get("name").and_then(Value::as_str) { + Some(name) => name, + None => match key.rfind(NODE_MODULES_SEG) { + Some(idx) => &key[idx + NODE_MODULES_SEG.len()..], + None => key.rsplit('/').next().unwrap_or(key), + }, + }; + out.push(NpmLockEntry { + section: NpmLockSection::Packages, + key, + pointer: format!("/packages/{}", escape_pointer_token(key)), + packages_key: Cow::Borrowed(key), + location: Cow::Borrowed(key), + value, + node: NpmLockNode::of(name, value), + link: npm_flag(value, "link"), + bundled: npm_flag(value, "inBundle"), + }); + } +} + +fn walk_npm_legacy<'a>(doc: &'a Value, out: &mut Vec>) { + if let Some(deps) = doc.get("dependencies").and_then(Value::as_object) { + walk_npm_legacy_dependencies(deps, 0, "/dependencies", "", "", out); + } +} + +/// The legacy tree: `{name: {version, resolved, integrity, dependencies: +/// {…}}}`, `pointer` / `parent_key` / `chain` being the enclosing node's. +fn walk_npm_legacy_dependencies<'a>( + deps: &'a serde_json::Map, + depth: usize, + pointer_base: &str, + parent_key: &str, + chain: &str, + out: &mut Vec>, +) { + if depth > MAX_LEGACY_NPM_DEPTH { + return; + } + for (key, value) in deps { + let pointer = format!("{pointer_base}/{}", escape_pointer_token(key)); + let packages_key = legacy_packages_key(parent_key, key); + let location = match chain { + "" => key.clone(), + _ => format!("{chain} > {key}"), + }; + // The nested tree is walked after its parent, from the parent's + // address (taken before the entry owns it). + let nested = value + .get("dependencies") + .and_then(Value::as_object) + .map(|nested| { + let nested_pointer = format!("{pointer}/dependencies"); + ( + nested, + nested_pointer, + packages_key.clone(), + location.clone(), + ) + }); + out.push(NpmLockEntry { + section: NpmLockSection::Legacy, + key, + pointer, + packages_key: Cow::Owned(packages_key), + location: Cow::Owned(location), + value, + node: NpmLockNode::legacy(key, value), + link: false, + bundled: npm_flag(value, "bundled"), + }); + if let Some((nested, nested_pointer, nested_key, nested_chain)) = nested { + walk_npm_legacy_dependencies( + nested, + depth + 1, + &nested_pointer, + &nested_key, + &nested_chain, + out, + ); + } + } +} + +/// RFC 6901 token escaping (`~` → `~0`, `/` → `~1`). +fn escape_pointer_token(token: &str) -> String { + token.replace('~', "~0").replace('/', "~1") +} + +/// Every install entry of a parsed npm lock, as npm reads it — the view +/// the inventory and lockfile discovery (`vex::discover::npm`) share: /// /// * `packages` (lockfileVersion 2/3) whenever it exists — skipping the root -/// `""`, workspace members (keys without `node_modules/` are source dirs; -/// mirrors `npm_lock::scan_lock_matches`' member rule) and `link: true` / -/// `inBundle: true` entries, which npm installs from elsewhere. A v2 -/// lock's `dependencies` is a legacy mirror npm 7+ never reads when -/// `packages` exists, so it is ignored there; +/// `""`, workspace members ([`NpmLockEntry::is_dependency`]) and +/// `link: true` / `inBundle: true` entries, which npm installs from +/// elsewhere. A v2 lock's `dependencies` is a legacy mirror npm 7+ never +/// reads when `packages` exists, so it is ignored there; /// * otherwise the lockfileVersion 1 `dependencies` tree, recursive /// through nested `dependencies`, `bundled: true` entries skipped (their /// nested trees are still walked), alias nodes decoded /// ([`npm_legacy_identity`]). pub(crate) fn npm_lock_nodes(doc: &Value) -> Vec> { - walk_npm_lock(doc, Bundled::Skip, false) - .into_iter() - .map(|(_, node)| node) + npm_install_entries(doc, false) + .map(|entry| entry.node) .collect() } @@ -56,7 +240,9 @@ pub(crate) fn npm_lock_nodes(doc: &Value) -> Vec> { /// [`npm_lock_bundled_nodes`] spelling), for diagnostics that must name /// the entry. pub(crate) fn npm_lock_located_nodes(doc: &Value) -> Vec<(String, NpmLockNode<'_>)> { - walk_npm_lock(doc, Bundled::Skip, true) + npm_install_entries(doc, false) + .map(|entry| (entry.location.into_owned(), entry.node)) + .collect() } /// The BUNDLED entries of a parsed npm lock, each with where the lock puts @@ -70,7 +256,9 @@ pub(crate) fn npm_lock_located_nodes(doc: &Value) -> Vec<(String, NpmLockNode<'_ /// warn `*_bundled_instance_skipped`), and lockfile discovery /// (`vex::discover::npm`) weighs it against the rewired entries. pub(crate) fn npm_lock_bundled_nodes(doc: &Value) -> Vec<(String, NpmLockNode<'_>)> { - walk_npm_lock(doc, Bundled::Only, true) + npm_install_entries(doc, true) + .map(|entry| (entry.location.into_owned(), entry.node)) + .collect() } /// The non-bundled nodes of a lockfileVersion 2 lock's legacy @@ -80,43 +268,27 @@ pub(crate) fn npm_lock_bundled_nodes(doc: &Value) -> Vec<(String, NpmLockNode<'_ /// walks). Lockfile discovery weighs these against the `packages` refs. pub(crate) fn npm_lock_legacy_mirror_nodes(doc: &Value) -> Vec> { let mut out = Vec::new(); - if doc.get("packages").and_then(Value::as_object).is_some() { - if let Some(deps) = doc.get("dependencies").and_then(Value::as_object) { - walk_npm_legacy_dependencies(deps, 0, Bundled::Skip, false, "", &mut out); - } + if npm_lock_install_section(doc) == NpmLockSection::Packages { + walk_npm_legacy(doc, &mut out); } - out.into_iter().map(|(_, node)| node).collect() -} - -/// Which side of the bundled split [`walk_npm_lock`] returns. -#[derive(Clone, Copy, PartialEq, Eq)] -enum Bundled { - Skip, - Only, + out.into_iter() + .filter(|entry| !entry.bundled) + .map(|entry| entry.node) + .collect() } -/// [`npm_lock_nodes`] / [`npm_lock_located_nodes`] / -/// [`npm_lock_bundled_nodes`]: one walk, split on the bundled flag. -/// Locations are built only when `locate` is set (empty otherwise), so the -/// common walk allocates nothing extra. -fn walk_npm_lock(doc: &Value, bundled: Bundled, locate: bool) -> Vec<(String, NpmLockNode<'_>)> { - let mut out = Vec::new(); - if let Some(packages) = doc.get("packages").and_then(Value::as_object) { - for (key, node) in packages { - let Some((_, key_name)) = key.rsplit_once("node_modules/") else { - continue; - }; - if npm_flag(node, "link") || npm_flag(node, "inBundle") != (bundled == Bundled::Only) { - continue; - } - let name = node.get("name").and_then(Value::as_str).unwrap_or(key_name); - let location = if locate { key.clone() } else { String::new() }; - out.push((location, NpmLockNode::of(name, node))); - } - } else if let Some(deps) = doc.get("dependencies").and_then(Value::as_object) { - walk_npm_legacy_dependencies(deps, 0, bundled, locate, "", &mut out); +/// The install section's entries on one side of the bundled split, minus +/// the root, workspace members and links (the [`npm_lock_nodes`] rule). +/// Only the install section is walked. +fn npm_install_entries(doc: &Value, bundled: bool) -> impl Iterator> { + let mut entries = Vec::new(); + match npm_lock_install_section(doc) { + NpmLockSection::Packages => walk_npm_packages(doc, &mut entries), + NpmLockSection::Legacy => walk_npm_legacy(doc, &mut entries), } - out + entries + .into_iter() + .filter(move |entry| entry.is_dependency() && !entry.link && entry.bundled == bundled) } impl<'a> NpmLockNode<'a> { @@ -178,34 +350,6 @@ fn npm_flag(node: &Value, key: &str) -> bool { node.get(key).and_then(Value::as_bool).unwrap_or(false) } -/// The v1 `dependencies` tree: `{name: {version, resolved, integrity, -/// dependencies: {…}}}`. -fn walk_npm_legacy_dependencies<'a>( - deps: &'a serde_json::Map, - depth: usize, - bundled: Bundled, - locate: bool, - parent: &str, - out: &mut Vec<(String, NpmLockNode<'a>)>, -) { - if depth > MAX_LEGACY_NPM_DEPTH { - return; - } - for (name, node) in deps { - let location = match locate { - true if parent.is_empty() => name.clone(), - true => format!("{parent} > {name}"), - false => String::new(), - }; - if npm_flag(node, "bundled") == (bundled == Bundled::Only) { - out.push((location.clone(), NpmLockNode::legacy(name, node))); - } - if let Some(nested) = node.get("dependencies").and_then(Value::as_object) { - walk_npm_legacy_dependencies(nested, depth + 1, bundled, locate, &location, out); - } - } -} - // ── registry view ── #[cfg(test)] @@ -255,3 +399,346 @@ pub(super) async fn inventory_package_lock_in( } Some(out) } + +#[cfg(test)] +mod tests { + use super::*; + + /// One lock with an alias, a scoped package, a `~` / `/` key, a nested + /// legacy tree, a link, a bundled copy and a git entry: the one walk + /// addresses every entry, and each caller's view is a filter over it + /// (#663). + #[test] + fn npm_lock_entries_address_every_entry_once() { + let doc = serde_json::json!({ + "lockfileVersion": 2, + "packages": { + "": { "name": "root", "version": "0.0.0" }, + "packages/ws": { "name": "ws-member", "version": "1.0.0" }, + "node_modules/ws-member": { "resolved": "packages/ws", "link": true }, + "node_modules/@scope/pkg": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@scope/pkg/-/pkg-2.0.0.tgz", + "integrity": "sha512-AAAA" + }, + "node_modules/@scope/pkg/node_modules/t~x": { "version": "0.1.0" }, + "node_modules/lp": { "name": "left-pad", "version": "1.3.0" }, + "node_modules/host": { "version": "1.0.0" }, + "node_modules/host/node_modules/inner": { "version": "1.0.0", "inBundle": true }, + "node_modules/g": { + "version": "1.0.0", + "resolved": "git+ssh://git@github.com/x/g.git#abc" + } + }, + "dependencies": { + "@scope/pkg": { + "version": "2.0.0", + "dependencies": { "t~x": { "version": "0.1.0" } } + }, + "lp": { "version": "npm:left-pad@1.3.0" }, + "host": { + "version": "1.0.0", + "dependencies": { "inner": { "version": "1.0.0", "bundled": true } } + } + } + }); + let entries = npm_lock_entries(&doc); + // (section, key, pointer, packages_key, location, name, version, + // link, bundled, is_dependency, is_legacy_alias) + type Row<'a> = ( + NpmLockSection, + &'a str, + &'a str, + &'a str, + &'a str, + &'a str, + Option<&'a str>, + bool, + bool, + bool, + bool, + ); + use NpmLockSection::{Legacy as L, Packages as P}; + let expected: Vec> = vec![ + ( + P, + "", + "/packages/", + "", + "", + "root", + Some("0.0.0"), + false, + false, + false, + false, + ), + ( + P, + "packages/ws", + "/packages/packages~1ws", + "packages/ws", + "packages/ws", + "ws-member", + Some("1.0.0"), + false, + false, + false, + false, + ), + ( + P, + "node_modules/ws-member", + "/packages/node_modules~1ws-member", + "node_modules/ws-member", + "node_modules/ws-member", + "ws-member", + None, + true, + false, + true, + false, + ), + ( + P, + "node_modules/@scope/pkg", + "/packages/node_modules~1@scope~1pkg", + "node_modules/@scope/pkg", + "node_modules/@scope/pkg", + "@scope/pkg", + Some("2.0.0"), + false, + false, + true, + false, + ), + ( + P, + "node_modules/@scope/pkg/node_modules/t~x", + "/packages/node_modules~1@scope~1pkg~1node_modules~1t~0x", + "node_modules/@scope/pkg/node_modules/t~x", + "node_modules/@scope/pkg/node_modules/t~x", + "t~x", + Some("0.1.0"), + false, + false, + true, + false, + ), + ( + P, + "node_modules/lp", + "/packages/node_modules~1lp", + "node_modules/lp", + "node_modules/lp", + "left-pad", + Some("1.3.0"), + false, + false, + true, + false, + ), + ( + P, + "node_modules/host", + "/packages/node_modules~1host", + "node_modules/host", + "node_modules/host", + "host", + Some("1.0.0"), + false, + false, + true, + false, + ), + ( + P, + "node_modules/host/node_modules/inner", + "/packages/node_modules~1host~1node_modules~1inner", + "node_modules/host/node_modules/inner", + "node_modules/host/node_modules/inner", + "inner", + Some("1.0.0"), + false, + true, + true, + false, + ), + ( + P, + "node_modules/g", + "/packages/node_modules~1g", + "node_modules/g", + "node_modules/g", + "g", + Some("1.0.0"), + false, + false, + true, + false, + ), + ( + L, + "@scope/pkg", + "/dependencies/@scope~1pkg", + "node_modules/@scope/pkg", + "@scope/pkg", + "@scope/pkg", + Some("2.0.0"), + false, + false, + true, + false, + ), + ( + L, + "t~x", + "/dependencies/@scope~1pkg/dependencies/t~0x", + "node_modules/@scope/pkg/node_modules/t~x", + "@scope/pkg > t~x", + "t~x", + Some("0.1.0"), + false, + false, + true, + false, + ), + ( + L, + "lp", + "/dependencies/lp", + "node_modules/lp", + "lp", + "left-pad", + Some("1.3.0"), + false, + false, + true, + true, + ), + ( + L, + "host", + "/dependencies/host", + "node_modules/host", + "host", + "host", + Some("1.0.0"), + false, + false, + true, + false, + ), + ( + L, + "inner", + "/dependencies/host/dependencies/inner", + "node_modules/host/node_modules/inner", + "host > inner", + "inner", + Some("1.0.0"), + false, + true, + true, + false, + ), + ]; + let actual: Vec> = entries + .iter() + .map(|e| { + ( + e.section, + e.key, + e.pointer.as_str(), + e.packages_key.as_ref(), + e.location.as_ref(), + e.node.name, + e.node.version, + e.link, + e.bundled, + e.is_dependency(), + e.is_legacy_alias(), + ) + }) + .collect(); + assert_eq!(actual, expected); + // Every pointer addresses its own entry, so a caller can edit it + // through `Value::pointer_mut`. + for e in &entries { + assert!( + std::ptr::eq(doc.pointer(&e.pointer).unwrap(), e.value), + "{}", + e.pointer + ); + } + // A legacy mirror node's derived `packages_key` names its twin. + for e in entries.iter().filter(|e| e.section == L) { + let twin = entries + .iter() + .find(|p| p.section == P && p.key == e.packages_key) + .unwrap(); + assert_eq!((twin.node.name, twin.bundled), (e.node.name, e.bundled)); + } + + // The install views npm >= 7 reads: `packages`, minus the root, + // the member, the link, and (split out) the bundled copy. + fn names<'a>(nodes: Vec>) -> Vec<&'a str> { + nodes.into_iter().map(|n| n.name).collect() + } + assert_eq!( + names(npm_lock_nodes(&doc)), + ["@scope/pkg", "t~x", "left-pad", "host", "g"] + ); + let located: Vec<(String, &str)> = npm_lock_located_nodes(&doc) + .into_iter() + .map(|(at, n)| (at, n.name)) + .collect(); + assert_eq!( + located[1], + ("node_modules/@scope/pkg/node_modules/t~x".into(), "t~x") + ); + let bundled: Vec<(String, &str)> = npm_lock_bundled_nodes(&doc) + .into_iter() + .map(|(at, n)| (at, n.name)) + .collect(); + assert_eq!( + bundled, + [("node_modules/host/node_modules/inner".into(), "inner")] + ); + // The v2 mirror npm 6 reads: alias decoded, bundled copy skipped. + assert_eq!( + names(npm_lock_legacy_mirror_nodes(&doc)), + ["@scope/pkg", "t~x", "left-pad", "host"] + ); + + // A lockfileVersion 1 lock: the legacy tree IS the install tree, + // located by its `>` chain, and has no mirror. + let v1 = serde_json::json!({ + "lockfileVersion": 1, + "dependencies": doc["dependencies"].clone() + }); + assert_eq!( + names(npm_lock_nodes(&v1)), + ["@scope/pkg", "t~x", "left-pad", "host"] + ); + let bundled: Vec<(String, &str)> = npm_lock_bundled_nodes(&v1) + .into_iter() + .map(|(at, n)| (at, n.name)) + .collect(); + assert_eq!(bundled, [("host > inner".into(), "inner")]); + assert!(npm_lock_legacy_mirror_nodes(&v1).is_empty()); + } + + /// The legacy walk stops at the depth bound instead of recursing on a + /// tamper-crafted tree (built in memory: serde_json's parser would + /// refuse it first). + #[test] + fn npm_lock_entries_bound_the_legacy_depth() { + let mut node = serde_json::json!({ "version": "1.0.0" }); + for _ in 0..(MAX_LEGACY_NPM_DEPTH + 10) { + node = serde_json::json!({ "version": "1.0.0", "dependencies": { "d": node } }); + } + let doc = serde_json::json!({ "dependencies": { "d": node } }); + assert_eq!(npm_lock_entries(&doc).len(), MAX_LEGACY_NPM_DEPTH + 1); + } +} diff --git a/crates/socket-patch-core/src/vendor/mod.rs b/crates/socket-patch-core/src/vendor/mod.rs index 0e9477a15..07318f258 100644 --- a/crates/socket-patch-core/src/vendor/mod.rs +++ b/crates/socket-patch-core/src/vendor/mod.rs @@ -622,6 +622,18 @@ fn harvest_zip_blobs(path: &Path, wanted: &[(String, String)]) -> HashMap, +) -> VendorOutcome { + let marker = VendorMarker::new("npm", &coords.base_purl, record, vendored_at); + write_marker_or_warn( + &project_root.join(&coords.uuid_dir_rel), + &marker, + &mut warnings, + ) + .await; + done(result, Some(entry), warnings) +} + +/// One npm vendoring request, as every tarball flavor's public entry point +/// receives it. +pub(super) struct NpmVendorRequest<'a> { + pub purl: &'a str, + /// The crawler's `node_modules/` dir (or a service-only source); + /// read-only — patching happens on a staged copy. + pub installed_dir: PackageSource<'a>, + pub project_root: &'a Path, + pub record: &'a PatchRecord, + pub sources: &'a PatchSources<'a>, + /// RFC3339 timestamp for the informational marker. + pub vendored_at: &'a str, + pub dry_run: bool, + pub force: bool, + pub service: Option<&'a VendorServiceConfig>, +} + +/// What a flavor's wiring step may consult besides its own plan. +pub(super) struct WireCx<'a> { + pub project_root: &'a Path, + pub coords: &'a NpmCoords, + pub record: &'a PatchRecord, + pub service: Option<&'a VendorServiceConfig>, +} + +/// A committed wiring: the records revert replays, plus the one flavor +/// extra the ledger entry carries. +#[derive(Default)] +pub(super) struct NpmCommit { + pub wiring: Vec, + /// pnpm: which override scaffolds this run created. + pub pnpm: Option, + /// yarn berry: the service's checksum of the tarball, when recorded. + pub yarn_berry10c0: Option, + /// package-lock: the wiring touched no entry whose fields mirror the + /// package's `package.json` (only the v2 legacy `dependencies` mirror + /// was rewired), so nothing was recomputed from a patched manifest and + /// the `package.json` advisory is withheld. + pub manifest_mirrors_untouched: bool, +} + +/// The per-flavor half of npm tarball vendoring: the lock grammar's +/// refusals and its splice. Everything around them — the coordinate guard, +/// staging, the in-sync return, unstaging on a failed wiring, the +/// `package.json` warning, the marker and the ledger entry — is +/// [`vendor_npm_family`]'s, so each cross-flavor rule lives once. +pub(super) trait NpmLockBackend { + /// The flavor's parsed, gated project state, carried from the + /// pre-flight to the wiring. + type Plan; + + /// The ledger `flavor` (`None` is package-lock's pre-flavor spelling). + fn flavor(&self) -> Option<&'static str>; + + /// Read and gate the project BEFORE staging: a refusal leaves the + /// project byte-untouched and nothing is fetched or packed. + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result>; + + /// Splice the staged tarball into the project's wiring and write it, + /// restoring whatever it already wrote on failure. `Ok(None)`: every + /// surface already points at this artifact (in sync, nothing recorded). + /// `Err` is the failure detail; the driver unstages the uuid dir. + async fn wire( + &self, + plan: Self::Plan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + warnings: &mut Vec, + ) -> Result, String>; + + /// The advisory for a patch that rewrites the package's own + /// `package.json`, whose mirrors the flavor's lock either recomputes or + /// keeps. + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning; +} + +/// Vendor one installed npm package through `backend` (see +/// [`NpmLockBackend`]). Refuse-early, wire-last: every refusal fires +/// before any write inside the project and the wiring is the final +/// mutation. `entry` is `None` for dry runs, failures and the in-sync +/// re-run (the existing ledger entry stays authoritative; a run never +/// re-records its own edit as an "original"). The `package.json` advisory +/// is emitted once, and only by a run that wired something. +pub(super) async fn vendor_npm_family( + backend: &B, + req: NpmVendorRequest<'_>, +) -> VendorOutcome { + let NpmVendorRequest { + purl, + installed_dir, + project_root, + record, + sources, + vendored_at, + dry_run, + force, + service, + } = req; + let mut warnings: Vec = Vec::new(); + + // Coordinates: fail-closed before any disk access (see + // `guard_coordinates` for the security note). + let coords = match guard_coordinates(purl, record) { + Ok(coords) => coords, + Err(outcome) => return *outcome, + }; + let plan = match backend + .preflight(project_root, &coords, &mut warnings) + .await + { + Ok(plan) => plan, + Err(outcome) => return *outcome, + }; + + // Stage → patch → pack: tempdir stage outside the project, nested + // node_modules prune, bundled-deps refusal, hardened apply, + // deterministic pack. + let (staged, result) = match stage_patch_pack( + purl, + installed_dir, + project_root, + record, + sources, + dry_run, + force, + &mut warnings, + service, + ) + .await + { + Ok(pair) => pair, + Err(outcome) => return *outcome, + }; + let Some(mut staged) = staged else { + // Failed patch (wiring is last, so the project is byte-untouched) + // or a dry run (stops after the verify). + return done(result, None, warnings); + }; + debug_assert_eq!( + (staged.name.as_str(), staged.version.as_str()), + (coords.name.as_str(), coords.version.as_str()) + ); + + let cx = WireCx { + project_root, + coords: &coords, + record, + service, + }; + let commit = match backend.wire(plan, &cx, &mut staged, &mut warnings).await { + Ok(Some(commit)) => commit, + Ok(None) => { + // In sync: the facts are those of the REUSED committed artifact + // (the pipeline wrote nothing) or, when reuse missed, of a fresh + // acquisition that reproduced the pinned bytes. Synthesize an + // AlreadyPatched-style success and record nothing. + return done( + already_patched_result(purl, &project_root.join(&staged.rel_tgz), &record.files), + None, + warnings, + ); + } + Err(e) => { + return done_failure_unstage( + purl, + e, + project_root, + &coords.uuid_dir_rel, + staged.uuid_dir_preexisted, + ) + .await + } + }; + if staged.staged_pkg_json.is_some() && !commit.manifest_mirrors_untouched { + warnings.push(backend.manifest_warning(&coords.name, &coords.version)); + } + + let mut entry = VendorEntry::npm( + coords.base_purl.clone(), + record.uuid.clone(), + VendorArtifact::tarball(staged.rel_tgz, &staged.packed), + commit.wiring, + backend.flavor(), + ); + entry.pnpm = commit.pnpm; + entry.artifact.yarn_berry10c0 = commit.yarn_berry10c0; + finish_vendored( + project_root, + &coords, + record, + vendored_at, + result, + entry, + warnings, + ) + .await +} + #[cfg(test)] mod tests { use super::*; @@ -1136,6 +1364,72 @@ mod tests { ); } + /// The warning codes riding a failed `Done` (the vendor loop's tell for + /// "not served (yet)", #954). + fn done_warning_codes(outcome: &VendorOutcome) -> Vec<&'static str> { + match outcome { + VendorOutcome::Done { warnings, .. } => warnings.iter().map(|w| w.code).collect(), + other => panic!("expected Done failure, got {other:?}"), + } + } + + /// #954: a patch the service has no artifact for — still building, or + /// `build_failed` / `not_found` / `withdrawn` — fails with the code the + /// vendor loop reads to keep an older vendored patch of the package. + #[tokio::test] + async fn unserved_artifact_failure_carries_the_unserved_code() { + for (status, code, needle) in [ + ( + "pending_build", + crate::vendor::VENDOR_PREBUILT_PENDING, + "still building", + ), + ( + "build_failed", + crate::vendor::VENDOR_PREBUILT_UNAVAILABLE, + "build_failed", + ), + ( + "not_found", + crate::vendor::VENDOR_PREBUILT_UNAVAILABLE, + "not_found", + ), + ( + "withdrawn", + crate::vendor::VENDOR_PREBUILT_UNAVAILABLE, + "withdrawn", + ), + ] { + let server = wiremock::MockServer::start().await; + mount_status_only(&server, status).await; + let tmp = tempfile::tempdir().unwrap(); + let record = record_with_uuid(UUID); + let cfg = service_cfg(&server.uri(), VendorSource::Service); + let err = expect_err(run_pipeline(tmp.path(), &record, Some(&cfg)).await); + assert_eq!(done_warning_codes(&err), vec![code], "{status}"); + expect_done_failure(err, needle); + } + } + + /// A request failure is not "unserved": no code, so the vendor loop + /// still fails the package (#954 keeps only the served-status misses). + #[tokio::test] + async fn request_failure_carries_no_unserved_code() { + use wiremock::matchers::{method, path}; + use wiremock::{Mock, ResponseTemplate}; + let server = wiremock::MockServer::start().await; + Mock::given(method("POST")) + .and(path("/v0/orgs/acme/patches/package")) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + let tmp = tempfile::tempdir().unwrap(); + let record = record_with_uuid(UUID); + let cfg = service_cfg(&server.uri(), VendorSource::Service); + let err = expect_err(run_pipeline(tmp.path(), &record, Some(&cfg)).await); + assert!(done_warning_codes(&err).is_empty()); + } + /// `--vendor-source=service` + a request/transport failure (HTTP 500 on /// the package-reference POST) = hard fail naming the request failure. #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/npm_flavor.rs b/crates/socket-patch-core/src/vendor/npm_flavor.rs index 209b1ba0c..59d57adaa 100644 --- a/crates/socket-patch-core/src/vendor/npm_flavor.rs +++ b/crates/socket-patch-core/src/vendor/npm_flavor.rs @@ -477,13 +477,27 @@ pub async fn vendor_npm_any<'a>( // backend. Each backend already self-stamps `flavor`; we re-assert it from // the probe for belt-and-braces (the values are identical). if let VendorOutcome::Done { - entry, warnings, .. + result, + entry, + warnings, } = &mut outcome { warnings.splice(0..0, probe_warnings); if let Some(entry) = entry { entry.flavor = Some(flavor.as_str().to_string()); } + // npm >= 11.14's `allow-file` gates the `file:` specs the + // package-lock wiring writes (#969); the other flavors' package + // managers do not read it. + if flavor == NpmLockFlavor::PackageLock && result.success { + if let Some((name, version)) = super::npm_common::parse_npm_purl(purl) { + if let Some(detail) = + npm_lock::allow_file_refusal(project_root, &name, &version).await + { + warnings.push(VendorWarning::new(npm_lock::ALLOW_FILE_WARNING, detail)); + } + } + } } outcome } @@ -795,7 +809,19 @@ pub async fn revert_npm_any( /// left to the artifact check and `vex`. pub async fn check_npm_wiring(entry: &VendorEntry, project_root: &Path) -> Result<(), String> { match NpmLockFlavor::from_recorded(entry.flavor.as_deref()) { - Some(NpmLockFlavor::PackageLock) => npm_lock::check_wiring(entry, project_root).await, + Some(NpmLockFlavor::PackageLock) => { + npm_lock::check_wiring(entry, project_root).await?; + // A lock npm's `allow-file` refuses fails every install (#969). + match super::npm_common::parse_npm_purl(&entry.base_purl) { + Some((name, version)) => { + match npm_lock::allow_file_refusal(project_root, &name, &version).await { + Some(detail) => Err(detail), + None => Ok(()), + } + } + None => Ok(()), + } + } _ => Ok(()), } } @@ -1705,6 +1731,47 @@ mod tests { ))); } + /// #969: a project `.npmrc` whose `allow-file` refuses the vendored + /// `file:` tarball (here `root`, and the root manifest does not declare + /// left-pad) still vendors, but the run SAYS every install will fail + /// EALLOWFILE, and `vendor --check` fails the entry; lifting the setting + /// clears both. + #[tokio::test] + async fn package_lock_arm_warns_and_check_fails_when_allow_file_refuses() { + let (tmp, record) = npm_project().await; + touch(tmp.path(), ".npmrc", "allow-file=root\n").await; + + let outcome = vendor_any(tmp.path(), &record).await; + let VendorOutcome::Done { + result, + entry, + warnings, + } = outcome + else { + panic!("expected Done, got {outcome:?}"); + }; + assert!(result.success, "{:?}", result.error); + let warning = warnings + .iter() + .find(|w| w.code == npm_lock::ALLOW_FILE_WARNING) + .unwrap_or_else(|| panic!("no allow-file advisory: {warnings:?}")); + assert!(warning.detail.contains("EALLOWFILE"), "{warning:?}"); + assert!( + warning + .detail + .contains("package-lock.json `node_modules/left-pad`"), + "{warning:?}" + ); + let entry = entry.expect("success carries a ledger entry"); + let err = check_npm_wiring(&entry, tmp.path()) + .await + .expect_err("vendor --check must flag the refused wiring"); + assert!(err.contains("allow-file"), "{err}"); + + touch(tmp.path(), ".npmrc", "allow-file=all\n").await; + assert_eq!(check_npm_wiring(&entry, tmp.path()).await, Ok(())); + } + /// #1094: a workspace member's own package-lock.json is a lock npm never /// reads (members install from the workspace root's lock), so vendoring /// into it would wire nothing. The member is refused as it is without diff --git a/crates/socket-patch-core/src/vendor/npm_lock.rs b/crates/socket-patch-core/src/vendor/npm_lock.rs index f1f49ffef..c467e185c 100644 --- a/crates/socket-patch-core/src/vendor/npm_lock.rs +++ b/crates/socket-patch-core/src/vendor/npm_lock.rs @@ -25,24 +25,25 @@ use crate::patch::apply::PatchSources; use crate::utils::fs::{atomic_write_bytes_preserving_mode, read_regular_to_bytes}; use crate::utils::socket_dir::remove_tree_and_prune; -use super::common::{already_patched_result, done, parse_json_manifest, refused, JsonLayout}; -use super::lock_inventory::npm_legacy_identity; +use super::common::{parse_json_manifest, refused, JsonLayout}; +use super::lock_inventory::{npm_lock_entries, NpmLockSection}; use super::npm_common::{ - done_failure_unstage, guard_coordinates, guard_revert_uuid_dir, stage_patch_pack, + guard_revert_uuid_dir, vendor_npm_family, NpmCommit, NpmCoords, NpmLockBackend, NpmStagedPack, + NpmVendorRequest, WireCx, }; -use super::npm_origin::{legacy_packages_key, npm_non_registry_entries, NpmOverrides}; +use super::npm_origin::{npm_non_registry_entries, npm_shrinkwrapped_entries, NpmOverrides}; use super::parse_memo::ParseMemo; use super::path::parse_vendor_path; use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, -}; +use super::state::{VendorEntry, WiringAction, WiringRecord}; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; // Test-only re-imports: the helpers moved to `npm_common` but the existing // suite exercises them through `use super::*` and stays unmodified. #[cfg(test)] use super::npm_common::{is_safe_npm_name, parse_npm_purl, tgz_rel_leaf}; +#[cfg(test)] +use super::state::VendorArtifact; use crate::constants::npm_family::NPM_LOCKS; /// `npm-shrinkwrap.json` is the primary lock when both exist (npm <= 11 @@ -89,6 +90,8 @@ const DEP_MANIFEST_FIELDS: [&str; 4] = [ /// produced. On success `entry` carries the ledger record to persist — /// `None` for dry runs and for the in-sync re-run (the existing ledger entry /// stays authoritative; we never re-record our own edit as an "original"). +/// The flow is [`vendor_npm_family`]'s; [`PackageLockBackend`] is the +/// package-lock grammar. #[allow(clippy::too_many_arguments)] pub async fn vendor_npm<'a>( purl: &str, @@ -101,319 +104,296 @@ pub async fn vendor_npm<'a>( force: bool, service: Option<&super::VendorServiceConfig>, ) -> VendorOutcome { - let installed_dir = installed_dir.into(); - let mut warnings: Vec = Vec::new(); - - // ── 1. Coordinates (shared guard: fail-closed before any disk access, - // see `npm_common::guard_coordinates` for the security note) ──── - let coords = match guard_coordinates(purl, record) { - Ok(coords) => coords, - Err(outcome) => return *outcome, - }; - let (name, version) = (coords.name.as_str(), coords.version.as_str()); - let uuid_dir_rel = coords.uuid_dir_rel; - let base_purl = coords.base_purl; + vendor_npm_family( + &PackageLockBackend, + NpmVendorRequest { + purl, + installed_dir: installed_dir.into(), + project_root, + record, + sources, + vendored_at, + dry_run, + force, + service, + }, + ) + .await +} - // ── 2. Lockfile selection ─────────────────────────────────────────── - let (lock_name, lock_bytes, sibling_locks) = match select_lockfile(project_root).await { - Ok(Some(found)) => found, - Ok(None) => { - return refused( - "vendor_lockfile_missing", - format!( - "no {PACKAGE_LOCK} or {SHRINKWRAP} {} — vendoring rewires the lockfile, \ - so one must exist (run `npm install` first)", - super::npm_flavor::project_root_location(project_root) - ), - ); - } - Err(e) => { - return refused( - "vendor_lockfile_missing", - format!("cannot read the lockfile: {e}"), - ); - } - }; - let lock = match LOCK_MEMO.parse(&lock_bytes, || parse_json_manifest(&lock_bytes)) { - Ok(v) => v, - Err(e) => { - return refused( - "vendor_lockfile_version_unsupported", - format!("{lock_name} is not parseable JSON: {e}"), - ); - } - }; - let lock_version = match lock_version_gate(&lock, &lock_name) { - Ok(lock_version) => lock_version, - Err(outcome) => return *outcome, - }; +/// The package-lock (`package-lock.json` / `npm-shrinkwrap.json`) half of +/// [`vendor_npm`]. +struct PackageLockBackend; - // The root manifest's `overrides` (#490): which git / url / `file:` - // dependent specs npm really installs from. - let overrides = NpmOverrides::read(project_root).await; +/// [`PackageLockBackend`]'s pre-flight product: the primary lock, its +/// rewritable instances, and every present sibling lock (npm 12). +struct PackageLockPlan { + lock_name: String, + lock_bytes: Vec, + lock: std::sync::Arc, + lock_version: Option, + overrides: NpmOverrides, + matches: Vec, + siblings: Vec, +} - // ── 3. Find the rewritable lock instances ─────────────────────────── - let matches = - match rewritable_matches(&lock, &overrides, name, version, &lock_name, &mut warnings) { - Ok(matches) => matches, - Err(outcome) => return *outcome, - }; +impl NpmLockBackend for PackageLockBackend { + type Plan = PackageLockPlan; - // ── 3b. Sibling lock (npm 12) ─────────────────────────────────────── - // npm 12 removed `npm shrinkwrap`, auto-creates a package-lock.json - // beside a committed npm-shrinkwrap.json on first install and then - // reifies FROM package-lock.json (verified against real npm 12.0.0 / - // 12.1.0; npm <= 11 installs from the shrinkwrap). Wiring only the - // shrinkwrap in that dual-lock state was a silent false success under - // npm 12 — the unpatched registry bytes kept installing. Every other - // present npm lock is therefore rewired identically (the hosted - // rewriter's rule), and one that cannot be is SAID. - let mut siblings: Vec = Vec::new(); - for (sib_name, sib_bytes) in sibling_locks { - match sibling_lock_target( - sib_name, - sib_bytes, - &overrides, - name, - version, - &mut warnings, - ) { - Ok(sib) => siblings.push(sib), - Err(why) => warnings.push(VendorWarning::new( - "vendor_npm_sibling_lock_unwired", - format!( - "{sib_name} beside {lock_name} was NOT rewired for {name}@{version} \ - ({why}) — npm >= 12 installs from {sib_name} when both exist, so those \ - installs stay UNPATCHED; regenerate it from {lock_name} (or delete it) \ - and re-run vendor" + fn flavor(&self) -> Option<&'static str> { + // Package-lock entries predate the flavor field and keep its + // absence as their spelling. + None + } + + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result> { + let (name, version) = (coords.name.as_str(), coords.version.as_str()); + + // ── 2. Lockfile selection ─────────────────────────────────────── + let (lock_name, lock_bytes, sibling_locks) = match select_lockfile(project_root).await { + Ok(Some(found)) => found, + Ok(None) => { + return Err(Box::new(refused( + "vendor_lockfile_missing", + format!( + "no {PACKAGE_LOCK} or {SHRINKWRAP} {} — vendoring rewires the lockfile, \ + so one must exist (run `npm install` first)", + super::npm_flavor::project_root_location(project_root) + ), + ))); + } + Err(e) => { + return Err(Box::new(refused( + "vendor_lockfile_missing", + format!("cannot read the lockfile: {e}"), + ))); + } + }; + let lock = match LOCK_MEMO.parse(&lock_bytes, || parse_json_manifest(&lock_bytes)) { + Ok(v) => v, + Err(e) => { + return Err(Box::new(refused( + "vendor_lockfile_version_unsupported", + format!("{lock_name} is not parseable JSON: {e}"), + ))); + } + }; + let lock_version = lock_version_gate(&lock, &lock_name)?; + + // The root manifest's `overrides` (#490): which git / url / `file:` + // dependent specs npm really installs from. + let overrides = NpmOverrides::read(project_root).await; + + // ── 3. Find the rewritable lock instances ─────────────────────── + let matches = rewritable_matches(&lock, &overrides, name, version, &lock_name, warnings)?; + + // ── 3b. Sibling lock (npm 12) ─────────────────────────────────── + // npm 12 removed `npm shrinkwrap`, never reads a committed + // npm-shrinkwrap.json, writes a package-lock.json (from the registry) + // beside it on first install and then reifies FROM package-lock.json (verified against real npm 12.0.0 / + // 12.1.0; npm <= 11 installs from the shrinkwrap). Wiring only the + // shrinkwrap in that dual-lock state was a silent false success under + // npm 12 — the unpatched registry bytes kept installing. Every other + // present npm lock is therefore rewired identically (the hosted + // rewriter's rule), and one that cannot be is SAID. + // A shrinkwrap with NO package-lock.json twin (#899): npm 12 never reads + // npm-shrinkwrap.json — it resolves from the registry and writes a fresh + // package-lock.json — so the wiring reaches npm <= 11 only. Still wired + // (npm <= 11 installs from it) and SAID, never a silent success. + if lock_name == SHRINKWRAP && sibling_locks.is_empty() { + warnings.push(VendorWarning::new( + "vendor_npm_shrinkwrap_only", + crate::patch::redirect::npm_shrinkwrap_only_detail( + &[format!("{name}@{version}")], + "is vendored", ), - )), + )); + } + let mut siblings: Vec = Vec::new(); + for (sib_name, sib_bytes) in sibling_locks { + match sibling_lock_target(sib_name, sib_bytes, &overrides, name, version, warnings) { + Ok(sib) => siblings.push(sib), + Err(why) => warnings.push(VendorWarning::new( + "vendor_npm_sibling_lock_unwired", + format!( + "{sib_name} beside {lock_name} was NOT rewired for {name}@{version} \ + ({why}) — npm >= 12 installs from {sib_name} when both exist, so those \ + installs stay UNPATCHED; regenerate it from {lock_name} (or delete it) \ + and re-run vendor" + ), + )), + } } + Ok(PackageLockPlan { + lock_name, + lock_bytes, + lock, + lock_version, + overrides, + matches, + siblings, + }) } - // ── 4–7. Stage → patch → pack (shared flavor-agnostic pipeline: - // tempdir stage outside the project, nested node_modules prune, - // bundled-deps refusal, hardened apply, deterministic pack) ──── - let (staged, result) = match stage_patch_pack( - purl, - installed_dir, - project_root, - record, - sources, - dry_run, - force, - &mut warnings, - service, - ) - .await - { - Ok(pair) => pair, - Err(outcome) => return *outcome, - }; - let Some(staged) = staged else { - // Failed patch (no lock writes — wiring is last, so the project is - // byte-untouched) or a dry run (stops after the verify). - return done(result, None, warnings); - }; - let uuid_dir_preexisted = staged.uuid_dir_preexisted; - // `staged.name`/`staged.version` echo the validated coords (the wiring - // below keeps using the borrowed `name`/`version`). - debug_assert_eq!( - (staged.name.as_str(), staged.version.as_str()), - (name, version) - ); - let rel_tgz = staged.rel_tgz; - let packed = staged.packed; - let staged_pkg_json = staged.staged_pkg_json; - // Forward slashes by construction (uuid_dir_rel + leaf are built with - // `/`), relative to the project dir — the spelling npm resolves - // `file:` specs against. - let resolved = format!("file:{rel_tgz}"); - - // ── 8. Lock rewrite (in-place Value mutation: untouched keys stay - // byte-stable thanks to serde_json's preserve_order) ──────────── - let mut wiring: Vec = Vec::new(); - let mut changed = false; - let mut recomputed_deps = false; - // The memo hands the parse out shared; the rewrite takes its own copy — - // the allocation the per-package parse it replaced would have made. - let mut lock = (*lock).clone(); - let rewire = LockRewire { - name, - version, - resolved: &resolved, - integrity: &packed.integrity, - staged_pkg_json: staged_pkg_json.as_ref(), - overrides: &overrides, - }; - if let Err(e) = rewire.apply( - &mut lock, - lock_version, - &matches, - &lock_name, - &mut wiring, - &mut changed, - &mut recomputed_deps, - ) { - return done_failure_unstage(purl, e, project_root, &uuid_dir_rel, uuid_dir_preexisted) - .await; - } - let primary_changed = changed; - // Sibling locks get the identical rewrite; their wiring records name - // their own file, so revert (which walks records per file) restores - // each. - let mut sibling_writes: Vec<(String, Vec, Vec)> = Vec::new(); - for sib in &mut siblings { - let mut sib_changed = false; - if let Err(e) = rewire.apply( - &mut sib.lock, - sib.lock_version, - &sib.matches, - &sib.name, + async fn wire( + &self, + plan: PackageLockPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + _warnings: &mut Vec, + ) -> Result, String> { + let PackageLockPlan { + lock_name, + lock_bytes, + lock, + lock_version, + overrides, + matches, + mut siblings, + } = plan; + let project_root = cx.project_root; + // Forward slashes by construction (uuid_dir_rel + leaf are built with + // `/`), relative to the project dir — the spelling npm resolves + // `file:` specs against. + let resolved = format!("file:{}", staged.rel_tgz); + + // ── 8. Lock rewrite (in-place Value mutation: untouched keys stay + // byte-stable thanks to serde_json's preserve_order) ──────── + let mut wiring: Vec = Vec::new(); + let mut changed = false; + // The memo hands the parse out shared; the rewrite takes its own + // copy — the allocation the per-package parse it replaced would have + // made. + let mut lock = (*lock).clone(); + let rewire = LockRewire { + name: &cx.coords.name, + version: &cx.coords.version, + resolved: &resolved, + integrity: &staged.packed.integrity, + staged_pkg_json: staged.staged_pkg_json.as_ref(), + overrides: &overrides, + }; + rewire.apply( + &mut lock, + lock_version, + &matches, + &lock_name, &mut wiring, - &mut sib_changed, - &mut recomputed_deps, - ) { - return done_failure_unstage(purl, e, project_root, &uuid_dir_rel, uuid_dir_preexisted) - .await; - } - if sib_changed { - changed = true; - let layout = JsonLayout::of(&String::from_utf8_lossy(&sib.bytes)); - match layout.render(&sib.lock) { - Ok(out) => sibling_writes.push((sib.name.clone(), sib.bytes.clone(), out)), - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot serialize {}: {e}", sib.name), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } + &mut changed, + )?; + let primary_changed = changed; + // Sibling locks get the identical rewrite; their wiring records name + // their own file, so revert (which walks records per file) restores + // each. + let mut sibling_writes: Vec<(String, Vec, Vec)> = Vec::new(); + for sib in &mut siblings { + let mut sib_changed = false; + rewire.apply( + &mut sib.lock, + sib.lock_version, + &sib.matches, + &sib.name, + &mut wiring, + &mut sib_changed, + )?; + if sib_changed { + changed = true; + let layout = JsonLayout::of(&String::from_utf8_lossy(&sib.bytes)); + let out = layout + .render(&sib.lock) + .map_err(|e| format!("cannot serialize {}: {e}", sib.name))?; + sibling_writes.push((sib.name.clone(), sib.bytes.clone(), out)); } } - } - if recomputed_deps { - warnings.push(VendorWarning::new( - "vendor_dep_manifest_rewritten", - format!( - "the patch rewrites {name}@{version}'s package.json; its lock entries' \ - dependency/bin fields were recomputed from the patched manifest" - ), - )); - } - if !changed { - // Every instance already points at this uuid with the packed - // integrity: the project is in sync. The facts are those of the - // REUSED committed artifact (the shared pipeline wrote nothing) or, - // when reuse missed (no ledger anchor, a tampered/missing tarball), - // of a freshly acquired one that reproduced the pinned bytes. Touch - // nothing and synthesize an AlreadyPatched-style success, mirroring - // the go_redirect hot path. - return done( - already_patched_result(purl, &project_root.join(&rel_tgz), &record.files), - None, - warnings, - ); - } + if !changed { + // Every instance already points at this uuid with the packed + // integrity: the project is in sync. + return Ok(None); + } - let layout = JsonLayout::of(&String::from_utf8_lossy(&lock_bytes)); - let out = match layout.render(&lock) { - Ok(out) => out, - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot serialize {lock_name}: {e}"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await + let layout = JsonLayout::of(&String::from_utf8_lossy(&lock_bytes)); + let out = layout + .render(&lock) + .map_err(|e| format!("cannot serialize {lock_name}: {e}"))?; + // Siblings first, the primary lock last (still the final mutation); + // a failed write restores every sibling already written, so no lock + // is left resolving through an artifact the unstage removes. + let mut written: Vec<(&str, &[u8])> = Vec::new(); + let mut write_err: Option = None; + // Dropped before the first write, so a torn one leaves nothing + // behind — but only for the locks about to be written. In npm 12's + // dual-lock state only one of the two may hold a match, and the one + // nobody writes is still on disk exactly as parsed: dropping it too + // would make every later package re-parse a lock this run never + // touched. + for (_, original, _) in &sibling_writes { + LOCK_MEMO.forget(original); } - }; - // Siblings first, the primary lock last (still the final mutation); a - // failed write restores every sibling already written, so no lock is - // left resolving through an artifact the unstage removes. - let mut written: Vec<(&str, &[u8])> = Vec::new(); - let mut write_err: Option = None; - // Dropped before the first write, so a torn one leaves nothing behind — - // but only for the locks about to be written. In npm 12's dual-lock - // state only one of the two may hold a match, and the one nobody writes - // is still on disk exactly as parsed: dropping it too would make every - // later package re-parse a lock this run never touched. - for (_, original, _) in &sibling_writes { - LOCK_MEMO.forget(original); - } - if primary_changed { - LOCK_MEMO.forget(&lock_bytes); - } - for (sib_name, original, out) in &sibling_writes { - if let Err(e) = atomic_write_bytes_preserving_mode(&project_root.join(sib_name), out).await - { - write_err = Some(format!("cannot write {sib_name}: {e}")); - break; + if primary_changed { + LOCK_MEMO.forget(&lock_bytes); } - written.push((sib_name, original)); - } - if write_err.is_none() && primary_changed { - if let Err(e) = - atomic_write_bytes_preserving_mode(&project_root.join(&lock_name), &out).await - { - write_err = Some(format!("cannot write {lock_name}: {e}")); + for (sib_name, original, out) in &sibling_writes { + if let Err(e) = + atomic_write_bytes_preserving_mode(&project_root.join(sib_name), out).await + { + write_err = Some(format!("cannot write {sib_name}: {e}")); + break; + } + written.push((sib_name, original)); } - } - if let Some(e) = write_err { - for (sib_name, original) in written { - // Best effort: the original bytes were read moments ago. - let _ = - atomic_write_bytes_preserving_mode(&project_root.join(sib_name), original).await; + if write_err.is_none() && primary_changed { + if let Err(e) = + atomic_write_bytes_preserving_mode(&project_root.join(&lock_name), &out).await + { + write_err = Some(format!("cannot write {lock_name}: {e}")); + } } - return done_failure_unstage(purl, e, project_root, &uuid_dir_rel, uuid_dir_preexisted) - .await; - } - // The bytes now on disk and the documents they were serialized from: the - // next package in this run reads them back and skips the parse. - for sib in siblings { - if let Some((_, _, written)) = sibling_writes.iter().find(|(name, ..)| *name == sib.name) { - LOCK_MEMO.store(written.clone(), sib.lock); + if let Some(e) = write_err { + for (sib_name, original) in written { + // Best effort: the original bytes were read moments ago. + let _ = atomic_write_bytes_preserving_mode(&project_root.join(sib_name), original) + .await; + } + return Err(e); } + // The bytes now on disk and the documents they were serialized from: + // the next package in this run reads them back and skips the parse. + for sib in siblings { + if let Some((_, _, written)) = + sibling_writes.iter().find(|(name, ..)| *name == sib.name) + { + LOCK_MEMO.store(written.clone(), sib.lock); + } + } + if primary_changed { + LOCK_MEMO.store(out, lock); + } + // Only a rewritten `packages` entry has its dependency/bin fields + // recomputed from the patched manifest; a run that rewired just the + // v2 legacy mirror recomputed nothing. + let manifest_mirrors_untouched = !wiring.iter().any(|w| w.kind == KIND_LOCK_ENTRY); + Ok(Some(NpmCommit { + wiring, + manifest_mirrors_untouched, + ..NpmCommit::default() + })) + } + + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning { + VendorWarning::new( + "vendor_dep_manifest_rewritten", + format!( + "the patch rewrites {name}@{version}'s package.json; its lock entries' \ + dependency/bin fields were recomputed from the patched manifest" + ), + ) } - if primary_changed { - LOCK_MEMO.store(out, lock); - } - - // ── 9. Marker + ledger entry ───────────────────────────────────────── - let marker = VendorMarker::new("npm", &base_purl, record, vendored_at); - write_marker_or_warn(&project_root.join(&uuid_dir_rel), &marker, &mut warnings).await; - - let entry = VendorEntry { - ecosystem: "npm".to_string(), - base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: rel_tgz, - sha256: packed.sha256_hex, - size: Some(packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: None, - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }; - done(result, Some(entry), warnings) } /// The project-level refusal [`vendor_npm`]'s step 2 raises whatever the @@ -463,13 +443,29 @@ fn lock_version_gate(lock: &Value, lock_name: &str) -> Result, Box= 12.1 writes lockfileVersion 4 for its native `npm patch` + // (root `patchedDependencies` + a `patched` record on the entry), + // and re-running `npm install` keeps it at 4: the npm >= 7 upgrade + // advice would be wrong (#711). + let detail = if lock_version == Some(4) { + format!( + "{lock_name} has lockfileVersion 4, which npm >= 12.1 writes for its native \ + `npm patch` (`patchedDependencies` in package.json); vendored mode supports \ + only v2/v3 locks and would drop or break that patch. Fold the Socket fix into \ + your own patch, or remove the `patchedDependencies` entries and \ + regenerate the lock with `npm install`, or use hosted mode, which leaves the \ + patched package alone" + ) + } else { format!( "{lock_name} has lockfileVersion {:?}; only v2/v3 locks (with a `packages` \ object) are supported — run `npm install` with npm >= 7 to upgrade it", lock_version - ), + ) + }; + return Err(Box::new(refused( + "vendor_lockfile_version_unsupported", + detail, ))); } Ok(lock_version) @@ -517,6 +513,7 @@ fn rewritable_matches( matches!( w.code, "vendor_bundled_instance_skipped" + | "vendor_shrinkwrapped_instance_skipped" | "vendor_link_entry_skipped" | "vendor_non_registry_entry_skipped" ) @@ -528,9 +525,9 @@ fn rewritable_matches( "vendor_lock_entry_not_rewritable", format!( "every {lock_name} entry for {name}@{version} is bundled inside a \ - parent's tarball, a link, or installed from a non-registry spec and \ - cannot be rewritten — those copies stay UNPATCHED and `npm install` \ - will not help: {}", + parent's tarball, installed from a dependency's own shrinkwrap, a link, \ + or installed from a non-registry spec and cannot be rewritten — those \ + copies stay UNPATCHED and `npm install` will not help: {}", skipped.join("; ") ), ))); @@ -894,7 +891,8 @@ struct LockMatch { /// What the `packages` scan found. enum LockScan { Matches(Vec), - /// A matching key outside `node_modules/` — the caller refuses. + /// A matching key outside `node_modules/` and no rewritable instance + /// besides it — the caller refuses. WorkspaceMember { key: String, }, @@ -916,6 +914,7 @@ pub(super) async fn check_wiring(entry: &VendorEntry, project_root: &Path) -> Re let wired = format!("file:{}", entry.artifact.path); let overrides = NpmOverrides::read(project_root).await; let mut unwired = Vec::new(); + let mut shrinkwrapped = Vec::new(); for lock_name in NPM_LOCKS { let bytes = match read_regular_to_bytes(&project_root.join(lock_name)).await { Ok(bytes) => bytes, @@ -938,6 +937,30 @@ pub(super) async fn check_wiring(entry: &VendorEntry, project_root: &Path) -> Re .filter(|m| m.original.get("resolved").and_then(Value::as_str) != Some(&wired)) .map(|m| format!("{lock_name} `{}`", m.key)), ); + // A copy npm 7–11 install from a dependency's own shrinkwrap + // (#753) installs unpatched whatever this lock says, and vendor + // cannot rewire it, so its "re-run vendor" advice does not apply. + let beneath = npm_shrinkwrapped_entries(&lock); + let mut copies: Vec<(&str, &String)> = npm_lock_entries(&lock) + .into_iter() + .filter(|e| e.section == NpmLockSection::Packages && e.value.is_object()) + .filter(|e| e.node.name == name && e.node.version == Some(version.as_str())) + .filter_map(|e| Some((e.key, beneath.get(e.key)?))) + .collect(); + // In key order, as the shrinkwrapped map lists them. + copies.sort_unstable_by_key(|&(key, _)| key); + for (key, ancestor) in copies { + shrinkwrapped.push(format!("{lock_name} `{key}` (beneath `{ancestor}`)")); + } + } + if !shrinkwrapped.is_empty() { + return Err(format!( + "{} install {name}@{version} from a dependency's own npm-shrinkwrap.json \ + (hasShrinkwrap), which npm 7–11 read instead of this lock, so that copy installs \ + unpatched whatever the lock says and vendoring cannot rewire it; update that \ + dependency to a release that ships a fixed copy", + shrinkwrapped.join(", ") + )); } if unwired.is_empty() { return Ok(()); @@ -951,7 +974,8 @@ pub(super) async fn check_wiring(entry: &VendorEntry, project_root: &Path) -> Re } /// Scan `packages` for instances of `name@version`, pushing skip warnings -/// for the link / inBundle instances that cannot be rewritten. +/// for the workspace-member / link / inBundle / non-registry instances that +/// cannot be rewritten. fn scan_lock_matches( lock: &Value, overrides: &NpmOverrides, @@ -960,35 +984,46 @@ fn scan_lock_matches( warnings: &mut Vec, ) -> LockScan { let mut matches = Vec::new(); - let Some(packages) = lock.get("packages").and_then(Value::as_object) else { + if lock.get("packages").and_then(Value::as_object).is_none() { return LockScan::Matches(matches); // validated earlier; defensive - }; + } let non_registry = npm_non_registry_entries(lock, overrides); - for (key, entry) in packages { + let mut member: Option = None; + let shrinkwrapped = npm_shrinkwrapped_entries(lock); + let candidates = npm_lock_entries(lock).into_iter().filter(|e| { // The root "" entry is the project itself, never a dependency. - if key.is_empty() { - continue; - } - let Some(obj) = entry.as_object() else { - continue; - }; - if entry_name(key, obj) != name { - continue; - } - if obj.get("version").and_then(Value::as_str) != Some(version) { + e.section == NpmLockSection::Packages + && !e.key.is_empty() + && e.node.name == name + && e.node.version == Some(version) + }); + for e in candidates { + let key = e.key; + if !e.is_dependency() { + // The project's own source (a workspace member or `file:` + // directory) that happens to carry this name@version. Vendoring + // it would shadow first-party code, but it does not make the + // registry copies elsewhere in the lock unrewritable (#688): + // skip it, and refuse only when nothing rewritable remains. + warnings.push(VendorWarning::new( + "vendor_workspace_member_skipped", + format!( + "lock entry `{key}` is the project's own source (a workspace member or \ + `file:` directory) with the same name@version; it is not vendored — \ + patch that source directly if it needs the fix" + ), + )); + member.get_or_insert_with(|| key.to_string()); continue; } - if !key.contains(NODE_MODULES_SEG) { - return LockScan::WorkspaceMember { key: key.clone() }; - } - if obj.get("link").and_then(Value::as_bool) == Some(true) { + if e.link { warnings.push(VendorWarning::new( "vendor_link_entry_skipped", format!("lock entry `{key}` is a link (npm workspaces/file: dir); skipped"), )); continue; } - if obj.get("inBundle").and_then(Value::as_bool) == Some(true) { + if e.bundled { // LOUD: this copy ships inside its PARENT's tarball, which we do // not repack — it will still be the unpatched bytes after vendor. warnings.push(VendorWarning::new( @@ -1001,7 +1036,23 @@ fn scan_lock_matches( )); continue; } - if let Some(reason) = non_registry.get(key.as_str()) { + if let Some(ancestor) = shrinkwrapped.get(key) { + // LOUD: npm 7–11 install this copy from `ancestor`'s own + // npm-shrinkwrap.json and ignore the root lock's entry, so a + // rewrite here would report the patch applied while the + // original bytes install (#753). + warnings.push(VendorWarning::new( + "vendor_shrinkwrapped_instance_skipped", + format!( + "lock entry `{key}` is installed from `{ancestor}`'s own \ + npm-shrinkwrap.json (hasShrinkwrap), which npm 7–11 read instead of this \ + lock, so it CANNOT be rewritten — that copy stays UNPATCHED; vendor or \ + update `{ancestor}` to cover it" + ), + )); + continue; + } + if let Some(reason) = non_registry.get(key) { // LOUD: npm installs a git / url / `file:` dependency from the // dependent's spec and ignores `resolved`, so a rewrite here // would report the patch applied while the original bytes @@ -1017,25 +1068,14 @@ fn scan_lock_matches( continue; } matches.push(LockMatch { - key: key.clone(), - original: entry.clone(), + key: key.to_string(), + original: e.value.clone(), }); } - LockScan::Matches(matches) -} - -/// The package name a lock entry stands for: the explicit `name` field when -/// present (npm writes it for aliases — `npm i alias@npm:real`), else the -/// path after the LAST `node_modules/` (handles nesting AND scopes), else -/// the key's basename (workspace-member keys, for classification only). -fn entry_name<'a>(key: &'a str, obj: &'a serde_json::Map) -> &'a str { - if let Some(n) = obj.get("name").and_then(Value::as_str) { - return n; - } - if let Some(idx) = key.rfind(NODE_MODULES_SEG) { - return &key[idx + NODE_MODULES_SEG.len()..]; + match member { + Some(key) if matches.is_empty() => LockScan::WorkspaceMember { key }, + _ => LockScan::Matches(matches), } - key.rsplit('/').next().unwrap_or(key) } fn entry_in_sync(live: &serde_json::Map, resolved: &str, integrity: &str) -> bool { @@ -1070,88 +1110,37 @@ fn recompute_dep_fields(live: &mut serde_json::Map, staged_pkg: & } } -/// Walk the v2 legacy `dependencies` tree and rewrite every node matching -/// `name`+`version`. Nodes are addressed for revert by RFC 6901 JSON -/// Pointer (names may contain `/` — scoped packages — so a plain -/// slash-joined key would be ambiguous; `Value::pointer_mut` handles the -/// `~1` escaping natively). -#[allow(clippy::too_many_arguments)] -fn rewrite_legacy_tree( - deps: &mut serde_json::Map, - pointer_base: &str, - parent_key: &str, +/// The v2 legacy `dependencies` nodes to rewire for `name`+`version`, by +/// RFC 6901 JSON Pointer (names may contain `/` — scoped packages — so a +/// plain slash-joined key would be ambiguous). npm 6 spells an alias +/// install `"": {"version": "npm:real@ver"}`; it installs that node +/// from a `file:` `resolved` like any other (verified against real npm +/// 6.14.18), so it is rewired with the rest (#432). +fn legacy_rewire_targets( + lock: &Value, non_registry: &BTreeMap, name: &str, version: &str, - resolved: &str, - integrity: &str, - lock_name: &str, - wiring: &mut Vec, - changed: &mut bool, -) { - for (dep_name, node) in deps.iter_mut() { - let Some(obj) = node.as_object_mut() else { - continue; - }; - let pointer = format!("{pointer_base}/{}", escape_json_pointer_token(dep_name)); - let packages_key = legacy_packages_key(parent_key, dep_name); - // npm 6 spells an alias install `"": {"version": - // "npm:real@ver"}`; it installs that node from a `file:` `resolved` - // like any other (verified against real npm 6.14.18), so it is - // rewired with the rest (#432). - let (node_name, node_version) = - npm_legacy_identity(dep_name, obj.get("version").and_then(Value::as_str)); - let is_match = node_name == name && node_version == Some(version); - if is_match && obj.get("bundled").and_then(Value::as_bool) == Some(true) { - // Parity with the `packages` scan's inBundle skip: this copy - // ships inside its parent's tarball, npm never installs it from - // `resolved`, and rewriting it would desync the two lock halves. - // (The `packages` twin carries `inBundle` and already pushed the - // stays-UNPATCHED warning.) - } else if is_match && non_registry.contains_key(&packages_key) { - // The mirror of a `packages` entry npm installs from a git / url - // / `file:` spec (#326): its twin was skipped with - // `vendor_non_registry_entry_skipped`, so rewiring this copy - // would record wiring for bytes that never install. - } else if is_match && !entry_in_sync(obj, resolved, integrity) { - let was_vendored = entry_points_into_vendor(obj); - let original = Value::Object(obj.clone()); - obj.insert("resolved".to_string(), Value::String(resolved.to_string())); - obj.insert( - "integrity".to_string(), - Value::String(integrity.to_string()), - ); - wiring.push(WiringRecord { - file: lock_name.to_string(), - kind: KIND_LOCK_LEGACY_ENTRY.to_string(), - action: WiringAction::Rewritten, - key: Some(pointer.clone()), - original: if was_vendored { None } else { Some(original) }, - new: Some(Value::Object(obj.clone())), - }); - *changed = true; - } - if let Some(sub) = obj.get_mut("dependencies").and_then(Value::as_object_mut) { - rewrite_legacy_tree( - sub, - &format!("{pointer}/dependencies"), - &packages_key, - non_registry, - name, - version, - resolved, - integrity, - lock_name, - wiring, - changed, - ); - } - } -} - -/// RFC 6901 token escaping (`~` → `~0`, `/` → `~1`). -fn escape_json_pointer_token(token: &str) -> String { - token.replace('~', "~0").replace('/', "~1") +) -> Vec { + npm_lock_entries(lock) + .into_iter() + .filter(|e| e.section == NpmLockSection::Legacy) + .filter(|e| e.node.name == name && e.node.version == Some(version)) + // Parity with the `packages` scan's inBundle skip: a bundled copy + // ships inside its parent's tarball, npm never installs it from + // `resolved`, and rewriting it would desync the two lock halves. + // (The `packages` twin carries `inBundle` and already pushed the + // stays-UNPATCHED warning.) + .filter(|e| !e.bundled) + // The mirror of a `packages` entry npm installs from a git / url / + // `file:` spec (#326) or from a dependency's shrinkwrap (#753): its + // twin was skipped with `vendor_non_registry_entry_skipped` / + // `vendor_shrinkwrapped_instance_skipped`, so rewiring this copy + // would record wiring for bytes that never install. + .filter(|e| !non_registry.contains_key(e.packages_key.as_ref())) + .filter(|e| e.value.is_object()) + .map(|e| e.pointer) + .collect() } /// The parenthetical of a drifted-lock-entry warning: where the entry @@ -1348,7 +1337,6 @@ struct LockRewire<'a> { } impl LockRewire<'_> { - #[allow(clippy::too_many_arguments)] fn apply( &self, lock: &mut Value, @@ -1357,10 +1345,11 @@ impl LockRewire<'_> { lock_name: &str, wiring: &mut Vec, changed: &mut bool, - recomputed_deps: &mut bool, ) -> Result<(), String> { - // Taken before any rewrite, for the legacy mirror below. - let non_registry = npm_non_registry_entries(lock, self.overrides); + // Taken before any rewrite, for the legacy mirror below: the + // mirror of an entry the `packages` scan skipped stays as it is. + let mut non_registry = npm_non_registry_entries(lock, self.overrides); + non_registry.extend(npm_shrinkwrapped_entries(lock)); let Some(packages) = lock.get_mut("packages").and_then(Value::as_object_mut) else { return Err("lock `packages` object vanished mid-rewrite".to_string()); }; @@ -1387,7 +1376,6 @@ impl LockRewire<'_> { ); if let Some(pkg) = self.staged_pkg_json { recompute_dep_fields(live, pkg); - *recomputed_deps = true; } wiring.push(WiringRecord { file: lock_name.to_string(), @@ -1407,26 +1395,261 @@ impl LockRewire<'_> { // npm 6); leaving the registry resolved/integrity there would let an // old client silently install unpatched bytes. if lock_version == Some(2) { - if let Some(deps) = lock.get_mut("dependencies").and_then(Value::as_object_mut) { - rewrite_legacy_tree( - deps, - "/dependencies", - "", - &non_registry, - self.name, - self.version, - self.resolved, - self.integrity, - lock_name, - wiring, - changed, + let targets = legacy_rewire_targets(lock, &non_registry, self.name, self.version); + for pointer in targets { + let Some(obj) = lock.pointer_mut(&pointer).and_then(Value::as_object_mut) else { + continue; + }; + if entry_in_sync(obj, self.resolved, self.integrity) { + continue; + } + let was_vendored = entry_points_into_vendor(obj); + let original = Value::Object(obj.clone()); + obj.insert( + "resolved".to_string(), + Value::String(self.resolved.to_string()), + ); + obj.insert( + "integrity".to_string(), + Value::String(self.integrity.to_string()), ); + wiring.push(WiringRecord { + file: lock_name.to_string(), + kind: KIND_LOCK_LEGACY_ENTRY.to_string(), + action: WiringAction::Rewritten, + key: Some(pointer), + original: if was_vendored { None } else { Some(original) }, + new: Some(Value::Object(obj.clone())), + }); + *changed = true; } } Ok(()) } } +// ── npm `allow-file` (#969) ───────────────────────────────────────────── +// +// npm >= 11.14 gates every dependency that resolves to a local tarball +// (`file:` spec) by `allow-file`: `all` (the default) admits them, `none` +// refuses every one, and any other value (`root`) admits only a node that +// satisfies a dependency DECLARED by the project root or a workspace +// (arborist's `#checkAllow` / reify `_isRoot`) — EALLOWFILE otherwise. The +// vendored rewiring writes exactly such specs, so an explicit non-`all` +// setting in any npm config layer makes every install of the vendored lock +// fail. The setting is respected (never rewritten — the hosted +// `allow-remote` precedent) and SAID: a vendor advisory and a `vendor +// --check` failure. + +/// Advisory code for a vendored lock entry npm's `allow-file` refuses. +pub const ALLOW_FILE_WARNING: &str = "vendor_npm_allow_file"; + +/// Where the effective non-`all` `allow-file` value comes from. +#[derive(Debug, Clone, PartialEq)] +enum AllowFileSource { + /// An `npm_config_allow_file` environment variable (beats every file). + Env(String), + /// The project `.npmrc`. + Project, + /// The user / global / builtin config file. + Outer { + layer: &'static str, + path: std::path::PathBuf, + }, +} + +/// The effective `allow-file` value npm would read, in its layer order +/// (env > project > user > global > builtin), when it is NOT `all`. +fn effective_allow_file( + project_npmrc: Option<&str>, + outer: &crate::patch::redirect::npmrc::OuterSetting, +) -> Option<(AllowFileSource, String)> { + use crate::patch::redirect::npmrc::npmrc_top_level_value; + let (source, value) = if let Some((var, value)) = &outer.env { + (AllowFileSource::Env(var.clone()), value.clone()) + } else if let Some(value) = project_npmrc.and_then(|t| npmrc_top_level_value(t, "allow-file")) { + (AllowFileSource::Project, value) + } else { + let file = outer.file.as_ref()?; + ( + AllowFileSource::Outer { + layer: file.layer, + path: file.path.clone(), + }, + file.value.clone(), + ) + }; + (value != "all").then_some((source, value)) +} + +/// Does an importer (`""` = the project root, or a workspace entry) declare +/// `folder` as a dependency of any kind? +fn importer_declares(importer: &serde_json::Map, folder: &str) -> bool { + [ + "dependencies", + "devDependencies", + "optionalDependencies", + "peerDependencies", + ] + .iter() + .any(|field| { + importer + .get(*field) + .and_then(Value::as_object) + .is_some_and(|deps| deps.contains_key(folder)) + }) +} + +/// The `packages` key node resolution reaches for `folder` from the +/// importer at `dir` (walking up `node_modules` dirs like Node), if any. +fn resolve_from<'a>( + packages: &'a serde_json::Map, + dir: &str, + folder: &str, +) -> Option<&'a str> { + let mut dir = dir; + loop { + let candidate = if dir.is_empty() { + format!("{NODE_MODULES_SEG}{folder}") + } else { + format!("{dir}/{NODE_MODULES_SEG}{folder}") + }; + if let Some((key, _)) = packages.get_key_value(&candidate) { + return Some(key.as_str()); + } + if dir.is_empty() { + return None; + } + dir = dir.rfind('/').map_or("", |i| &dir[..i]); + } +} + +/// Is the lock node at `key` a dependency npm counts as "root" for +/// `allow-file=root`: one the project root or a workspace declares and +/// resolves to this very node? +fn is_root_dependency(packages: &serde_json::Map, key: &str) -> bool { + let Some(idx) = key.rfind(NODE_MODULES_SEG) else { + return false; + }; + let folder = &key[idx + NODE_MODULES_SEG.len()..]; + packages.iter().any(|(importer_key, importer)| { + !importer_key.contains(NODE_MODULES_SEG) + && importer + .as_object() + .is_some_and(|obj| importer_declares(obj, folder)) + && resolve_from(packages, importer_key, folder) == Some(key) + }) +} + +/// The ` ``` instances of `name@version` (the set vendoring +/// rewires) npm's `allow-file=` refuses, across every present npm lock. +async fn allow_file_refused_instances( + project_root: &Path, + name: &str, + version: &str, + value: &str, +) -> Vec { + let overrides = NpmOverrides::read(project_root).await; + let mut refused = Vec::new(); + for lock_name in NPM_LOCKS { + let Ok(bytes) = read_regular_to_bytes(&project_root.join(lock_name)).await else { + continue; + }; + let Ok(lock) = parse_json_manifest(&bytes) else { + continue; + }; + let LockScan::Matches(matches) = + scan_lock_matches(&lock, &overrides, name, version, &mut Vec::new()) + else { + continue; + }; + let Some(packages) = lock.get("packages").and_then(Value::as_object) else { + continue; + }; + refused.extend( + matches + .iter() + .filter(|m| value == "none" || !is_root_dependency(packages, &m.key)) + .map(|m| format!("{lock_name} `{}`", m.key)), + ); + } + refused +} + +/// The human reason npm >= 11.14 refuses `name@version`'s vendored `file:` +/// tarball under the effective `allow-file` setting, or `None` when it +/// installs. `env` locates npm's config layers ([`NpmConfigEnv`]). +/// +/// [`NpmConfigEnv`]: crate::patch::redirect::npmrc::NpmConfigEnv +pub(super) async fn allow_file_refusal_with( + project_root: &Path, + name: &str, + version: &str, + env: &crate::patch::redirect::npmrc::NpmConfigEnv, +) -> Option { + use crate::patch::redirect::npmrc::{resolve_outer_npm_setting, NPMRC_REL}; + use crate::utils::fs::read_regular_to_string_sync; + let read = |path: &Path| read_regular_to_string_sync(path).ok(); + let outer = resolve_outer_npm_setting(env, read, "allow-file", "all"); + let project = read(&project_root.join(NPMRC_REL)); + let (source, value) = effective_allow_file(project.as_deref(), &outer)?; + let refused = allow_file_refused_instances(project_root, name, version, &value).await; + if refused.is_empty() { + return None; + } + let setter = match &source { + AllowFileSource::Env(var) => format!("the environment variable {var}={value}"), + AllowFileSource::Project => format!("the project .npmrc (`allow-file={value}`)"), + AllowFileSource::Outer { layer, path } => format!( + "the {layer} npm config ({}: `allow-file={value}`)", + path.display() + ), + }; + let why = if value == "none" { + "which refuses every `file:` dependency".to_string() + } else { + "which admits a `file:` dependency only when the project root or a workspace \ + declares it, and these vendored copies are transitive" + .to_string() + }; + let remedy = match &source { + AllowFileSource::Env(var) => { + format!("unset {var} (or install with `npm ci --allow-file=all`)") + } + AllowFileSource::Project => "set `allow-file=all` in the project .npmrc (or install \ + with `npm ci --allow-file=all`)" + .to_string(), + AllowFileSource::Outer { layer, .. } => format!( + "set `allow-file=all` in the project .npmrc (it outranks the {layer} config) or \ + install with `npm ci --allow-file=all`" + ), + }; + Some(format!( + "npm >= 11.14 refuses {name}@{version}'s vendored `file:` tarball (EALLOWFILE): \ + {setter} sets `allow-file`, {why} ({}), so every `npm ci` / `npm install` of this \ + lock fails until it is lifted — {remedy}. The setting was respected and left \ + untouched (npm <= 11.13 has no such setting)", + refused.join(", ") + )) +} + +/// [`allow_file_refusal_with`] against this process's npm config layers. +/// Unit tests see only the project `.npmrc`: the developer's or runner's +/// own layers (an `npm_config_allow_file` variable, a `~/.npmrc`) must not +/// decide them; the layer order itself is tested through +/// [`allow_file_refusal_with`]. +pub(super) async fn allow_file_refusal( + project_root: &Path, + name: &str, + version: &str, +) -> Option { + #[cfg(not(test))] + let env = crate::patch::redirect::npmrc::NpmConfigEnv::from_process(); + #[cfg(test)] + let env = crate::patch::redirect::npmrc::NpmConfigEnv::default(); + allow_file_refusal_with(project_root, name, version, &env).await +} + #[cfg(test)] mod tests { use super::*; @@ -2295,6 +2518,57 @@ mod tests { ); } + /// REGRESSION (#753): npm 7–11 install a copy beneath a `hasShrinkwrap` + /// package from that package's own npm-shrinkwrap.json, so the nested + /// entry is skipped loudly while the hoisted copy is still vendored; a + /// shrinkwrapped-only target refuses instead of reporting success. + #[tokio::test] + async fn instance_under_has_shrinkwrap_parent_is_skipped_with_warning() { + let mut lock = default_lock(); + lock["packages"]["node_modules/foo"]["hasShrinkwrap"] = json!(true); + let fx = fixture_with("left-pad", "1.3.0", lock.clone()).await; + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success); + assert_eq!(entry.unwrap().wiring.len(), 1, "only the hoisted copy"); + let skipped = warnings + .iter() + .find(|w| w.code == "vendor_shrinkwrapped_instance_skipped") + .unwrap_or_else(|| panic!("{warnings:?}")); + assert!( + skipped.detail.contains("UNPATCHED") + && skipped + .detail + .contains("node_modules/foo/node_modules/left-pad") + && skipped.detail.contains("`node_modules/foo`"), + "{}", + skipped.detail + ); + let live = fx.read_lock().await; + assert_eq!( + live["packages"]["node_modules/foo/node_modules/left-pad"], + lock["packages"]["node_modules/foo/node_modules/left-pad"], + "the shrinkwrapped copy is byte-untouched" + ); + + // Only the shrinkwrapped copy: refuse, write nothing. + let mut lock = lock; + lock["packages"] + .as_object_mut() + .unwrap() + .shift_remove("node_modules/left-pad"); + let fx = fixture_with("left-pad", "1.3.0", lock).await; + let detail = expect_refused(fx.vendor(false).await, "vendor_lock_entry_not_rewritable"); + assert!( + detail.contains("UNPATCHED") && detail.contains("hasShrinkwrap"), + "{detail}" + ); + assert_eq!( + tokio::fs::read(fx.lock_path()).await.unwrap(), + fx.lock_bytes, + "lock untouched by the refusal" + ); + } + /// When EVERY lock instance of the target is bundled or a link, the /// refusal must state the real reason (the entry IS in the lock and /// `npm install` will not help) and keep the stays-UNPATCHED advisory — @@ -2392,6 +2666,66 @@ mod tests { ); } + /// #688: a local directory (`file:` dependency or workspace member) whose + /// package.json carries the patched name@version must not refuse the + /// REGISTRY copies elsewhere in the lock — those are rewired, the local + /// source is skipped with a warning and left untouched. + #[tokio::test] + async fn namesake_local_directory_does_not_block_registry_copies() { + let mut lock = default_lock(); + lock["packages"][""]["dependencies"]["lp-local"] = json!("file:./third_party/left-pad"); + lock["packages"]["node_modules/lp-local"] = json!({ + "resolved": "third_party/left-pad", + "link": true + }); + lock["packages"]["third_party/left-pad"] = json!({ + "name": "left-pad", + "version": "1.3.0" + }); + let fx = fixture_with("left-pad", "1.3.0", lock).await; + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success, "{:?}", result.error); + let entry = entry.expect("vendored entry"); + assert_eq!(entry.wiring.len(), 2, "both registry instances rewired"); + + let member = warnings + .iter() + .find(|w| w.code == "vendor_workspace_member_skipped") + .expect("the skipped local source is named"); + assert!( + member.detail.contains("third_party/left-pad"), + "{}", + member.detail + ); + + let wired = fx.read_lock().await; + let tgz = json!(format!("file:{}", fx.expected_rel_tgz())); + assert_eq!(wired["packages"]["node_modules/left-pad"]["resolved"], tgz); + assert_eq!( + wired["packages"]["node_modules/foo/node_modules/left-pad"]["resolved"], + tgz + ); + assert_eq!( + wired["packages"]["third_party/left-pad"], + json!({ "name": "left-pad", "version": "1.3.0" }), + "the first-party source entry is left untouched" + ); + assert_eq!( + wired["packages"]["node_modules/lp-local"], + json!({ "resolved": "third_party/left-pad", "link": true }) + ); + + // The wiring audit covers the same registry copies: restoring the + // pre-vendor lock is drift, not silently skipped because of the + // namesake member. + assert_eq!(check_wiring(&entry, fx.root()).await, Ok(())); + tokio::fs::write(fx.lock_path(), &fx.lock_bytes) + .await + .unwrap(); + let drift = check_wiring(&entry, fx.root()).await.unwrap_err(); + assert!(drift.contains("`node_modules/left-pad`"), "{drift}"); + } + #[tokio::test] async fn lockfile_v1_is_refused() { let lock = json!({ @@ -2409,6 +2743,40 @@ mod tests { ); } + /// REGRESSION (#711): npm >= 12.1 writes lockfileVersion 4 for its + /// native `npm patch`, and `npm install` keeps it there. The refusal + /// names that cause instead of the "upgrade with npm >= 7" advice that + /// fits only a v1 lock. + #[tokio::test] + async fn lockfile_v4_refusal_names_npm_patch() { + let lock = json!({ + "name": "fixture", + "version": "1.0.0", + "lockfileVersion": 4, + "packages": { + "": { "name": "fixture", "version": "1.0.0" }, + "node_modules/left-pad": { + "version": "1.3.0", + "resolved": REG_RESOLVED, + "integrity": "sha512-orig==", + "patched": { "integrity": "sha512-user==", "path": "patches/left-pad@1.3.0.patch" } + } + } + }); + let fx = fixture_with("left-pad", "1.3.0", lock).await; + let detail = expect_refused( + fx.vendor(false).await, + "vendor_lockfile_version_unsupported", + ); + assert!( + detail.contains("lockfileVersion 4") + && detail.contains("npm patch") + && detail.contains("patchedDependencies") + && !detail.contains("npm >= 7"), + "{detail}" + ); + } + /// The takeover preflight raises exactly the backend's own version /// refusal on a v1 lock (#659), before any write. #[tokio::test] @@ -2496,6 +2864,11 @@ mod tests { ); } + /// A shrinkwrap-only project is still rewired (npm <= 11 installs from + /// it). REGRESSION (#899): npm 12 never reads npm-shrinkwrap.json, so + /// every run — the in-sync re-run too — warns + /// `vendor_npm_shrinkwrap_only`; a project with the package-lock.json + /// twin does not. #[tokio::test] async fn shrinkwrap_only_project_rewires_the_shrinkwrap() { let fx = fixture().await; @@ -2522,6 +2895,31 @@ mod tests { shrink["packages"]["node_modules/left-pad"]["resolved"], json!(format!("file:{}", fx.expected_rel_tgz())) ); + let shrinkwrap_only = |warnings: &[VendorWarning]| { + warnings + .iter() + .find(|w| w.code == "vendor_npm_shrinkwrap_only") + .map(|w| w.detail.clone()) + }; + let detail = shrinkwrap_only(&warnings) + .unwrap_or_else(|| panic!("shrinkwrap-only warning missing: {warnings:?}")); + for needle in ["left-pad@1.3.0", "npm >= 12", "no package-lock.json"] { + assert!(detail.contains(needle), "{needle}: {detail}"); + } + + // The in-sync re-run still says it. + let (result, again, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success); + assert!(again.is_none()); + assert!(shrinkwrap_only(&warnings).is_some(), "{warnings:?}"); + + // With the twin npm 12 reads, no shrinkwrap-only warning. + tokio::fs::copy(fx.root().join(SHRINKWRAP), fx.lock_path()) + .await + .unwrap(); + let (result, _, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success); + assert!(shrinkwrap_only(&warnings).is_none(), "{warnings:?}"); } /// npm 12 auto-creates package-lock.json beside a committed @@ -3120,6 +3518,113 @@ mod tests { assert_eq!(e["license"], json!("WTFPL"), "non-dep fields untouched"); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture().await; + let before = installed_pkg_json("left-pad", "1.3.0"); + let after: &[u8] = + br#"{"name":"left-pad","version":"1.3.0","dependencies":{"wow":"^1.0.0"}}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(&before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + + /// The `package.json` advisory says the lock entries' dependency/bin + /// fields were recomputed, so it fires only when a `packages` entry + /// was. A re-run that rewires just the v2 legacy `dependencies` mirror + /// (an npm 6 install re-saved it to the registry) recomputes nothing. + #[tokio::test] + async fn legacy_mirror_only_rewire_omits_the_manifest_warning() { + let lock = json!({ + "name": "fixture", + "version": "1.0.0", + "lockfileVersion": 2, + "requires": true, + "packages": { + "": { "name": "fixture", "version": "1.0.0" }, + "node_modules/left-pad": { + "version": "1.3.0", + "resolved": REG_RESOLVED, + "integrity": "sha512-orig==" + } + }, + "dependencies": { + "left-pad": { + "version": "1.3.0", + "resolved": REG_RESOLVED, + "integrity": "sha512-orig==" + } + } + }); + let mut fx = fixture_with("left-pad", "1.3.0", lock).await; + let before = installed_pkg_json("left-pad", "1.3.0"); + let after: &[u8] = + br#"{"name":"left-pad","version":"1.3.0","dependencies":{"wow":"^1.0.0"}}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(&before), + after_hash, + }, + ); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code == "vendor_dep_manifest_rewritten") + .count() + }; + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + // npm 6 re-saves the mirror from the registry; `packages` stays wired. + let mut live = fx.read_lock().await; + live["dependencies"]["left-pad"]["resolved"] = json!(REG_RESOLVED); + live["dependencies"]["left-pad"]["integrity"] = json!("sha512-orig=="); + tokio::fs::write(fx.lock_path(), serialize_json(&live, " ").unwrap()) + .await + .unwrap(); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success, "{:?}", result.error); + let entry = entry.expect("the mirror rewire is recorded"); + assert!( + entry + .wiring + .iter() + .all(|r| r.kind == KIND_LOCK_LEGACY_ENTRY), + "{:?}", + entry.wiring + ); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + /// #324: vendoring keeps a CRLF, tab-indented or BOM-prefixed lock's /// layout (a BOM lock is read the way npm reads it, not refused), and /// `vendor --revert` restores its original bytes. @@ -4146,14 +4651,11 @@ mod tests { } #[test] - fn indent_detection_and_pointer_escaping() { + fn indent_detection() { assert_eq!(detect_indent("{\n \"a\": 1\n}\n"), " "); assert_eq!(detect_indent("{\n\t\"a\": 1\n}\n"), "\t"); assert_eq!(detect_indent("{\n \"a\": 1\n}\n"), " "); assert_eq!(detect_indent("{}"), " ", "default for flat files"); - - assert_eq!(escape_json_pointer_token("@scope/name"), "@scope~1name"); - assert_eq!(escape_json_pointer_token("a~b"), "a~0b"); } use crate::api::client::{ApiClient, ApiClientOptions}; @@ -5119,4 +5621,168 @@ mod tests { assert_eq!(planned, Ok(()), "the plan cannot see a staged-copy gate"); assert_eq!(looped, Ok(())); } + + /// #969: a project lock with a direct `left-pad` and a transitive + /// `is-number` (via `to-regex-range`), plus a workspace that declares + /// `is-number` with its own nested copy. + async fn allow_file_project(npmrc: Option<&str>) -> (tempfile::TempDir, std::path::PathBuf) { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path().join("proj"); + let home = tmp.path().join("home"); + tokio::fs::create_dir_all(&root).await.unwrap(); + tokio::fs::create_dir_all(&home).await.unwrap(); + let lock = json!({ + "name": "t", + "lockfileVersion": 3, + "packages": { + "": { + "name": "t", + "workspaces": ["ws/a"], + "dependencies": {"to-regex-range": "5.0.1", "left-pad": "1.3.0"} + }, + "ws/a": {"name": "a", "dependencies": {"is-number": "6.0.0"}}, + "node_modules/a": {"resolved": "ws/a", "link": true}, + "node_modules/left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz" + }, + "node_modules/to-regex-range": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", + "dependencies": {"is-number": "^7.0.0"} + }, + "node_modules/is-number": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz" + }, + "ws/a/node_modules/is-number": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-6.0.0.tgz" + } + } + }); + tokio::fs::write( + root.join(PACKAGE_LOCK), + serde_json::to_vec_pretty(&lock).unwrap(), + ) + .await + .unwrap(); + if let Some(npmrc) = npmrc { + tokio::fs::write(root.join(".npmrc"), npmrc).await.unwrap(); + } + (tmp, home) + } + + fn allow_file_env( + home: &Path, + extra: &[(&str, &str)], + ) -> crate::patch::redirect::npmrc::NpmConfigEnv { + let mut vars = vec![("HOME".to_string(), home.display().to_string())]; + vars.extend(extra.iter().map(|(k, v)| (k.to_string(), v.to_string()))); + crate::patch::redirect::npmrc::NpmConfigEnv::from_parts(vars, None, false) + } + + /// #969: npm >= 11.14 refuses a vendored `file:` tarball under + /// `allow-file=root` when the copy is transitive, and under + /// `allow-file=none` always; the default `all` (or no setting) admits it. + #[tokio::test] + async fn allow_file_refusal_follows_npms_root_rule() { + let (tmp, home) = allow_file_project(Some("allow-file=root\n")).await; + let root = tmp.path().join("proj"); + let env = allow_file_env(&home, &[]); + // Transitive (pulled in by to-regex-range): refused, remedy named. + let detail = allow_file_refusal_with(&root, "is-number", "7.0.0", &env) + .await + .expect("transitive copy is refused under allow-file=root"); + assert!(detail.contains("EALLOWFILE"), "{detail}"); + assert!(detail.contains("`node_modules/is-number`"), "{detail}"); + assert!( + detail.contains("project .npmrc (`allow-file=root`)"), + "{detail}" + ); + assert!(detail.contains("npm ci --allow-file=all"), "{detail}"); + // Direct root dependency and a workspace's own nested dependency: + // both are "root" to npm. + assert_eq!( + allow_file_refusal_with(&root, "left-pad", "1.3.0", &env).await, + None + ); + assert_eq!( + allow_file_refusal_with(&root, "is-number", "6.0.0", &env).await, + None + ); + + // `none` refuses even the direct dependency. + let (tmp, home) = allow_file_project(Some("allow-file=none\n")).await; + let root = tmp.path().join("proj"); + let detail = + allow_file_refusal_with(&root, "left-pad", "1.3.0", &allow_file_env(&home, &[])) + .await + .expect("allow-file=none refuses every file: dependency"); + assert!( + detail.contains("refuses every `file:` dependency"), + "{detail}" + ); + + // Default / explicit `all`: nothing to say. + for npmrc in [None, Some("allow-file=all\n"), Some("allow-remote=none\n")] { + let (tmp, home) = allow_file_project(npmrc).await; + let root = tmp.path().join("proj"); + assert_eq!( + allow_file_refusal_with(&root, "is-number", "7.0.0", &allow_file_env(&home, &[])) + .await, + None, + "{npmrc:?}" + ); + } + } + + /// #969: every npm config layer counts, in npm's precedence order — the + /// env var beats the project file, which beats the user config. + #[tokio::test] + async fn allow_file_refusal_reads_every_npm_config_layer() { + let (tmp, home) = allow_file_project(None).await; + let root = tmp.path().join("proj"); + let detail = allow_file_refusal_with( + &root, + "is-number", + "7.0.0", + &allow_file_env(&home, &[("npm_config_allow_file", "root")]), + ) + .await + .expect("env layer refuses"); + assert!(detail.contains("npm_config_allow_file=root"), "{detail}"); + assert!(detail.contains("unset npm_config_allow_file"), "{detail}"); + + tokio::fs::write(home.join(".npmrc"), "allow-file=none\n") + .await + .unwrap(); + let detail = + allow_file_refusal_with(&root, "left-pad", "1.3.0", &allow_file_env(&home, &[])) + .await + .expect("user layer refuses"); + assert!(detail.contains("the user npm config"), "{detail}"); + // A project `allow-file=all` outranks the user config… + tokio::fs::write(root.join(".npmrc"), "allow-file=all\n") + .await + .unwrap(); + assert_eq!( + allow_file_refusal_with(&root, "left-pad", "1.3.0", &allow_file_env(&home, &[])).await, + None + ); + // …and an env `all` outranks a project `none`. + tokio::fs::write(root.join(".npmrc"), "allow-file=none\n") + .await + .unwrap(); + assert_eq!( + allow_file_refusal_with( + &root, + "left-pad", + "1.3.0", + &allow_file_env(&home, &[("npm_config_allow_file", "all")]) + ) + .await, + None + ); + } } diff --git a/crates/socket-patch-core/src/vendor/npm_origin.rs b/crates/socket-patch-core/src/vendor/npm_origin.rs index 9c8a31840..51ef5ecf3 100644 --- a/crates/socket-patch-core/src/vendor/npm_origin.rs +++ b/crates/socket-patch-core/src/vendor/npm_origin.rs @@ -407,6 +407,42 @@ pub(crate) fn npm_non_registry_entries( out } +/// Every `packages` key installed beneath a package that ships its own +/// `npm-shrinkwrap.json` (the lock marks that ancestor `"hasShrinkwrap": +/// true`, e.g. `firebase-tools`, `netlify-cli`), mapped to the outermost +/// such ancestor's key (#753). npm 7–11 install that subtree from the +/// dependency's own shrinkwrap at reify time and ignore the root lock's +/// entries for it, so a rewrite of one of them installs nothing (npm 12 +/// honors the root lock, but the files cannot tell which npm installs). +/// The rewriters skip these entries loudly and lockfile discovery never +/// attests them. Empty for a lock without `packages`: a lockfileVersion 1 +/// lock does not record `hasShrinkwrap`. +pub(crate) fn npm_shrinkwrapped_entries(lock: &Value) -> BTreeMap { + let mut out = BTreeMap::new(); + let Some(packages) = lock.get("packages").and_then(Value::as_object) else { + return out; + }; + let ships_shrinkwrap = |key: &str| { + packages + .get(key) + .and_then(|entry| entry.get("hasShrinkwrap")) + .and_then(Value::as_bool) + == Some(true) + }; + for key in packages.keys() { + // Each `/node_modules/` segment closes an enclosing package's path, + // outermost first. + let ancestor = key + .match_indices("/node_modules/") + .map(|(at, _)| &key[..at]) + .find(|prefix| ships_shrinkwrap(prefix)); + if let Some(ancestor) = ancestor { + out.insert(key.clone(), ancestor.to_string()); + } + } + out +} + /// The `packages` key node's module lookup picks for `dep_name` required /// from the package at `from`: `/node_modules/`, then the same /// under each ancestor directory, up to the project root. @@ -448,10 +484,10 @@ fn resolved_is_non_registry(resolved: &str) -> bool { return true; } match resolved.strip_prefix("file:") { - Some(path) => { - let path = path.trim_start_matches("./"); - !path.starts_with(&format!("{SOCKET_DIR}/vendor/")) - } + Some(path) => !path + .trim_start_matches("./") + .strip_prefix(SOCKET_DIR) + .is_some_and(|rest| rest.starts_with("/vendor/")), None => false, } } @@ -479,8 +515,11 @@ pub(crate) fn npm_spec_is_registry(spec: &str) -> bool { if spec.starts_with('.') { return false; } - let lower = spec.to_ascii_lowercase(); - !(lower.ends_with(".tgz") || lower.ends_with(".tar.gz") || lower.ends_with(".tar")) + let has_suffix = |suffix: &str| { + spec.len() >= suffix.len() + && spec.as_bytes()[spec.len() - suffix.len()..].eq_ignore_ascii_case(suffix.as_bytes()) + }; + !(has_suffix(".tgz") || has_suffix(".tar.gz") || has_suffix(".tar")) } #[cfg(test)] @@ -489,6 +528,45 @@ mod tests { use super::*; + /// #753: every entry under a `hasShrinkwrap` package maps to that + /// (outermost) package; the package itself, its siblings and workspace + /// members' own entries do not. + #[test] + fn shrinkwrapped_entries_are_the_descendants_of_a_has_shrinkwrap_package() { + let lock = json!({ + "lockfileVersion": 3, + "packages": { + "": {}, + "node_modules/@bh/sw": { "version": "1.0.0", "hasShrinkwrap": true }, + "node_modules/@bh/sw/node_modules/left-pad": { "version": "1.3.0" }, + "node_modules/@bh/sw/node_modules/a": { "version": "1.0.0", "hasShrinkwrap": true }, + "node_modules/@bh/sw/node_modules/a/node_modules/b": { "version": "1.0.0" }, + "node_modules/left-pad": { "version": "1.1.3" }, + "node_modules/other": { "version": "1.0.0", "hasShrinkwrap": false }, + "node_modules/other/node_modules/left-pad": { "version": "1.3.0" }, + "packages/ws": { "version": "1.0.0" }, + "packages/ws/node_modules/left-pad": { "version": "1.3.0" } + } + }); + let got = npm_shrinkwrapped_entries(&lock); + let want: BTreeMap = [ + ( + "node_modules/@bh/sw/node_modules/left-pad", + "node_modules/@bh/sw", + ), + ("node_modules/@bh/sw/node_modules/a", "node_modules/@bh/sw"), + ( + "node_modules/@bh/sw/node_modules/a/node_modules/b", + "node_modules/@bh/sw", + ), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + assert_eq!(got, want); + assert!(npm_shrinkwrapped_entries(&json!({"dependencies": {}})).is_empty()); + } + #[test] fn registry_specs_are_recognized() { for spec in [ @@ -529,6 +607,9 @@ mod tests { "/abs/left-pad", "~/left-pad", "left-pad-1.3.0.tgz", + "left-pad-1.3.0.TGZ", + "left-pad-1.3.0.Tar.Gz", + "left-pad-1.3.0.tar", "C:\\pkgs\\left-pad.tgz", "npm:left-pad@github:stevemao/left-pad", ] { @@ -539,6 +620,29 @@ mod tests { } } + /// socket-patch's own vendored wiring (with or without `./`) is not a + /// local source; any other `file:` path, git and git hosts are. + #[test] + fn non_registry_resolved_spares_only_socket_vendor_paths() { + for resolved in [ + "file:.socket/vendor/npm/u/left-pad-1.3.0.tgz", + "file:./.socket/vendor/npm/u/left-pad-1.3.0.tgz", + "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + ] { + assert!(!resolved_is_non_registry(resolved), "{resolved:?}"); + } + for resolved in [ + "file:.socketx/vendor/left-pad-1.3.0.tgz", + "file:.socket/vendorx/left-pad-1.3.0.tgz", + "file:.socket", + "file:../left-pad", + "git+ssh://git@github.com/stevemao/left-pad.git#ff8e7ba", + "github:stevemao/left-pad", + ] { + assert!(resolved_is_non_registry(resolved), "{resolved:?}"); + } + } + fn lock(packages: Value) -> Value { json!({ "lockfileVersion": 3, "packages": packages }) } diff --git a/crates/socket-patch-core/src/vendor/pnpm_lock.rs b/crates/socket-patch-core/src/vendor/pnpm_lock.rs index 706fb239a..1fbcf7ebb 100644 --- a/crates/socket-patch-core/src/vendor/pnpm_lock.rs +++ b/crates/socket-patch-core/src/vendor/pnpm_lock.rs @@ -63,18 +63,15 @@ use crate::utils::fs::{ }; use crate::utils::socket_dir::remove_tree_and_prune; -use super::common::{already_patched_result, done, parse_json_manifest, refused, JsonLayout}; +use super::common::{parse_json_manifest, refused, JsonLayout}; use super::npm_common::{ - done_failure_unstage, gate_packages, guard_coordinates, guard_revert_uuid_dir, refusal_code, - stage_patch_pack, tgz_rel_leaf, + gate_packages, guard_revert_uuid_dir, refusal_code, tgz_rel_leaf, vendor_npm_family, NpmCommit, + NpmCoords, NpmLockBackend, NpmStagedPack, NpmVendorRequest, WireCx, }; use super::parse_memo::ParseMemo; use super::path::parse_vendor_path; use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, PnpmMeta, VendorArtifact, VendorEntry, VendorMarker, WiringAction, - WiringRecord, -}; +use super::state::{PnpmMeta, VendorEntry, WiringAction, WiringRecord}; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; use crate::constants::npm_family::PNPM_LOCK; use crate::formats::pnpm::lines::{ @@ -178,252 +175,189 @@ pub(super) async fn vendor_pnpm_dialect( service: Option<&super::VendorServiceConfig>, dialect: PnpmDialect, ) -> VendorOutcome { - let mut warnings: Vec = Vec::new(); - - // ── 1. Coordinates (shared fail-closed guard) ───────────────────────── - let coords = match guard_coordinates(purl, record) { - Ok(coords) => coords, - Err(outcome) => return *outcome, - }; - let (name, version) = (coords.name.as_str(), coords.version.as_str()); - let rel_tgz = format!("{}/{}", coords.uuid_dir_rel, tgz_rel_leaf(name, version)); - // pnpm spells the override target `file:` with NO - // `./` (spike P1 fixtures, verbatim). - let spec = format!("file:{rel_tgz}"); - let override_key = format!("{name}@{version}"); - - // ── 2. Read the pair (refuse before any write) ─────────────────────── - let project = match read_project(project_root, dialect).await { - Ok(project) => project, - Err(outcome) => return *outcome, - }; - - // ── 3. Pre-flight refusals (override conflicts, entry present) ─────── - let effective_key = match preflight_package(&project, name, version, &override_key) { - Ok(key) => key, - Err(outcome) => return *outcome, - }; - let PnpmProject { - pkg_bytes, - mut pkg, - mut lock, - ws_text, - } = project; - - // ── 4. Stage → patch → pack (shared flavor-agnostic pipeline) ──────── - let (staged, result) = match stage_patch_pack( - purl, - installed_dir, - project_root, - record, - sources, - dry_run, - force, - &mut warnings, - service, + vendor_npm_family( + &PnpmBackend { dialect }, + NpmVendorRequest { + purl, + installed_dir, + project_root, + record, + sources, + vendored_at, + dry_run, + force, + service, + }, ) .await - { - Ok(pair) => pair, - Err(outcome) => return *outcome, - }; - let Some(staged) = staged else { - // Failed patch or dry run: wiring never ran, project byte-untouched. - return done(result, None, warnings); - }; - let uuid_dir_preexisted = staged.uuid_dir_preexisted; - debug_assert_eq!(staged.rel_tgz, rel_tgz); - let packed = staged.packed; - if staged.staged_pkg_json.is_some() { - // pnpm snapshots mirror the package's own dependency maps; the spike - // has no fixture for a manifest-rewriting patch, so the mirrors are - // preserved verbatim and the user is told to re-resolve. - warnings.push(VendorWarning::new( - "vendor_dep_manifest_stale", - format!( - "the patch rewrites {name}@{version}'s package.json; pnpm-lock.yaml's \ - dependency mirrors were preserved verbatim — if the patch changed \ - dependency ranges, run `pnpm install` to re-resolve them" - ), - )); - } +} - // ── 5. Compute both edits in memory (nothing written yet) ──────────── - let ctx = EditCtx { - name, - version, - rel_tgz: &rel_tgz, - spec: &spec, - integrity: &packed.integrity, - override_key: &effective_key, - }; - let mut wiring: Vec = Vec::new(); - - // The package.json copy is a back-compat surface for pnpm that reads - // overrides only from package.json. When the lock shows the project's - // pnpm reads them from pnpm-workspace.yaml, a new package.json - // `pnpm.overrides` would REPLACE the user's workspace overrides on - // pnpm 10 (#360), so only the workspace file and the lock are wired. - let skip_pkg_copy = dialect == PnpmDialect::V9 - && workspace_overrides_govern(&pkg, ws_text.as_deref(), lock.lines()); - let (pkg_changed, created_pnpm_table, created_overrides_table) = if skip_pkg_copy { - (false, false, false) - } else { - match apply_pkg_override(&mut pkg, &effective_key, &spec, &mut wiring) { - Ok(out) => out, - Err(e) => { - return done_failure_unstage( - purl, - e, - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } - } - }; - let (lock_changed, lock_warning) = match lock.edit(&ctx, &mut wiring) { - Ok(edit) => edit, - Err(e) => { - return done_failure_unstage( - purl, - format!("{PNPM_LOCK} surgery failed: {e}"), - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } - }; +/// The pnpm half of [`vendor_pnpm`] (both lock dialects). +struct PnpmBackend { + dialect: PnpmDialect, +} - // Only modern locks mirror overrides into pnpm-workspace.yaml. Legacy - // pnpm reads package.json alone; creating a workspace changes its mode. - let ws_edit = if dialect == PnpmDialect::V9 { - match apply_workspace_override(ws_text.as_deref(), &effective_key, &spec, &mut wiring) { - Ok(edit) => edit, - Err(e) => { - return done_failure_unstage( - purl, - format!("{PNPM_WORKSPACE} surgery failed: {e}"), - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } - } - } else { - WorkspaceEdit::default() - }; +/// [`PnpmBackend`]'s pre-flight product: the gated project and the +/// override key the wiring writes under. +struct PnpmPlan { + project: PnpmProject, + effective_key: String, +} - if !pkg_changed && !lock_changed && ws_edit.new_text.is_none() { - // Everything already carries this uuid + the packed integrity: the - // project is in sync. The integrity is that of the reused committed - // tarball (the shared pipeline wrote nothing) or, when reuse missed, - // of a fresh acquisition that reproduced the pinned bytes; - // synthesize AlreadyPatched and record nothing (the existing ledger - // entry stays authoritative). - return done( - already_patched_result(purl, &project_root.join(&rel_tgz), &record.files), - None, - warnings, - ); +impl NpmLockBackend for PnpmBackend { + type Plan = PnpmPlan; + + fn flavor(&self) -> Option<&'static str> { + Some(self.dialect.flavor()) } - if let Some(warning) = lock_warning { - warnings.push(warning); + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + _warnings: &mut Vec, + ) -> Result> { + let override_key = format!("{}@{}", coords.name, coords.version); + + // ── 2. Read the pair (refuse before any write) ─────────────────── + let project = read_project(project_root, self.dialect).await?; + + // ── 3. Pre-flight refusals (override conflicts, entry present) ─── + let effective_key = + preflight_package(&project, &coords.name, &coords.version, &override_key)?; + Ok(PnpmPlan { + project, + effective_key, + }) } - // ── 6. Commit: package.json + pnpm-workspace.yaml FIRST, lock second, - // unwind the override surfaces on a lock failure (P3 desync safety). - // Re-render package.json in its own layout (BOM, indent, line ending, - // trailer) so a Windows / autocrlf manifest diffs only in the override. - let new_pkg_bytes = match JsonLayout::of(&String::from_utf8_lossy(&pkg_bytes)).render(&pkg) { - Ok(bytes) => bytes, - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot serialize {PACKAGE_JSON}: {e}"), - project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, - ) - .await + async fn wire( + &self, + plan: PnpmPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + warnings: &mut Vec, + ) -> Result, String> { + let dialect = self.dialect; + let project_root = cx.project_root; + let PnpmPlan { + project: + PnpmProject { + pkg_bytes, + mut pkg, + mut lock, + ws_text, + }, + effective_key, + } = plan; + let rel_tgz = staged.rel_tgz.as_str(); + // pnpm spells the override target `file:` with + // NO `./` (spike P1 fixtures, verbatim). + let spec = format!("file:{rel_tgz}"); + + // ── 5. Compute both edits in memory (nothing written yet) ──────── + let ctx = EditCtx { + name: &cx.coords.name, + version: &cx.coords.version, + rel_tgz, + spec: &spec, + integrity: &staged.packed.integrity, + override_key: &effective_key, + }; + let mut wiring: Vec = Vec::new(); + + // The package.json copy is a back-compat surface for pnpm that reads + // overrides only from package.json. When the lock shows the + // project's pnpm reads them from pnpm-workspace.yaml, a new + // package.json `pnpm.overrides` would REPLACE the user's workspace + // overrides on pnpm 10 (#360), so only the workspace file and the + // lock are wired. + let skip_pkg_copy = dialect == PnpmDialect::V9 + && workspace_overrides_govern(&pkg, ws_text.as_deref(), lock.lines()); + let (pkg_changed, created_pnpm_table, created_overrides_table) = if skip_pkg_copy { + (false, false, false) + } else { + apply_pkg_override(&mut pkg, &effective_key, &spec, &mut wiring)? + }; + let (lock_changed, lock_warning) = lock + .edit(&ctx, &mut wiring) + .map_err(|e| format!("{PNPM_LOCK} surgery failed: {e}"))?; + + // Only modern locks mirror overrides into pnpm-workspace.yaml. + // Legacy pnpm reads package.json alone; creating a workspace changes + // its mode. + let ws_edit = if dialect == PnpmDialect::V9 { + apply_workspace_override(ws_text.as_deref(), &effective_key, &spec, &mut wiring) + .map_err(|e| format!("{PNPM_WORKSPACE} surgery failed: {e}"))? + } else { + WorkspaceEdit::default() + }; + + if !pkg_changed && !lock_changed && ws_edit.new_text.is_none() { + // Everything already carries this uuid + the packed integrity: + // the project is in sync. + return Ok(None); } - }; - let lock_out = lock.lines().join("\n"); - if let Err(e) = commit_surfaces( - project_root, - pkg_changed.then_some(new_pkg_bytes.as_slice()), - &pkg_bytes, - ws_edit.new_text.as_deref().map(str::as_bytes), - ws_text.as_deref().map(str::as_bytes), - ws_edit.created_file, - lock_changed.then_some(lock_out.as_bytes()), - ) - .await - { - return done_failure_unstage( - purl, - e, + + if let Some(warning) = lock_warning { + warnings.push(warning); + } + + // ── 6. Commit: package.json + pnpm-workspace.yaml FIRST, lock + // second, unwind the override surfaces on a lock failure (P3 + // desync safety). Re-render package.json in its own layout (BOM, + // indent, line ending, trailer) so a Windows / autocrlf manifest + // diffs only in the override. + let new_pkg_bytes = JsonLayout::of(&String::from_utf8_lossy(&pkg_bytes)) + .render(&pkg) + .map_err(|e| format!("cannot serialize {PACKAGE_JSON}: {e}"))?; + let lock_out = lock.lines().join("\n"); + commit_surfaces( project_root, - &coords.uuid_dir_rel, - uuid_dir_preexisted, + pkg_changed.then_some(new_pkg_bytes.as_slice()), + &pkg_bytes, + ws_edit.new_text.as_deref().map(str::as_bytes), + ws_text.as_deref().map(str::as_bytes), + ws_edit.created_file, + lock_changed.then_some(lock_out.as_bytes()), ) - .await; - } - if lock_changed { - // Re-seed the memo with the lock just written, so the next package - // reads it back without re-splitting it. Only when the split of - // those bytes is provably these lines (no line carries a `\n`); - // otherwise the next read simply misses. - if let ProjectLock::V9(LockLines::Owned(written)) = lock { - if !written.iter().any(|l| l.contains('\n')) { - LOCK_MEMO.store(lock_out.into_bytes(), LockDoc::new(written)); + .await?; + if lock_changed { + // Re-seed the memo with the lock just written, so the next + // package reads it back without re-splitting it. Only when the + // split of those bytes is provably these lines (no line carries + // a `\n`); otherwise the next read simply misses. + if let ProjectLock::V9(LockLines::Owned(written)) = lock { + if !written.iter().any(|l| l.contains('\n')) { + LOCK_MEMO.store(lock_out.into_bytes(), LockDoc::new(written)); + } } } + Ok(Some(NpmCommit { + wiring, + pnpm: Some(PnpmMeta { + created_overrides_table, + created_pnpm_table, + created_workspace_file: ws_edit.created_file, + created_workspace_overrides: ws_edit.created_overrides, + }), + ..NpmCommit::default() + })) } - // ── 7. Marker + ledger entry ───────────────────────────────────────── - let marker = VendorMarker::new("npm", &coords.base_purl, record, vendored_at); - write_marker_or_warn( - &project_root.join(&coords.uuid_dir_rel), - &marker, - &mut warnings, - ) - .await; - - let entry = VendorEntry { - ecosystem: "npm".to_string(), - base_purl: coords.base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: rel_tgz, - sha256: packed.sha256_hex, - size: Some(packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some(dialect.flavor().to_string()), - uv: None, - pnpm: Some(PnpmMeta { - created_overrides_table, - created_pnpm_table, - created_workspace_file: ws_edit.created_file, - created_workspace_overrides: ws_edit.created_overrides, - }), - poetry: None, - pdm: None, - pipenv: None, - }; - done(result, Some(entry), warnings) + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning { + // pnpm snapshots mirror the package's own dependency maps; the spike + // has no fixture for a manifest-rewriting patch, so the mirrors are + // preserved verbatim and the user is told to re-resolve. + VendorWarning::new( + "vendor_dep_manifest_stale", + format!( + "the patch rewrites {name}@{version}'s package.json; pnpm-lock.yaml's \ + dependency mirrors were preserved verbatim — if the patch changed \ + dependency ranges, run `pnpm install` to re-resolve them" + ), + ) + } } /// The pair the pnpm wiring edits, read and structurally gated before any @@ -7965,6 +7899,39 @@ snapshots: .contains(&format!(" left-pad@file:{}:", fx.rel_tgz()))); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture_with(P1_BEFORE_PKG, P1_BEFORE_LOCK).await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = br#"{"name":"left-pad","version":"1.3.0","sideEffects":false}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + // ── parser + revert-uuid guard micro edges ────────────────────────────── #[test] diff --git a/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs b/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs index 660c980d2..7af6a71d0 100644 --- a/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs +++ b/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs @@ -3281,6 +3281,39 @@ packages: ); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture_with(T_BEFORE_PKG, T7_BEFORE_LOCK).await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = br#"{"name":"left-pad","version":"1.3.0","sideEffects":false}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + /// A non-object package.json refuses before any write. #[tokio::test] async fn non_object_package_json_refuses() { diff --git a/crates/socket-patch-core/src/vendor/service_fetch.rs b/crates/socket-patch-core/src/vendor/service_fetch.rs index 283eb1733..f19f477fc 100644 --- a/crates/socket-patch-core/src/vendor/service_fetch.rs +++ b/crates/socket-patch-core/src/vendor/service_fetch.rs @@ -224,6 +224,26 @@ impl<'a> ServicePolicy<'a> { })) } + /// Refuse a patch the service has no artifact for (`code` is + /// [`super::VENDOR_PREBUILT_PENDING`] or [`super::VENDOR_PREBUILT_UNAVAILABLE`]): + /// the same hard failure as [`Self::hard`], except that the npm backends' + /// failed `Done` also carries `code` as a warning, so the vendor loop can + /// tell "not served (yet)" from a real failure and keep an older vendored + /// patch of the same package instead of failing the run (#954). + fn unserved(&self, code: &'static str, detail: String) -> ServiceAttempt { + match self.terminal { + ServiceTerminal::Refused => self.hard("vendor_prebuilt_required", detail), + ServiceTerminal::Failure(purl) => { + let warning = VendorWarning::new(code, detail.clone()); + ServiceAttempt::HardFail(Box::new(super::common::done( + super::common::failed_result(purl, std::path::Path::new(""), detail), + None, + vec![warning], + ))) + } + } + } + /// Refuse an unavailable server artifact. pub(crate) fn miss( &self, @@ -254,14 +274,13 @@ impl<'a> ServicePolicy<'a> { refusing to fall back to a local build on tampered bytes" ), )), - ServiceArtifact::Pending => Err(self.miss( - warnings, - "vendor_prebuilt_pending", + ServiceArtifact::Pending => Err(self.unserved( + super::VENDOR_PREBUILT_PENDING, format!("prebuilt {noun} is still building"), )), // No artifact is available for these coordinates or entitlements. - ServiceArtifact::Unavailable(reason) => Err(self.hard( - "vendor_prebuilt_required", + ServiceArtifact::Unavailable(reason) => Err(self.unserved( + super::VENDOR_PREBUILT_UNAVAILABLE, format!("prebuilt {noun} unavailable: {reason}"), )), ServiceArtifact::Failed(reason) => Err(self.miss( diff --git a/crates/socket-patch-core/src/vendor/state.rs b/crates/socket-patch-core/src/vendor/state.rs index 4919f33cf..aad7a0687 100644 --- a/crates/socket-patch-core/src/vendor/state.rs +++ b/crates/socket-patch-core/src/vendor/state.rs @@ -80,6 +80,35 @@ pub struct VendorArtifact { pub file_inventory: Option>, } +impl VendorArtifact { + /// The packed npm tarball at `rel_tgz`, pinned by its sha256 and size. + /// `yarn_berry10c0` stays `None`: only the yarn-berry flavor records + /// the checksum, and only when it is the service's own. + pub(crate) fn tarball(rel_tgz: String, packed: &super::npm_pack::PackedTarball) -> Self { + Self { + yarn_berry10c0: None, + path: rel_tgz, + sha256: packed.sha256_hex.clone(), + size: Some(packed.size), + platform_locked: None, + file_inventory: None, + } + } + + /// A vendored package DIRECTORY at `rel_dir` (vlt): no file hash or + /// size, its whole-tree `inventory` instead. + pub(crate) fn dir(rel_dir: String, inventory: BTreeMap) -> Self { + Self { + yarn_berry10c0: None, + path: rel_dir, + sha256: String::new(), + size: None, + platform_locked: None, + file_inventory: Some(inventory), + } + } +} + /// How a wiring edit changed a file. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "camelCase")] @@ -283,6 +312,36 @@ pub struct VendorEntry { } impl VendorEntry { + /// The ledger entry every npm-family vendor backend records: ecosystem + /// `npm`, the wiring flavor (`None` is package-lock's pre-flavor + /// spelling) and no other ecosystem's extras. A flavor sets its one + /// meta field afterwards (`pnpm`, `artifact.yarn_berry10c0`). + pub(crate) fn npm( + base_purl: String, + uuid: String, + artifact: VendorArtifact, + wiring: Vec, + flavor: Option<&str>, + ) -> Self { + Self { + ecosystem: "npm".to_string(), + base_purl, + uuid, + artifact, + wiring, + lock: None, + took_over_go_patches: false, + flavor: flavor.map(str::to_string), + uv: None, + pnpm: None, + poetry: None, + pdm: None, + pipenv: None, + detached: false, + record: None, + } + } + /// Whether this entry's committed artifact is on disk under /// `project_root` — for a FILE artifact (wheel, tarball: a recorded /// `sha256`), only when its bytes still hash to that pin; a copy dir @@ -2137,4 +2196,99 @@ mod tests { assert!(!b.join(VENDOR_STATE_REL).exists(), "grouped {grouped}"); } } + + /// #922: the npm-family constructor serializes to exactly the ledger + /// JSON the yarn-classic backend's literal entry did. + #[test] + fn npm_constructor_matches_the_yarn_classic_literal() { + let packed = super::super::npm_pack::PackedTarball::from_bytes(b"tarball bytes"); + let rel = ".socket/vendor/npm/9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f/left-pad-1.3.0.tgz"; + let wiring = vec![WiringRecord { + file: "yarn.lock".to_string(), + kind: "yarn_lock_block".to_string(), + action: WiringAction::Rewritten, + key: Some("left-pad@^1.3.0".to_string()), + original: Some(serde_json::json!(["left-pad@^1.3.0:"])), + new: Some(serde_json::json!([ + "left-pad@^1.3.0:", + " version \"1.3.0\"" + ])), + }]; + let literal = VendorEntry { + ecosystem: "npm".to_string(), + base_purl: "pkg:npm/left-pad@1.3.0".to_string(), + uuid: "9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f".to_string(), + artifact: VendorArtifact { + yarn_berry10c0: None, + path: rel.to_string(), + sha256: packed.sha256_hex.clone(), + size: Some(packed.size), + platform_locked: None, + file_inventory: None, + }, + wiring: wiring.clone(), + lock: None, + took_over_go_patches: false, + detached: false, + record: None, + flavor: Some("yarn-classic".to_string()), + uv: None, + pnpm: None, + poetry: None, + pdm: None, + pipenv: None, + }; + let built = VendorEntry::npm( + "pkg:npm/left-pad@1.3.0".to_string(), + "9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f".to_string(), + VendorArtifact::tarball(rel.to_string(), &packed), + wiring, + Some("yarn-classic"), + ); + assert_eq!(built, literal); + assert_eq!( + serde_json::to_string_pretty(&built).unwrap(), + serde_json::to_string_pretty(&literal).unwrap() + ); + // The pinned JSON: no other ecosystem's extras, no + // `yarnBerry10c0`, no `fileInventory`. + assert_eq!( + serde_json::to_value(&built).unwrap(), + serde_json::json!({ + "ecosystem": "npm", + "basePurl": "pkg:npm/left-pad@1.3.0", + "uuid": "9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f", + "artifact": {"path": rel, "sha256": packed.sha256_hex, "size": packed.size}, + "wiring": [{ + "file": "yarn.lock", + "kind": "yarn_lock_block", + "action": "rewritten", + "key": "left-pad@^1.3.0", + "original": ["left-pad@^1.3.0:"], + "new": ["left-pad@^1.3.0:", " version \"1.3.0\""], + }], + "flavor": "yarn-classic", + }) + ); + + // package-lock keeps the pre-flavor spelling; vlt's dir artifact + // carries its inventory and no file hash. + assert_eq!( + VendorEntry::npm( + String::new(), + String::new(), + built.artifact.clone(), + vec![], + None + ) + .flavor, + None + ); + let inventory = BTreeMap::from([("index.js".to_string(), "ab".repeat(32))]); + let dir = VendorArtifact::dir("dir".to_string(), inventory.clone()); + assert_eq!( + serde_json::to_value(&dir).unwrap(), + serde_json::json!({"path": "dir", "fileInventory": inventory}) + ); + } } diff --git a/crates/socket-patch-core/src/vendor/vlt_lock.rs b/crates/socket-patch-core/src/vendor/vlt_lock.rs index 3939052bd..5a6752302 100644 --- a/crates/socket-patch-core/src/vendor/vlt_lock.rs +++ b/crates/socket-patch-core/src/vendor/vlt_lock.rs @@ -31,14 +31,11 @@ use crate::utils::socket_dir::remove_tree_and_prune; use super::common::{already_patched_result, done, refused}; use super::npm_common::{ - done_failure_unstage, guard_coordinates, guard_revert_uuid_dir, parse_npm_purl, + done_failure_unstage, finish_vendored, guard_coordinates, guard_revert_uuid_dir, parse_npm_purl, }; use super::npm_dir::{dependency_token, replace_dependency_token, stage_patch_dir, SpanError}; use super::source::PackageSource; -use super::state::{ - load_state, write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, - WiringRecord, -}; +use super::state::{load_state, VendorArtifact, VendorEntry, WiringAction, WiringRecord}; use super::vlt_lock_text::{ brotli_for_slot3, edges_block, entry_text, file_dep_id, has_brotli_flag, installs_outside_registry, is_default_registry, is_importer_dep_id, is_registry_url_segment, @@ -1342,6 +1339,28 @@ pub(crate) async fn vendor_vlt<'a>( if staged.links_dropped { warnings.push(links_dropped_warning(name, version)); } + let entry_for = |wiring: Vec| { + VendorEntry::npm( + coords.base_purl.clone(), + record.uuid.clone(), + VendorArtifact::dir(staged.rel_dir.clone(), staged.inventory.clone()), + wiring, + Some(FLAVOR), + ) + }; + + if wiring.is_none() && staged.reused { + // In sync: the committed dir was reused and the wiring already + // points at it. Record nothing. + let rel_abs = project_root.join(&staged.rel_dir); + return done( + already_patched_result(purl, &rel_abs, &record.files), + None, + warnings, + ); + } + // Once per run that records an entry, after the in-sync return (the + // tarball flavors' rule, see `npm_common::vendor_npm_family`). if staged.staged_pkg_json.is_some() { warnings.push(VendorWarning::new( "vendor_dep_manifest_stale", @@ -1353,42 +1372,7 @@ pub(crate) async fn vendor_vlt<'a>( ), )); } - let entry_for = |wiring: Vec| VendorEntry { - ecosystem: "npm".to_string(), - base_purl: coords.base_purl.clone(), - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: staged.rel_dir.clone(), - sha256: String::new(), - size: None, - platform_locked: None, - file_inventory: Some(staged.inventory.clone()), - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some(FLAVOR.to_string()), - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }; - let marker = VendorMarker::new("npm", &coords.base_purl, record, vendored_at); - let uuid_dir = project_root.join(&coords.uuid_dir_rel); - let Some(wiring) = wiring else { - if staged.reused { - let rel_abs = project_root.join(&staged.rel_dir); - return done( - already_patched_result(purl, &rel_abs, &record.files), - None, - warnings, - ); - } warnings.push(VendorWarning::new( "vendor_artifact_rebuilt", format!( @@ -1397,8 +1381,17 @@ pub(crate) async fn vendor_vlt<'a>( staged.rel_dir ), )); - write_marker_or_warn(&uuid_dir, &marker, &mut warnings).await; - return done(result, Some(entry_for(Vec::new())), warnings); + let entry = entry_for(Vec::new()); + return finish_vendored( + project_root, + &coords, + record, + vendored_at, + result, + entry, + warnings, + ) + .await; }; if let Err(e) = commit(project_root, &wiring, &analysis.pkgs).await { return done_failure_unstage( @@ -1410,8 +1403,17 @@ pub(crate) async fn vendor_vlt<'a>( ) .await; } - write_marker_or_warn(&uuid_dir, &marker, &mut warnings).await; - done(result, Some(entry_for(wiring.records)), warnings) + let entry = entry_for(wiring.records); + finish_vendored( + project_root, + &coords, + record, + vendored_at, + result, + entry, + warnings, + ) + .await } /// Which of `packages` [`vendor_vlt`] would refuse before it first asks @@ -3284,6 +3286,62 @@ mod tests { ); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let fx = fx(&basic_lock(), &[(PACKAGE_JSON, ROOT_PKG)]).await; + let patched_pkg = b"{\"name\":\"left-pad\",\"version\":\"1.3.0\",\"main\":\"i.js\"}"; + tokio::fs::write( + fx.blobs.join(compute_git_sha256_from_bytes(patched_pkg)), + patched_pkg, + ) + .await + .unwrap(); + let mut rec = record(UUID); + rec.files.insert( + "package/package.json".into(), + PatchFileInfo { + before_hash: String::new(), + after_hash: compute_git_sha256_from_bytes(patched_pkg), + }, + ); + let sources = PatchSources::blobs_only(&fx.blobs); + let vendor = || { + crate::vendor::test_support::vendor_vlt( + PURL, + &fx.installed, + &fx.root, + &rec, + &sources, + "t", + false, + true, + None, + ) + }; + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + let (entry, warnings) = entry_of(vendor().await); + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + persist(&fx, &entry).await; + + let VendorOutcome::Done { + result, + entry, + warnings, + } = vendor().await + else { + panic!("expected Done"); + }; + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + #[tokio::test] async fn a_ledgerless_entry_verifies_a_stripped_manifest_by_its_blob() { let fx = fx(&basic_lock(), &[(PACKAGE_JSON, ROOT_PKG)]).await; diff --git a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs index da654820e..13596dfb2 100644 --- a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs @@ -46,12 +46,10 @@ use sha2::{Digest, Sha256, Sha512}; use crate::constants::SOCKET_DIR; use crate::formats::yarn::berry_entry::{manifest_bin, render_pinned_entry, Pin}; use crate::formats::yarn::berry_gates::{self, BerryGate, Yarnrc, SUPPORTED_CACHE_KEY}; -use crate::formats::yarn::blocks::{ - berry_field, block_eol, replace_block, scan_blocks, LockBlock, -}; +use crate::formats::yarn::blocks::{berry_field, block_eol, replace_block, scan_blocks, LockBlock}; use crate::formats::yarn::patterns::{pattern_real_name, split_berry_key_patterns, split_pattern}; use crate::manifest::schema::PatchRecord; -use crate::patch::apply::{normalize_file_path, PatchSources}; +use crate::patch::apply::PatchSources; use crate::utils::fs::{ atomic_write_bytes_preserving_mode, read_regular_to_bytes, read_regular_to_string, }; @@ -60,16 +58,15 @@ use crate::utils::uri::encode_uri_component; #[cfg(test)] use super::berry_zip::berry_cache_checksum_10c0; -use super::common::{already_patched_result, parse_json_manifest, refused, JsonLayout}; +use super::common::{parse_json_manifest, refused, JsonLayout}; use super::npm_common::{ - done_failure_unstage, guard_coordinates, guard_revert_uuid_dir, stage_patch_pack, tgz_rel_leaf, + guard_revert_uuid_dir, vendor_npm_family, NpmCommit, NpmCoords, NpmLockBackend, NpmStagedPack, + NpmVendorRequest, WireCx, }; use super::parse_memo::ParseMemo; use super::path::parse_vendor_path; use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, -}; +use super::state::{VendorEntry, WiringAction, WiringRecord}; use super::yarn_classic_lock::{ forget_block_scans, lines_to_json, read_yarn_lock, revert_recorded_block, scan_blocks_shared, }; @@ -89,7 +86,9 @@ const KIND_LOCK_ENTRY: &str = "yarn_berry_lock_entry"; /// Vendor one installed npm package into a yarn-berry (4.x, cacheKey 10c0) /// project. Same contract as [`super::npm_lock::vendor_npm`]: refuse-early, -/// wire-last; `entry` is `None` for dry runs and the in-sync re-run. +/// wire-last; `entry` is `None` for dry runs and the in-sync re-run. The +/// flow is [`vendor_npm_family`]'s; [`YarnBerryBackend`] is the berry lock +/// grammar. #[allow(clippy::too_many_arguments)] pub async fn vendor_yarn_berry<'a>( purl: &str, @@ -102,395 +101,353 @@ pub async fn vendor_yarn_berry<'a>( force: bool, service: Option<&super::VendorServiceConfig>, ) -> VendorOutcome { - let installed_dir = installed_dir.into(); - let mut warnings: Vec = Vec::new(); + vendor_npm_family( + &YarnBerryBackend, + NpmVendorRequest { + purl, + installed_dir: installed_dir.into(), + project_root, + record, + sources, + vendored_at, + dry_run, + force, + service, + }, + ) + .await +} - // ── 1. Coordinates (shared fail-closed guard, before any disk access) ─ - let coords = match guard_coordinates(purl, record) { - Ok(coords) => coords, - Err(outcome) => return *outcome, - }; - let (name, version) = (coords.name.as_str(), coords.version.as_str()); - let uuid_dir_rel = coords.uuid_dir_rel.clone(); - let base_purl = coords.base_purl.clone(); - let rel_tgz = format!("{}/{}", coords.uuid_dir_rel, tgz_rel_leaf(name, version)); - // The resolutions spec — `file:./` spelling per the B3 fixture. - let spec = format!("file:./{rel_tgz}"); - - // ── 2. Lockfile + cacheKey gate ─────────────────────────────────────── - let lock_text = match read_yarn_lock(project_root).await { - Ok(t) => t, - Err(outcome) => return *outcome, - }; - // A uniformly CRLF lock (what yarn writes on Windows) is spliced in its - // own line ending below; only a mixed one is refused. - if let Some(outcome) = refuse_mixed_line_endings(YARN_LOCK, &lock_text) { - return outcome; - } - let blocks = scan_blocks_shared(&lock_text); - if let Some(outcome) = refuse_unsupported_cache(&lock_text) { - return outcome; - } +/// The yarn-berry half of [`vendor_yarn_berry`]. +struct YarnBerryBackend; - // ── 3. .yarnrc.yml knobs that change the checksum ─────────────────── - if let Some(outcome) = refuse_unsupported_compression(project_root).await { - return outcome; - } +/// [`YarnBerryBackend`]'s pre-flight product: the gated lock and project +/// `package.json`, and the one lock entry the wiring replaces. +struct YarnBerryPlan { + lock_text: String, + blocks: std::sync::Arc>, + /// The root workspace's name (the lock key/resolution embed it). + workspace: String, + pkg_bytes: Vec, + pkg: std::sync::Arc, + /// A taken-over user `resolutions` pin's value (see + /// [`resolutions_gate`]). + takeover_original: Option, + /// Index of the replaceable entry in `blocks`. + target_idx: usize, + /// The entry is already one of our `file:` entries. + target_is_ours: bool, +} - // ── 4. Root workspace name (the lock key/resolution embed it) ──────── - let workspace = match root_workspace_gate(&blocks) { - Ok(workspace) => workspace, - Err(outcome) => return *outcome, - }; +impl NpmLockBackend for YarnBerryBackend { + type Plan = YarnBerryPlan; - // ── 5. package.json + user-override conflict gate ───────────────────── - let pkg_path = project_root.join(PACKAGE_JSON); - let pkg_bytes = match read_regular_to_bytes(&pkg_path).await { - Ok(b) => b, - Err(e) => { - return refused( - "vendor_yarn_berry_manifest_unreadable", - format!("cannot read the project {PACKAGE_JSON}: {e}"), - ); - } - }; - // Its layout (BOM, indent, line ending, trailing newline) carries into - // the rewritten bytes; a mixed-ending file has none to carry. - let pkg_text = String::from_utf8_lossy(&pkg_bytes); - if let Some(outcome) = refuse_mixed_line_endings(PACKAGE_JSON, &pkg_text) { - return outcome; + fn flavor(&self) -> Option<&'static str> { + Some("yarn-berry") } - let pkg = match PKG_JSON_MEMO.parse(&pkg_bytes, || parse_json_manifest(&pkg_bytes)) { - Ok(v) => v, - Err(e) => { - return refused( - "vendor_yarn_berry_manifest_unreadable", - format!("{PACKAGE_JSON} is not parseable JSON: {e}"), - ); + + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result> { + let (name, version) = (coords.name.as_str(), coords.version.as_str()); + + // ── 2. Lockfile + cacheKey gate ─────────────────────────────────── + let lock_text = read_yarn_lock(project_root).await?; + // A uniformly CRLF lock (what yarn writes on Windows) is spliced in + // its own line ending below; only a mixed one is refused. + if let Some(outcome) = refuse_mixed_line_endings(YARN_LOCK, &lock_text) { + return Err(Box::new(outcome)); + } + let blocks = scan_blocks_shared(&lock_text); + if let Some(outcome) = refuse_unsupported_cache(&lock_text) { + return Err(Box::new(outcome)); } - }; - let Some(pkg_obj) = pkg.as_object() else { - return refused( - "vendor_yarn_berry_manifest_unreadable", - format!("{PACKAGE_JSON} root is not an object"), - ); - }; - let takeover_original = match resolutions_gate(pkg_obj, name, version) { - Ok(takeover_original) => takeover_original, - Err(outcome) => return *outcome, - }; - // ── 6. The single replaceable lock entry ────────────────────────────── - let scan = match scan_berry_target(&blocks, name, version) { - Ok(scan) => scan, - Err((code, detail)) => return refused(code, detail), - }; - // An `alias@npm:@…` descriptor consumes the patched package under - // a different ident; the bare-name resolutions entry vendoring writes - // can never move it, so that copy keeps installing the UNPATCHED bytes. - // Surface every such entry loudly instead of silently part-patching. - for key in &scan.alias_keys { - warnings.push(VendorWarning::new( - "vendor_alias_entry_skipped", - format!( - "{YARN_LOCK} entry `{key}` consumes {name}@{version} through an npm: alias; \ - the bare-name resolutions entry vendoring writes cannot move aliased \ - descriptors, so that copy keeps installing the UNPATCHED registry bytes" - ), - )); - } - let (target, target_is_ours) = match target_gate(&scan, name, version) { - Ok((idx, is_ours)) => (&blocks[idx], is_ours), - Err(outcome) => return *outcome, - }; - let patches_manifest = record - .files - .keys() - .any(|k| normalize_file_path(k) == "package.json"); - - // ── 7. Stage → patch → pack (shared flavor-agnostic pipeline) ───────── - let (staged, result) = match stage_patch_pack( - purl, - installed_dir, - project_root, - record, - sources, - dry_run, - force, - &mut warnings, - service, - ) - .await - { - Ok(pair) => pair, - Err(outcome) => return *outcome, - }; - let Some(staged) = staged else { - // Failed patch (wiring is last — project byte-untouched) or dry run. - return VendorOutcome::Done { - result, - entry: None, - warnings, - }; - }; - let uuid_dir_preexisted = staged.uuid_dir_preexisted; - debug_assert_eq!(staged.rel_tgz, rel_tgz); - let mut packed = staged.packed; - let dest = project_root.join(&rel_tgz); - - // ── 8. Berry identity facts of the packed tarball ───────────────────── - // A reuse hands over the exact bytes it verified; a fresh pack is - // re-read and must still be the bytes the pack hashed (the lock's - // checksum and `hash=` are derived from these, so a file swapped after - // verification must fail, never be pinned). - let reused = staged.verified_bytes.is_some(); - let tgz_bytes = match staged.verified_bytes { - Some(bytes) => bytes, - None => match tokio::fs::read(&dest).await { + // ── 3. .yarnrc.yml knobs that change the checksum ─────────────── + if let Some(outcome) = refuse_unsupported_compression(project_root).await { + return Err(Box::new(outcome)); + } + + // ── 4. Root workspace name (the lock key/resolution embed it) ──── + let workspace = root_workspace_gate(&blocks)?; + + // ── 5. package.json + user-override conflict gate ───────────────── + let pkg_bytes = match read_regular_to_bytes(&project_root.join(PACKAGE_JSON)).await { Ok(b) => b, Err(e) => { - return done_failure_unstage( - purl, - format!("cannot re-read the packed tarball: {e}"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await + return Err(Box::new(refused( + "vendor_yarn_berry_manifest_unreadable", + format!("cannot read the project {PACKAGE_JSON}: {e}"), + ))); } - }, - }; - if hex::encode(Sha256::digest(&tgz_bytes)) != packed.sha256_hex { - return done_failure_unstage( - purl, - format!("the packed tarball {rel_tgz} changed on disk after it was verified"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await; + }; + // Its layout (BOM, indent, line ending, trailing newline) carries + // into the rewritten bytes; a mixed-ending file has none to carry. + if let Some(outcome) = + refuse_mixed_line_endings(PACKAGE_JSON, &String::from_utf8_lossy(&pkg_bytes)) + { + return Err(Box::new(outcome)); + } + let pkg = match PKG_JSON_MEMO.parse(&pkg_bytes, || parse_json_manifest(&pkg_bytes)) { + Ok(v) => v, + Err(e) => { + return Err(Box::new(refused( + "vendor_yarn_berry_manifest_unreadable", + format!("{PACKAGE_JSON} is not parseable JSON: {e}"), + ))); + } + }; + let Some(pkg_obj) = pkg.as_object() else { + return Err(Box::new(refused( + "vendor_yarn_berry_manifest_unreadable", + format!("{PACKAGE_JSON} root is not an object"), + ))); + }; + let takeover_original = resolutions_gate(pkg_obj, name, version)?; + + // ── 6. The single replaceable lock entry ────────────────────────── + let scan = scan_berry_target(&blocks, name, version) + .map_err(|(code, detail)| Box::new(refused(code, detail)))?; + // An `alias@npm:@…` descriptor consumes the patched package + // under a different ident; the bare-name resolutions entry vendoring + // writes can never move it, so that copy keeps installing the + // UNPATCHED bytes. Surface every such entry loudly instead of + // silently part-patching. + for key in &scan.alias_keys { + warnings.push(VendorWarning::new( + "vendor_alias_entry_skipped", + format!( + "{YARN_LOCK} entry `{key}` consumes {name}@{version} through an npm: alias; \ + the bare-name resolutions entry vendoring writes cannot move aliased \ + descriptors, so that copy keeps installing the UNPATCHED registry bytes" + ), + )); + } + let (target_idx, target_is_ours) = target_gate(&scan, name, version)?; + Ok(YarnBerryPlan { + lock_text, + blocks, + workspace, + pkg_bytes, + pkg, + takeover_original, + target_idx, + target_is_ours, + }) } - let tgz_sha512 = hex::encode(Sha512::digest(&tgz_bytes)); - // `hash=` — the first 6 hex chars of sha512(tgz): the lock-committed - // tamper guard on the tarball itself (flips on any byte edit). - let hash6 = &tgz_sha512[..6]; - let locator = encode_uri_component(&format!("{workspace}@workspace:.")); - let resolution = format!("{name}@file:./{rel_tgz}#./{rel_tgz}::hash={hash6}&locator={locator}"); - // The service is the authority for the checksum of these bytes. - let mut service_serves_other_bytes = false; - if packed.yarn_berry10c0.is_none() { - if let Some(cfg) = service.filter(|cfg| cfg.service_enabled()) { - if let super::service_fetch::ServiceArtifact::Ready(archive) = - super::service_fetch::fetch_verified_archive(cfg, &record.uuid).await - { - if hex::encode(Sha256::digest(&archive.bytes)) == packed.sha256_hex { - packed.yarn_berry10c0 = archive.yarn_berry10c0; - } else { - service_serves_other_bytes = true; + + async fn wire( + &self, + plan: YarnBerryPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + _warnings: &mut Vec, + ) -> Result, String> { + let YarnBerryPlan { + lock_text, + blocks, + workspace, + pkg_bytes, + pkg, + takeover_original, + target_idx, + target_is_ours, + } = plan; + let name = cx.coords.name.as_str(); + let project_root = cx.project_root; + let target = &blocks[target_idx]; + let rel_tgz = staged.rel_tgz.clone(); + // The resolutions spec — `file:./` spelling per the B3 fixture. + let spec = format!("file:./{rel_tgz}"); + let packed = &mut staged.packed; + + // ── 8. Berry identity facts of the packed tarball ───────────────── + // A reuse hands over the exact bytes it verified; a fresh pack is + // re-read and must still be the bytes the pack hashed (the lock's + // checksum and `hash=` are derived from these, so a file swapped + // after verification must fail, never be pinned). + let reused = staged.verified_bytes.is_some(); + let tgz_bytes = match staged.verified_bytes.take() { + Some(bytes) => bytes, + None => tokio::fs::read(project_root.join(&rel_tgz)) + .await + .map_err(|e| format!("cannot re-read the packed tarball: {e}"))?, + }; + if hex::encode(Sha256::digest(&tgz_bytes)) != packed.sha256_hex { + return Err(format!( + "the packed tarball {rel_tgz} changed on disk after it was verified" + )); + } + let tgz_sha512 = hex::encode(Sha512::digest(&tgz_bytes)); + // `hash=` — the first 6 hex chars of sha512(tgz): the lock-committed + // tamper guard on the tarball itself (flips on any byte edit). + let hash6 = &tgz_sha512[..6]; + let locator = encode_uri_component(&format!("{workspace}@workspace:.")); + let resolution = + format!("{name}@file:./{rel_tgz}#./{rel_tgz}::hash={hash6}&locator={locator}"); + // The service is the authority for the checksum of these bytes. + let mut service_serves_other_bytes = false; + if packed.yarn_berry10c0.is_none() { + if let Some(cfg) = cx.service.filter(|cfg| cfg.service_enabled()) { + if let super::service_fetch::ServiceArtifact::Ready(archive) = + super::service_fetch::fetch_verified_archive(cfg, &cx.record.uuid).await + { + if hex::encode(Sha256::digest(&archive.bytes)) == packed.sha256_hex { + packed.yarn_berry10c0 = archive.yarn_berry10c0; + } else { + service_serves_other_bytes = true; + } } } } - } - // A reused ledger entry written before the checksum was recorded (or by - // another npm flavor) carries none. When the service cannot vouch - // (offline, unavailable, or serving other bytes) but our own lock entry - // already pins `hash=` of these verified bytes, it was written from - // them, so an in-sync re-run can keep its checksum. Such a checksum only - // re-wires; it is never recorded in the ledger. - let mut recovered_from_lock = false; - if packed.yarn_berry10c0.is_none() - && reused - && target_is_ours - && berry_field(&target.lines, "resolution") == Some(resolution.as_str()) - { - if let Some(c) = berry_field(&target.lines, "checksum") { - let full = if c.contains('/') { - c.to_string() - } else { - format!("{SUPPORTED_CACHE_KEY}/{c}") - }; - if valid_berry_checksum(&full) { - packed.yarn_berry10c0 = Some(full); - recovered_from_lock = true; + // A reused ledger entry written before the checksum was recorded (or + // by another npm flavor) carries none. When the service cannot vouch + // (offline, unavailable, or serving other bytes) but our own lock + // entry already pins `hash=` of these verified bytes, it was written + // from them, so an in-sync re-run can keep its checksum. Such a + // checksum only re-wires; it is never recorded in the ledger. + let mut recovered_from_lock = false; + if packed.yarn_berry10c0.is_none() + && reused + && target_is_ours + && berry_field(&target.lines, "resolution") == Some(resolution.as_str()) + { + if let Some(c) = berry_field(&target.lines, "checksum") { + let full = if c.contains('/') { + c.to_string() + } else { + format!("{SUPPORTED_CACHE_KEY}/{c}") + }; + if valid_berry_checksum(&full) { + packed.yarn_berry10c0 = Some(full); + recovered_from_lock = true; + } } } - } - let checksum = match packed.yarn_berry10c0.as_deref().filter(|c| valid_berry_checksum(c)) { - Some(c) => checksum_in_lock_spelling(&lock_text, c), - // A reused tarball is kept as is, so retrying cannot help when the - // service serves other bytes: only a fresh vendor can wire it. - None if reused && service_serves_other_bytes => return done_failure_unstage(purl, - format!("the patch service now serves other bytes than the committed {rel_tgz}, and no Yarn Berry checksum is recorded for it; restore yarn.lock from version control, or {}", super::common::REVERT_ALL_AND_REVENDOR), - project_root, &uuid_dir_rel, uuid_dir_preexisted).await, - None if reused => return done_failure_unstage(purl, - format!("no Yarn Berry checksum is recorded for the committed {rel_tgz}; re-run online so the patch service can supply it"), - project_root, &uuid_dir_rel, uuid_dir_preexisted).await, - None => return done_failure_unstage(purl, - format!("the patch service supplied no Yarn Berry checksum for {name}; retry after the server artifact is ready"), - project_root, &uuid_dir_rel, uuid_dir_preexisted).await, - }; - - // ── 9. The replacement lock entry (verbatim B3 shape) ───────────────── - let lock_key = format!("\"{name}@file:./{rel_tgz}::locator={locator}\""); - if patches_manifest { - warnings.push(VendorWarning::new( - "vendor_dep_manifest_stale", - format!( - "the patch rewrites {name}@{version}'s package.json; the yarn.lock entry \ - keeps the registry entry's dependency fields — if the patch changed \ - dependencies, run `yarn install` once to refresh them" - ), - )); - } - // Yarn builds a `file:` entry from the tarball's own package.json, whose - // `bin` keeps its published spelling where the registry entry's is - // normalized (#718). A tarball without a readable manifest keeps the - // registry's map (yarn cannot install it either way). - let tarball_bin = crate::patch::package::read_archive_bytes_to_map(&tgz_bytes) - .ok() - .and_then(|members| serde_json::from_slice::(members.get(PACKAGE_JSON)?).ok()) - .filter(Value::is_object) - .map(|manifest| manifest_bin(&manifest)); - // The exact entry yarn 4 emits for a resolutions-driven `file:` tarball - // (the B3 fixture shape), fields in yarn's order (#697). - let lock_key_line = format!("{lock_key}:"); - let new_lines = render_pinned_entry( - &target.lines[1..], - &Pin { - key_line: &lock_key_line, - resolution: &resolution, - checksum: Some(&checksum), - bin: tarball_bin.as_ref(), - }, - ); - - // ── 10. In-sync hot path: nothing to write, nothing to record ───────── - let existing_res = pkg_obj.get("resolutions").and_then(|r| r.get(name)); - let pkg_in_sync = existing_res.and_then(Value::as_str) == Some(spec.as_str()); - if pkg_in_sync && target_is_ours && target.lines == new_lines { - return VendorOutcome::Done { - result: already_patched_result(purl, &dest, &record.files), - entry: None, - warnings, + let checksum = match packed.yarn_berry10c0.as_deref().filter(|c| valid_berry_checksum(c)) { + Some(c) => checksum_in_lock_spelling(&lock_text, c), + // A reused tarball is kept as is, so retrying cannot help when + // the service serves other bytes: only a fresh vendor can wire it. + None if reused && service_serves_other_bytes => return Err(format!("the patch service now serves other bytes than the committed {rel_tgz}, and no Yarn Berry checksum is recorded for it; restore yarn.lock from version control, or {}", super::common::REVERT_ALL_AND_REVENDOR)), + None if reused => return Err(format!("no Yarn Berry checksum is recorded for the committed {rel_tgz}; re-run online so the patch service can supply it")), + None => return Err(format!("the patch service supplied no Yarn Berry checksum for {name}; retry after the server artifact is ready")), }; - } - // ── 11. Build both new byte images, then commit pkg-first/lock-second ─ - let existing_entry = existing_res.is_some(); - let mut new_pkg = (*pkg).clone(); - { - let obj = new_pkg.as_object_mut().expect("validated above"); - let res = obj - .entry("resolutions".to_string()) - .or_insert_with(|| Value::Object(serde_json::Map::new())); - let Some(res_obj) = res.as_object_mut() else { - return done_failure_unstage( - purl, - "resolutions table vanished mid-edit".to_string(), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await; - }; - res_obj.insert(name.to_string(), Value::String(spec.clone())); - } - let new_pkg_bytes = match JsonLayout::of(&pkg_text).render(&new_pkg) { - Ok(b) => b, - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot serialize {PACKAGE_JSON}: {e}"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } - }; - let new_lock_text = replace_block( - &lock_text, - target, - &new_lines, - block_eol(&lock_text, target), - ); - if let Err(e) = commit_pair( - project_root, - &new_pkg_bytes, - &pkg_bytes, - new_lock_text.as_bytes(), - ) - .await - { - return done_failure_unstage(purl, e, project_root, &uuid_dir_rel, uuid_dir_preexisted) - .await; - } + // ── 9. The replacement lock entry (verbatim B3 shape) ───────────── + let lock_key = format!("\"{name}@file:./{rel_tgz}::locator={locator}\""); + // Yarn builds a `file:` entry from the tarball's own package.json, + // whose `bin` keeps its published spelling where the registry + // entry's is normalized (#718). A tarball without a readable + // manifest keeps the registry's map (yarn cannot install it either + // way). + let tarball_bin = crate::patch::package::read_archive_bytes_to_map(&tgz_bytes) + .ok() + .and_then(|members| serde_json::from_slice::(members.get(PACKAGE_JSON)?).ok()) + .filter(Value::is_object) + .map(|manifest| manifest_bin(&manifest)); + // The exact entry yarn 4 emits for a resolutions-driven `file:` + // tarball (the B3 fixture shape), fields in yarn's order (#697). + let lock_key_line = format!("{lock_key}:"); + let new_lines = render_pinned_entry( + &target.lines[1..], + &Pin { + key_line: &lock_key_line, + resolution: &resolution, + checksum: Some(&checksum), + bin: tarball_bin.as_ref(), + }, + ); - // ── 12. Marker + ledger entry ───────────────────────────────────────── - let marker = VendorMarker::new("npm", &base_purl, record, vendored_at); - write_marker_or_warn(&project_root.join(&uuid_dir_rel), &marker, &mut warnings).await; + // ── 10. In-sync hot path: nothing to write, nothing to record ───── + let pkg_obj = pkg.as_object().expect("validated in the pre-flight"); + let existing_res = pkg_obj.get("resolutions").and_then(|r| r.get(name)); + let pkg_in_sync = existing_res.and_then(Value::as_str) == Some(spec.as_str()); + if pkg_in_sync && target_is_ours && target.lines == new_lines { + return Ok(None); + } - let wiring = vec![ - WiringRecord { - file: PACKAGE_JSON.to_string(), - kind: KIND_RESOLUTION.to_string(), - // Rewritten when replacing our own stale entry (no `original` — - // never record our own edit as a pre-vendor fragment) or a - // taken-over user pin (whose value IS the `original`, restored - // verbatim on revert). - action: if existing_entry { - WiringAction::Rewritten - } else { - WiringAction::Added + // ── 11. Build both new byte images, then commit pkg-first/lock-second + let existing_entry = existing_res.is_some(); + let mut new_pkg = (*pkg).clone(); + { + let obj = new_pkg.as_object_mut().expect("validated above"); + let res = obj + .entry("resolutions".to_string()) + .or_insert_with(|| Value::Object(serde_json::Map::new())); + let Some(res_obj) = res.as_object_mut() else { + return Err("resolutions table vanished mid-edit".to_string()); + }; + res_obj.insert(name.to_string(), Value::String(spec.clone())); + } + let new_pkg_bytes = JsonLayout::of(&String::from_utf8_lossy(&pkg_bytes)) + .render(&new_pkg) + .map_err(|e| format!("cannot serialize {PACKAGE_JSON}: {e}"))?; + let new_lock_text = replace_block( + &lock_text, + target, + &new_lines, + block_eol(&lock_text, target), + ); + commit_pair( + project_root, + &new_pkg_bytes, + &pkg_bytes, + new_lock_text.as_bytes(), + ) + .await?; + + let wiring = vec![ + WiringRecord { + file: PACKAGE_JSON.to_string(), + kind: KIND_RESOLUTION.to_string(), + // Rewritten when replacing our own stale entry (no + // `original` — never record our own edit as a pre-vendor + // fragment) or a taken-over user pin (whose value IS the + // `original`, restored verbatim on revert). + action: if existing_entry { + WiringAction::Rewritten + } else { + WiringAction::Added + }, + key: Some(name.to_string()), + original: takeover_original.map(Value::String), + new: Some(Value::String(spec)), }, - key: Some(name.to_string()), - original: takeover_original.map(Value::String), - new: Some(Value::String(spec)), - }, - WiringRecord { - file: YARN_LOCK.to_string(), - kind: KIND_LOCK_ENTRY.to_string(), - action: WiringAction::Rewritten, - key: Some(lock_key), - original: if target_is_ours { - None - } else { - Some(lines_to_json(&target.lines)) + WiringRecord { + file: YARN_LOCK.to_string(), + kind: KIND_LOCK_ENTRY.to_string(), + action: WiringAction::Rewritten, + key: Some(lock_key), + original: if target_is_ours { + None + } else { + Some(lines_to_json(&target.lines)) + }, + new: Some(lines_to_json(&new_lines)), }, - new: Some(lines_to_json(&new_lines)), - }, - ]; - let entry = VendorEntry { - ecosystem: "npm".to_string(), - base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { + ]; + Ok(Some(NpmCommit { + wiring, yarn_berry10c0: packed .yarn_berry10c0 .clone() .filter(|_| !recovered_from_lock), - path: rel_tgz, - sha256: packed.sha256_hex, - size: Some(packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some("yarn-berry".to_string()), - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }; - VendorOutcome::Done { - result, - entry: Some(entry), - warnings, + ..NpmCommit::default() + })) + } + + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning { + VendorWarning::new( + "vendor_dep_manifest_stale", + format!( + "the patch rewrites {name}@{version}'s package.json; the yarn.lock entry \ + keeps the registry entry's dependency fields — if the patch changed \ + dependencies, run `yarn install` once to refresh them" + ), + ) } } @@ -2188,6 +2145,39 @@ __metadata: ); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture().await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = br#"{"name":"left-pad","version":"1.3.0","sideEffects":false}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + /// #697: yarn writes `checksum:` after `bin:` and before `conditions:` /// (special keys first, the rest alphabetically). A platform-conditional /// entry's `conditions:` must stay after the checksum the vendored entry diff --git a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs index 5e650c3d4..fb1fb2f17 100644 --- a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs @@ -38,16 +38,15 @@ use crate::patch::apply::PatchSources; use crate::utils::fs::{atomic_write_bytes_preserving_mode, read_regular_to_string}; use crate::utils::socket_dir::remove_tree_and_prune; -use super::common::{already_patched_result, detect_eol, refused}; +use super::common::{detect_eol, refused}; use super::npm_common::{ - done_failure_unstage, guard_coordinates, guard_revert_uuid_dir, stage_patch_pack, + guard_revert_uuid_dir, vendor_npm_family, NpmCommit, NpmCoords, NpmLockBackend, NpmStagedPack, + NpmVendorRequest, WireCx, }; use super::parse_memo::ParseMemo; use super::path::parse_vendor_path; use super::source::PackageSource; -use super::state::{ - write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, -}; +use super::state::{VendorEntry, WiringAction, WiringRecord}; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; const YARN_LOCK: &str = "yarn.lock"; @@ -61,7 +60,8 @@ const KIND_LOCK_BLOCK: &str = "yarn_lock_block"; /// Same contract as [`super::npm_lock::vendor_npm`]: refuse-early, wire-last /// (every refusal fires before any write inside the project; the lock edit is /// the final mutation), `entry` is `None` for dry runs and the in-sync -/// re-run. +/// re-run. The flow is [`vendor_npm_family`]'s; [`YarnClassicBackend`] is +/// the v1 lock grammar. #[allow(clippy::too_many_arguments)] pub async fn vendor_yarn_classic<'a>( purl: &str, @@ -74,202 +74,157 @@ pub async fn vendor_yarn_classic<'a>( force: bool, service: Option<&super::VendorServiceConfig>, ) -> VendorOutcome { - let installed_dir = installed_dir.into(); - let mut warnings: Vec = Vec::new(); + vendor_npm_family( + &YarnClassicBackend, + NpmVendorRequest { + purl, + installed_dir: installed_dir.into(), + project_root, + record, + sources, + vendored_at, + dry_run, + force, + service, + }, + ) + .await +} - // ── 1. Coordinates (shared fail-closed guard, before any disk access) ─ - let coords = match guard_coordinates(purl, record) { - Ok(coords) => coords, - Err(outcome) => return *outcome, - }; - let (name, version) = (coords.name.as_str(), coords.version.as_str()); - let uuid_dir_rel = coords.uuid_dir_rel; - let base_purl = coords.base_purl; +/// The yarn-classic half of [`vendor_yarn_classic`]. +struct YarnClassicBackend; - // ── 2. Lockfile ─────────────────────────────────────────────────────── - let lock_path = project_root.join(YARN_LOCK); - let text = match read_yarn_lock(project_root).await { - Ok(t) => t, - Err(outcome) => return *outcome, - }; - if let Err(outcome) = refuse_berry_lock(&text) { - return *outcome; - } - - // ── 3. Find the rewritable blocks (pre-flight, BEFORE staging) ──────── - let blocks = scan_blocks_shared(&text); - let candidate_keys = match rewritable_candidates(&blocks, name, version) { - Ok((keys, skipped)) => { - warnings.extend(skipped); - keys - } - Err(outcome) => return *outcome, - }; - drop(blocks); +/// [`YarnClassicBackend`]'s pre-flight product: the lock text and the keys +/// of the blocks to rewrite. +struct YarnClassicPlan { + text: String, + candidate_keys: Vec, +} - // ── 4–7. Stage → patch → pack (shared flavor-agnostic pipeline) ─────── - let (staged, result) = match stage_patch_pack( - purl, - installed_dir, - project_root, - record, - sources, - dry_run, - force, - &mut warnings, - service, - ) - .await - { - Ok(pair) => pair, - Err(outcome) => return *outcome, - }; - let Some(staged) = staged else { - // Failed patch (no lock writes — wiring is last) or a dry run. - return VendorOutcome::Done { - result, - entry: None, - warnings, - }; - }; - let uuid_dir_preexisted = staged.uuid_dir_preexisted; - let rel_tgz = staged.rel_tgz; - let packed = staged.packed; - let staged_pkg_json = staged.staged_pkg_json; - let dest = project_root.join(&rel_tgz); - // SECURITY/CORRECTNESS: the `file:./` prefix is load-bearing — a bare - // path is registry-relative to yarn classic (spike Y2: 404). - let resolved_value = format!("file:./{rel_tgz}#{}", packed.sha1_hex); - - // ── 8. Lock rewrite: splice each candidate block, byte-preserving ───── - let eol = detect_eol(&text); - let mut new_text = text; - let mut wiring: Vec = Vec::new(); - for key in &candidate_keys { - let edit = { - // While nothing has been spliced yet, `new_text` is still the - // text scanned above and every candidate hits that scan — the - // whole idempotent re-run takes this arm. Once a splice has - // rewritten it, each key sees text no later read can ask for - // again, so scan it without paying the memo's copy of it. - let blocks = if wiring.is_empty() { - scan_blocks_shared(&new_text) - } else { - Arc::new(scan_blocks(&new_text)) - }; - let Some(block) = blocks.iter().find(|b| &b.key == key) else { - return done_failure_unstage( - purl, - format!("lock block `{key}` vanished mid-rewrite"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await; - }; - let new_lines = rewrite_classic_block( - &block.lines, - &resolved_value, - &packed.integrity, - staged_pkg_json.as_ref(), - ); - if new_lines == block.lines { - // Idempotency: already carrying our exact spec — no edit, no - // wiring record. - None - } else { - // Never record one of our own (stale) edits as the - // "original" — revert must restore the pre-vendor registry - // fragment, not a dangling `.socket/vendor/` pointer. - let was_vendored = block_points_into_vendor(&block.lines); - let rec = WiringRecord { - file: YARN_LOCK.to_string(), - kind: KIND_LOCK_BLOCK.to_string(), - action: WiringAction::Rewritten, - key: Some(key.clone()), - original: if was_vendored { - None - } else { - Some(lines_to_json(&block.lines)) - }, - new: Some(lines_to_json(&new_lines)), +impl NpmLockBackend for YarnClassicBackend { + type Plan = YarnClassicPlan; + + fn flavor(&self) -> Option<&'static str> { + Some("yarn-classic") + } + + async fn preflight( + &self, + project_root: &Path, + coords: &NpmCoords, + warnings: &mut Vec, + ) -> Result> { + // ── 2. Lockfile ─────────────────────────────────────────────────── + let text = read_yarn_lock(project_root).await?; + refuse_berry_lock(&text)?; + + // ── 3. Find the rewritable blocks (pre-flight, BEFORE staging) ──── + let blocks = scan_blocks_shared(&text); + let (candidate_keys, skipped) = + rewritable_candidates(&blocks, &coords.name, &coords.version)?; + warnings.extend(skipped); + Ok(YarnClassicPlan { + text, + candidate_keys, + }) + } + + async fn wire( + &self, + plan: YarnClassicPlan, + cx: &WireCx<'_>, + staged: &mut NpmStagedPack, + _warnings: &mut Vec, + ) -> Result, String> { + let YarnClassicPlan { + text, + candidate_keys, + } = plan; + // SECURITY/CORRECTNESS: the `file:./` prefix is load-bearing — a + // bare path is registry-relative to yarn classic (spike Y2: 404). + let resolved_value = format!("file:./{}#{}", staged.rel_tgz, staged.packed.sha1_hex); + + // ── 8. Lock rewrite: splice each candidate block, byte-preserving ─ + let eol = detect_eol(&text); + let mut new_text = text; + let mut wiring: Vec = Vec::new(); + for key in &candidate_keys { + let edit = { + // While nothing has been spliced yet, `new_text` is still the + // text scanned above and every candidate hits that scan — the + // whole idempotent re-run takes this arm. Once a splice has + // rewritten it, each key sees text no later read can ask for + // again, so scan it without paying the memo's copy of it. + let blocks = if wiring.is_empty() { + scan_blocks_shared(&new_text) + } else { + Arc::new(scan_blocks(&new_text)) + }; + let Some(block) = blocks.iter().find(|b| &b.key == key) else { + return Err(format!("lock block `{key}` vanished mid-rewrite")); }; - Some((replace_block(&new_text, block, &new_lines, eol), rec)) + let new_lines = rewrite_classic_block( + &block.lines, + &resolved_value, + &staged.packed.integrity, + staged.staged_pkg_json.as_ref(), + ); + if new_lines == block.lines { + // Idempotency: already carrying our exact spec — no edit, + // no wiring record. + None + } else { + // Never record one of our own (stale) edits as the + // "original" — revert must restore the pre-vendor + // registry fragment, not a dangling `.socket/vendor/` + // pointer. + let was_vendored = block_points_into_vendor(&block.lines); + let rec = WiringRecord { + file: YARN_LOCK.to_string(), + kind: KIND_LOCK_BLOCK.to_string(), + action: WiringAction::Rewritten, + key: Some(key.clone()), + original: if was_vendored { + None + } else { + Some(lines_to_json(&block.lines)) + }, + new: Some(lines_to_json(&new_lines)), + }; + Some((replace_block(&new_text, block, &new_lines, eol), rec)) + } + }; + if let Some((replaced, rec)) = edit { + new_text = replaced; + wiring.push(rec); } - }; - if let Some((replaced, rec)) = edit { - new_text = replaced; - wiring.push(rec); } + + if wiring.is_empty() { + // Every block already points at this uuid with the packed + // hashes (`#sha1` and `integrity`): in sync. + return Ok(None); + } + + forget_block_scans(); + atomic_write_bytes_preserving_mode(&cx.project_root.join(YARN_LOCK), new_text.as_bytes()) + .await + .map_err(|e| format!("cannot write {YARN_LOCK}: {e}"))?; + Ok(Some(NpmCommit { + wiring, + ..NpmCommit::default() + })) } - if staged_pkg_json.is_some() && !wiring.is_empty() { - warnings.push(VendorWarning::new( + + fn manifest_warning(&self, name: &str, version: &str) -> VendorWarning { + VendorWarning::new( "vendor_dep_manifest_rewritten", format!( "the patch rewrites {name}@{version}'s package.json; its lock blocks' \ dependencies/optionalDependencies sub-maps were recomputed from the patched \ manifest" ), - )); - } - - if wiring.is_empty() { - // Every block already points at this uuid with the packed hashes: - // in sync. `#sha1` and `integrity` were derived from the reused - // committed tarball (or, when reuse missed, from a fresh acquisition - // that reproduced them); touch nothing and synthesize - // AlreadyPatched. - return VendorOutcome::Done { - result: already_patched_result(purl, &dest, &record.files), - entry: None, - warnings, - }; - } - - forget_block_scans(); - if let Err(e) = atomic_write_bytes_preserving_mode(&lock_path, new_text.as_bytes()).await { - return done_failure_unstage( - purl, - format!("cannot write {YARN_LOCK}: {e}"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, ) - .await; - } - - // ── 9. Marker + ledger entry ────────────────────────────────────────── - let marker = VendorMarker::new("npm", &base_purl, record, vendored_at); - write_marker_or_warn(&project_root.join(&uuid_dir_rel), &marker, &mut warnings).await; - - let entry = VendorEntry { - ecosystem: "npm".to_string(), - base_purl, - uuid: record.uuid.clone(), - artifact: VendorArtifact { - yarn_berry10c0: None, - path: rel_tgz, - sha256: packed.sha256_hex, - size: Some(packed.size), - platform_locked: None, - file_inventory: None, - }, - wiring, - lock: None, - took_over_go_patches: false, - detached: false, - record: None, - flavor: Some("yarn-classic".to_string()), - uv: None, - pnpm: None, - poetry: None, - pdm: None, - pipenv: None, - }; - VendorOutcome::Done { - result, - entry: Some(entry), - warnings, } } @@ -1384,6 +1339,40 @@ left-pad@^1.3.0: ); } + /// #920: the `package.json` advisory is emitted once, by the run that + /// wires — an in-sync re-run of a manifest-rewriting patch is a quiet + /// AlreadyPatched. + #[tokio::test] + async fn manifest_rewriting_rerun_is_in_sync_without_the_manifest_warning() { + let mut fx = fixture_with_lock(Y2_BEFORE).await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = + br#"{"name":"left-pad","version":"1.3.0","dependencies":{"wow":"^1.0.0"}}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_some(), "{:?}", result.error); + let manifest_warnings = |w: &[VendorWarning]| { + w.iter() + .filter(|w| w.code.starts_with("vendor_dep_manifest")) + .count() + }; + assert_eq!(manifest_warnings(&warnings), 1, "{warnings:?}"); + + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success && entry.is_none(), "{:?}", result.error); + assert_eq!(manifest_warnings(&warnings), 0, "{warnings:?}"); + } + /// Twin of npm_lock's relock re-pin test: a relock back to the registry /// block, re-pinned from the REUSED tarball (no request under the /// outage), recomputes the dependency sub-maps from the reused bytes' diff --git a/crates/socket-patch-core/src/vex/discover/bun.rs b/crates/socket-patch-core/src/vex/discover/bun.rs index b219ebaae..e8a7e7c70 100644 --- a/crates/socket-patch-core/src/vex/discover/bun.rs +++ b/crates/socket-patch-core/src/vex/discover/bun.rs @@ -239,6 +239,7 @@ impl Bundled { r.purl, ), ); + out.shadow(r); } } } diff --git a/crates/socket-patch-core/src/vex/discover/mod.rs b/crates/socket-patch-core/src/vex/discover/mod.rs index 9e1d43274..3cce5cc88 100644 --- a/crates/socket-patch-core/src/vex/discover/mod.rs +++ b/crates/socket-patch-core/src/vex/discover/mod.rs @@ -464,6 +464,11 @@ pub enum UnattestedKind { /// a `package.json` project's npm deps from it, never from the /// npm-family lock that carries the wiring. DenoLock, + /// The wiring is in a root `npm-shrinkwrap.json` with no + /// `package-lock.json` twin (#899): npm >= 12 never reads the + /// shrinkwrap, resolves the package from the registry and writes a + /// fresh package-lock.json, so only npm <= 11 installs the patch. + NpmShrinkwrapOnly, } /// A ref discovery emits (so rollback, remove and list find the wiring, @@ -555,6 +560,15 @@ pub struct Discovery { pub unwired_copies: Vec, /// Refs dropped because another lock contests them ([`ContestedRef`]). pub contested: Vec, + /// Refs withheld from `refs` only because the build ALSO installs an + /// unpatched copy of the same `name@version` that no rewire can reach + /// (a bundled copy unpacked from its parent's tarball, or a same-lock + /// [`Discovery::unpatched_copy`]), each diagnosed + /// [`DIAG_REF_UNATTRIBUTABLE`]. Never attested, but the wiring itself + /// is the rewriters' own output and names exactly one package version, + /// so the management commands (rollback, remove, list, the vendored + /// takeover) still see and unwind it (#828). Validated like `refs`. + pub shadowed: Vec, /// The bundled copies (purl → root-relative directory) the vlt /// extractor found in the installed store for the lock's nodes, exactly /// as [`crate::vendor::vlt_bundled::bundled_copies`] reports them: the @@ -609,7 +623,28 @@ impl Discovery { /// * the purl is `pkg:/@` (then canonicalized); /// * a vendored ref names a root-anchored artifact under its OWN uuid dir /// and its purl's ecosystem dir. - pub fn push(&mut self, mut r: PatchedRef) { + pub fn push(&mut self, r: PatchedRef) { + if let Some(r) = self.validated(r) { + if !self.refs.contains(&r) { + self.refs.push(r); + } + } + } + + /// Withhold the wired `r` from attestation because an unpatched copy of + /// its `name@version` installs beside it ([`Discovery::shadowed`]). The + /// caller diagnoses why. Validated exactly like [`Discovery::push`]. + pub(crate) fn shadow(&mut self, r: PatchedRef) { + if let Some(r) = self.validated(r) { + if !self.shadowed.contains(&r) { + self.shadowed.push(r); + } + } + } + + /// [`Discovery::push`]'s gate: the canonicalized ref, or `None` after + /// diagnosing why it is invalid. + fn validated(&mut self, mut r: PatchedRef) -> Option { let file = r.source_file.to_string_lossy().into_owned(); if !is_canonical_uuid(&r.uuid) { self.diag( @@ -620,7 +655,7 @@ impl Discovery { r.purl, r.uuid ), ); - return; + return None; } // Every identity an extractor builds a ref from is recognized — the // valid ref and the one rejected below alike (rule 11). This is also @@ -634,7 +669,7 @@ impl Discovery { &file, format!("{file}: {:?} is not a usable package purl", r.purl), ); - return; + return None; }; r.purl = purl; match r.mode { @@ -658,7 +693,7 @@ impl Discovery { r.purl, r.artifact_rel, r.uuid ), ); - return; + return None; } r.integrity_required = false; r.url = None; @@ -668,9 +703,7 @@ impl Discovery { if matches!(r.locked_integrity, Some(LockIntegrity::None)) { r.locked_integrity = None; } - if !self.refs.contains(&r) { - self.refs.push(r); - } + Some(r) } /// Record a diagnostic for root-relative `file`. `detail` is shown to @@ -723,16 +756,17 @@ impl Discovery { /// package managers that keep several lockfiles side by side (npm, /// pnpm, yarn, bun; uv, pylock, poetry, pdm, Pipfile, requirements) /// call it for every such entry with exact coordinates. + /// + /// Duplicates are kept until [`Discovery::finalize`] sorts and dedups + /// the list: a membership check here made recording quadratic in the + /// lock's size (#993), and every reader before then only asks whether + /// some entry matches. pub(crate) fn resolved_elsewhere(&mut self, file: &str, purl: Option) { let Some(purl) = purl else { return }; - let entry = ResolvedElsewhere { + self.elsewhere.push(ResolvedElsewhere { purl: canonical_base_purl(&purl), file: PathBuf::from(file), - }; - // Deduplicated once, in `finalize` (a per-push scan is quadratic in - // the lock's size); an earlier duplicate is identical, so the first - // match `contest_across_locks` finds is the same either way. - self.elsewhere.push(entry); + }); } /// Record that lock `file`'s entry `key` installs its own copy of `purl` @@ -767,8 +801,8 @@ impl Discovery { /// Drop every ref whose OWN lock also installs an unpatched copy of the /// same `name@version` ([`Discovery::unpatched_copy`]): the build ships /// that copy whatever the wiring does, so the ref is diagnosed - /// ([`DIAG_REF_UNATTRIBUTABLE`], naming the entry) and not emitted. Its - /// uuid stays recognized (rule 11). + /// ([`DIAG_REF_UNATTRIBUTABLE`], naming the entry) and not emitted, but + /// [`Discovery::shadowed`]. Its uuid stays recognized (rule 11). fn contest_within_locks(&mut self) { if self.unpatched_copies.is_empty() { return; @@ -792,6 +826,9 @@ impl Discovery { r.purl, r.uuid, c.key, c.how ), ); + // Withheld, not lost: rollback / remove / the takeover + // still unwind the wiring (#828). + self.shadow(r); } None => self.refs.push(r), } @@ -1018,15 +1055,17 @@ impl Discovery { self.unattested.dedup(); self.contested.sort(); self.contested.dedup(); - self.refs.sort_by(|a, b| { - (&a.source_file, &a.purl, &a.uuid, a.mode).cmp(&( - &b.source_file, - &b.purl, - &b.uuid, - b.mode, - )) - }); - self.refs.dedup(); + for refs in [&mut self.refs, &mut self.shadowed] { + refs.sort_by(|a, b| { + (&a.source_file, &a.purl, &a.uuid, a.mode).cmp(&( + &b.source_file, + &b.purl, + &b.uuid, + b.mode, + )) + }); + refs.dedup(); + } self.diagnostics .sort_by(|a, b| (&a.file, a.code, &a.detail).cmp(&(&b.file, b.code, &b.detail))); self.recognized.sort(); diff --git a/crates/socket-patch-core/src/vex/discover/npm.rs b/crates/socket-patch-core/src/vex/discover/npm.rs index 00ee740e6..5a89a6320 100644 --- a/crates/socket-patch-core/src/vex/discover/npm.rs +++ b/crates/socket-patch-core/src/vex/discover/npm.rs @@ -28,6 +28,12 @@ //! — a stale mirror entry must not outvote the `packages` entry npm //! installs. //! +//! Entries npm installs from somewhere other than their `resolved` — a git / +//! url / `file:` spec, or a dependency's own npm-shrinkwrap.json (an entry +//! beneath a `hasShrinkwrap: true` package, #753) — are never attested and +//! contest every ref for the same `name@version` +//! ([`drop_non_registry_installs`]). +//! //! An entry is a ref when its `resolved` is //! //! * a Socket-HOSTED url ([`DiscoverCtx::hosted_uuid`]) → [`WiringMode::Hosted`]. @@ -61,26 +67,36 @@ use crate::vendor::lock_inventory::{ npm_lock_bundled_nodes, npm_lock_legacy_mirror_nodes, npm_lock_located_nodes, LockIntegrity, NpmLockNode, }; -use crate::vendor::npm_origin::{npm_non_registry_entries, NpmOverrides}; +use crate::vendor::npm_origin::{ + npm_non_registry_entries, npm_shrinkwrapped_entries, NpmOverrides, +}; pub(crate) async fn extract(ctx: &DiscoverCtx<'_>, out: &mut Discovery) { let mut locks: Vec = Vec::new(); + let mut twin_present = false; for lock in NPM_LOCKS { if let Some(read) = extract_package_lock(ctx, lock, out).await { locks.push(read); + } else if lock == NPM_LOCKS[1] { + // Present but unreadable / unparseable (already diagnosed): + // npm 12 cannot install from it either, but the shrinkwrap-only + // detail must not claim it is missing. + twin_present = ctx.exists(lock).await; } } - push_uncontested(locks, out); + push_uncontested(locks, twin_present, out); extract_pnpm(ctx, out).await; } /// What one parsed npm lock wires, plus the packages it resolves ELSEWHERE /// (an entry whose `resolved` is not a Socket reference), the packages it /// installs BUNDLED (each purl → the first such entry's lock location) and -/// every `name@version` it has any entry for, at any path. +/// every `name@version` it has any entry for, at any path. `ref_locations` +/// names the lock entry each ref was read from (purl, location). struct NpmLockRefs { file: &'static str, refs: Vec, + ref_locations: Vec<(String, String)>, unwired: BTreeMap, bundled: BTreeMap, mentioned: BTreeSet, @@ -102,9 +118,9 @@ impl NpmLockRefs { } /// Push every ref no OTHER npm lock contests. npm <= 11 installs from -/// npm-shrinkwrap.json when both exist; npm 12 auto-creates a -/// package-lock.json beside it and installs from THAT (verified against real -/// npm 12.0.0 / 12.1.0). A package one lock wires to a Socket patch while the +/// npm-shrinkwrap.json when both exist; npm 12 never reads the shrinkwrap, +/// writes a package-lock.json (from the registry) beside it and installs from +/// THAT (verified against real npm 12.0.0 / 12.1.0). A package one lock wires to a Socket patch while the /// other resolves it only elsewhere (the registry) is therefore installed /// patched by some npm majors and unpatched by others — not decidable from /// the files, so it is diagnosed and not attested (the same call as the v2 @@ -127,7 +143,13 @@ impl NpmLockRefs { /// `name@version` elsewhere (#588 — e.g. a workspace member added after /// the rewire, then `npm install`): npm installs every entry, and that one /// fetches the unpatched registry bytes. -fn push_uncontested(locks: Vec, out: &mut Discovery) { +/// +/// A ref in a lone `npm-shrinkwrap.json` (no usable package-lock.json +/// twin) is pushed but marked [`UnattestedKind::NpmShrinkwrapOnly`] (#899): +/// npm 12 never reads the shrinkwrap and installs from the registry +/// instead. `twin_present`: a package-lock.json exists but could not be +/// read or parsed (the detail says so instead of calling it missing). +fn push_uncontested(locks: Vec, twin_present: bool, out: &mut Discovery) { let wired: Vec> = locks .iter() .map(|l| l.refs.iter().map(|r| r.purl.clone()).collect()) @@ -153,6 +175,9 @@ fn push_uncontested(locks: Vec, out: &mut Discovery) { lock.file, r.purl, r.uuid, ), ); + // Still the hosted rewriter's own output for exactly this + // version, so rollback / remove / the takeover unwind it. + out.shadow(r.clone()); continue; } if let Some(location) = lock.unwired.get(&r.purl) { @@ -206,6 +231,39 @@ fn push_uncontested(locks: Vec, out: &mut Discovery) { ), ); } else { + if locks.len() == 1 && lock.file == NPM_LOCKS[0] { + // A shrinkwrap with no package-lock.json twin (#899): + // npm 12 no longer reads npm-shrinkwrap.json at all — it + // resolves the tree fresh from the registry and writes + // its own package-lock.json — so the wiring reaches npm + // <= 11 only. The ref stays (rollback, remove and list + // still manage it); VEX omits it. + let detail = if twin_present { + format!( + "its {} twin cannot be read or parsed — npm >= 12 never reads \ + {}, so only npm <= 11 installs the patched bytes; repair {} \ + (e.g. `npm install --package-lock-only`) and re-run the scan \ + (`scan --mode hosted` / `scan --mode vendored`)", + NPM_LOCKS[1], NPM_LOCKS[0], NPM_LOCKS[1], + ) + } else { + format!( + "{} has no {} twin — npm >= 12 never reads {}, resolves the \ + package from the registry and writes a fresh {}, so only npm \ + <= 11 installs the patched bytes; rename the lock to {} (or \ + commit a copy under that name) and re-run the scan (`scan \ + --mode hosted` / `scan --mode vendored`)", + lock.file, NPM_LOCKS[1], NPM_LOCKS[0], NPM_LOCKS[1], NPM_LOCKS[1], + ) + }; + out.unattested( + &r.purl, + &r.uuid, + lock.file, + detail, + UnattestedKind::NpmShrinkwrapOnly, + ); + } out.push(r.clone()); } } @@ -224,6 +282,7 @@ async fn extract_package_lock( let mut read = NpmLockRefs { file, refs: Vec::new(), + ref_locations: Vec::new(), unwired: BTreeMap::new(), bundled: BTreeMap::new(), mentioned: BTreeSet::new(), @@ -278,6 +337,14 @@ async fn extract_package_lock( /// mirror node that agrees, or a mirror that does not mention the package, /// contests nothing; a mirror node wired while `packages` is not is never a /// ref (see [`npm_lock_nodes`]). +/// +/// A mirror node with NO `resolved` but the wired ref's (patched) +/// `integrity` agrees too (#879): npm 7-12's serializer never writes +/// `resolved` for a `file:` resolution in the mirror, so every `npm install` +/// on a vendored v2 lock leaves exactly that shape. npm 6 cannot install +/// unpatched bytes from it: it fetches `name@version` from the registry and +/// fails closed on the patched pin (EINTEGRITY), the same outcome accepted +/// for a hosted alias mirror node. fn drop_mirror_unwired( ctx: &DiscoverCtx<'_>, file: &str, @@ -290,6 +357,9 @@ fn drop_mirror_unwired( let Some(purl) = node.version.and_then(|v| npm_purl(node.name, v)) else { continue; }; + if node.resolved.is_none() && pins_a_wired_ref(read, &purl, node.sri_pin()) { + continue; + } let located = node.resolved.map_or_else(Located::default, |r| { ctx.locate(r, LocateOpts::LITERAL_CHECKED) }); @@ -327,12 +397,31 @@ fn drop_mirror_unwired( }); } +/// Whether `pin` is the integrity a `packages` ref of this lock pins `purl` +/// to — the patched bytes, so nothing installs unpatched against it. +fn pins_a_wired_ref(read: &NpmLockRefs, purl: &str, pin: Option<&str>) -> bool { + let Some(pin) = pin else { + return false; + }; + read.refs.iter().any(|r| { + r.purl == purl && matches!(&r.locked_integrity, Some(LockIntegrity::Sri(sri)) if sri == pin) + }) +} + /// npm installs a git / url / `file:` dependency from the dependent's spec -/// and ignores the entry's `resolved` (`vendor::npm_origin`, #326), so such -/// an entry stays unpatched whatever its `resolved` says. Every ref for the -/// same `name@version` is dropped (that copy is live beside it), and the -/// copy counts as resolved elsewhere, so other locks' wiring for it is -/// contested too. +/// and ignores the entry's `resolved` (`vendor::npm_origin`, #326), and npm +/// 7–11 install everything beneath a `hasShrinkwrap` package from that +/// package's own npm-shrinkwrap.json (#753), so such an entry stays +/// unpatched whatever its `resolved` says. Every ref for the same +/// `name@version` is dropped from attestation (that copy is live beside +/// it), and the copy counts as resolved elsewhere, so other locks' wiring +/// for it is contested too. +/// +/// A dropped ref wired at a REGISTRY entry (the rewriters' own output +/// beside a copy they skip) or beneath a `hasShrinkwrap` package (a +/// pre-#753 run's wiring) is still shadowed, so rollback / remove / the +/// takeover unwind it (#828). Only a wiring of the git / url / `file:` +/// entry itself — which no Socket rewriter writes — is not. fn drop_non_registry_installs( file: &str, doc: &Value, @@ -341,11 +430,34 @@ fn drop_non_registry_installs( out: &mut Discovery, ) { let non_registry = npm_non_registry_entries(doc, overrides); - if non_registry.is_empty() { + // Locations a wiring of which is not the rewriters' own output. + let foreign: BTreeSet = non_registry.keys().cloned().collect(); + // (lock key, why that copy installs from elsewhere) + let elsewhere: Vec<(String, String)> = non_registry + .into_iter() + .map(|(key, reason)| { + let why = format!( + "is not installed from the registry ({reason}); npm installs it from that spec" + ); + (key, why) + }) + .chain( + npm_shrinkwrapped_entries(doc) + .into_iter() + .map(|(key, ancestor)| { + let why = format!( + "is installed from `{ancestor}`'s own npm-shrinkwrap.json \ + (hasShrinkwrap), which npm 7–11 read instead of this lock" + ); + (key, why) + }), + ) + .collect(); + if elsewhere.is_empty() { return; } let mut unpatched: Vec<(String, &str, &str)> = Vec::new(); - for (key, reason) in &non_registry { + for (key, why) in &elsewhere { let entry = &doc["packages"][key.as_str()]; let key_name = key.rsplit_once("node_modules/").map_or("", |(_, n)| n); let name = entry @@ -364,22 +476,28 @@ fn drop_non_registry_installs( read.unwired .entry(purl.clone()) .or_insert_with(|| key.clone()); - unpatched.push((purl, key, reason)); + unpatched.push((purl, key, why)); } + let ref_locations = &read.ref_locations; read.refs.retain(|r| { - let Some((_, key, reason)) = unpatched.iter().find(|(p, _, _)| *p == r.purl) else { + let Some((_, key, why)) = unpatched.iter().find(|(p, _, _)| *p == r.purl) else { return true; }; out.diag( DIAG_REF_UNATTRIBUTABLE, file, format!( - "{file}: {} is wired to a Socket patch but lock entry `{key}` is not \ - installed from the registry ({reason}); npm installs it from that spec, so \ - that copy stays UNPATCHED and nothing is attested", + "{file}: {} is wired to a Socket patch but lock entry `{key}` {why}, so that \ + copy stays UNPATCHED and nothing is attested", r.purl ), ); + let ours = ref_locations + .iter() + .any(|(purl, location)| *purl == r.purl && !foreign.contains(location)); + if ours { + out.shadow(r.clone()); + } false }); } @@ -395,7 +513,12 @@ fn entry_ref( out: &mut Discovery, ) { let name = node.name; - read.mention(name, node.version); + // Built once: every entry is mentioned, and a registry entry (nearly + // all of them) is also recorded as resolved elsewhere. + let purl = node.version.and_then(|v| npm_purl(name, v)); + if let Some(purl) = &purl { + read.mentioned.insert(purl.clone()); + } let resolved = node.resolved; let Located { vendored, @@ -422,7 +545,7 @@ fn entry_ref( // A registry / git / tarball dependency: not ours. Remembered so a // sibling lock's Socket wiring for the same package is contested // (the npm pair here, any other lock by the orchestrator). - if let Some(purl) = node.version.and_then(|v| npm_purl(name, v)) { + if let Some(purl) = purl { out.resolved_elsewhere(file, Some(purl.clone())); read.unwired .entry(purl) @@ -462,9 +585,13 @@ fn entry_ref( ); return; } + read.ref_locations + .push((purl.clone(), location.to_string())); read.refs .push(PatchedRef::vendored(purl, &vref, file, integrity)); } else if let Some(uuid) = hosted_uuid { + read.ref_locations + .push((purl.clone(), location.to_string())); read.refs.push(PatchedRef::hosted( purl, uuid, @@ -1218,6 +1345,77 @@ mod tests { } } + /// REGRESSION (#899): a shrinkwrap with NO package-lock.json twin is + /// not attested. npm 12 never reads npm-shrinkwrap.json: it resolves the + /// tree from the registry and writes a fresh package-lock.json, so the + /// wiring reaches npm <= 11 only — hosted or vendored. The refs stay + /// (rollback, remove and list manage them) and are marked + /// [`UnattestedKind::NpmShrinkwrapOnly`]. A package-lock.json alone, and + /// a shrinkwrap with a wired twin, still attest. + #[tokio::test] + async fn a_shrinkwrap_without_a_package_lock_twin_is_not_attested() { + let hosted = hosted_url("npm", "left-pad", "1.3.0", UUID_A, "left-pad-1.3.0.tgz"); + let vendored = format!("file:.socket/vendor/npm/{UUID_B}/minimist-1.2.5.tgz"); + let wired = || { + lock_with_packages(serde_json::json!({ + "node_modules/left-pad": { "version": "1.3.0", "resolved": hosted, "integrity": SRI }, + "node_modules/minimist": { "version": "1.2.5", "resolved": vendored, "integrity": SRI }, + })) + }; + let p = Project::new(); + p.write("npm-shrinkwrap.json", wired()); + let out = run(&p).await; + assert_eq!(out.refs.len(), 2, "the wiring stays managed: {:#?}", out); + assert!(out.diagnostics.is_empty(), "{:?}", out.diagnostics); + assert_eq!(out.unattested.len(), 2, "{:#?}", out.unattested); + for (purl, uuid) in [ + ("pkg:npm/left-pad@1.3.0", UUID_A), + ("pkg:npm/minimist@1.2.5", UUID_B), + ] { + assert!( + out.unattested.iter().any(|u| u.purl == purl + && u.uuid == uuid + && u.kind == UnattestedKind::NpmShrinkwrapOnly + && u.file == std::path::Path::new("npm-shrinkwrap.json") + && u.detail.contains("no package-lock.json") + && u.detail.contains("npm >= 12") + && u.detail.contains("re-run the scan")), + "{purl}: {:#?}", + out.unattested + ); + } + + // A twin that exists but does not parse is unusable to npm 12 too, + // but the detail must not call it missing (or say to create it). + p.write("package-lock.json", "{ not json"); + let out = run(&p).await; + assert_eq!(out.unattested.len(), 2, "{:#?}", out.unattested); + for u in &out.unattested { + assert!( + u.kind == UnattestedKind::NpmShrinkwrapOnly + && u.detail.contains("cannot be read or parsed") + && !u.detail.contains("no package-lock.json") + && !u.detail.contains("rename the lock"), + "{u:#?}" + ); + } + + // The twin npm 12 reads makes both attestable again. + p.write("package-lock.json", wired()); + let out = run(&p).await; + assert_eq!(out.refs.len(), 4, "{:#?}", out.diagnostics); + assert!(out.diagnostics.is_empty(), "{:?}", out.diagnostics); + assert!(out.unattested.is_empty(), "{:#?}", out.unattested); + + // package-lock.json alone is what every npm >= 7 reads. + let p = Project::new(); + p.write("package-lock.json", wired()); + let out = run(&p).await; + assert_eq!(out.refs.len(), 2, "{:#?}", out.diagnostics); + assert!(out.diagnostics.is_empty(), "{:?}", out.diagnostics); + assert!(out.unattested.is_empty(), "{:#?}", out.unattested); + } + /// REGRESSION (#798 review): a sibling npm lock holding the wired /// package only at ANOTHER version contests it as well. npm keeps that /// entry only while it satisfies `package.json` (`^1.3.0` here rejects @@ -1532,6 +1730,82 @@ mod tests { assert!(out.diagnostics.is_empty(), "{:?}", out.diagnostics); } + /// #879: npm 7-12 `npm install` on a vendored lockfileVersion 2 lock (or + /// shrinkwrap) re-saves the legacy mirror node without `resolved` (npm + /// never writes one for a `file:` resolution there), keeping the patched + /// `integrity`. npm 6 fails closed on that pin, so the mirror agrees and + /// the `packages` ref is attested — for a plain dep and an alias alike. + #[tokio::test] + async fn v2_mirror_without_resolved_but_the_patched_pin_attests() { + let vendored = format!("file:.socket/vendor/npm/{UUID_B}/left-pad-1.3.0.tgz"); + for lock in ["package-lock.json", "npm-shrinkwrap.json"] { + for (key, mirror_version) in [("left-pad", "1.3.0"), ("lp", "npm:left-pad@1.3.0")] { + let p = Project::new(); + p.write( + lock, + serde_json::json!({ + "lockfileVersion": 2, + "packages": { + "": { "name": "app", "version": "1.0.0" }, + format!("node_modules/{key}"): { + "name": "left-pad", "version": "1.3.0", + "resolved": vendored, "integrity": SRI + } + }, + "dependencies": { + key: { "version": mirror_version, "integrity": SRI } + } + }) + .to_string(), + ); + let out = run(&p).await; + assert_refs( + &out, + &[("pkg:npm/left-pad@1.3.0", UUID_B, WiringMode::Vendored)], + ); + assert!( + out.diagnostics.is_empty(), + "{lock} {key}: {:?}", + out.diagnostics + ); + } + } + } + + /// #879's boundary: a `resolved`-less mirror node pinned to OTHER bytes + /// (the registry tarball's integrity) is what npm 6 installs unpatched, + /// so it still contests the ref (#432). + #[tokio::test] + async fn v2_mirror_without_resolved_on_another_pin_contests_the_ref() { + let vendored = format!("file:.socket/vendor/npm/{UUID_B}/left-pad-1.3.0.tgz"); + let p = Project::new(); + p.write( + "package-lock.json", + serde_json::json!({ + "lockfileVersion": 2, + "packages": { + "": { "name": "app", "version": "1.0.0" }, + "node_modules/left-pad": { + "version": "1.3.0", "resolved": vendored, "integrity": SRI + } + }, + "dependencies": { + "left-pad": { "version": "1.3.0", "integrity": "sha512-ORIG" } + } + }) + .to_string(), + ); + let out = run(&p).await; + assert!(out.refs.is_empty(), "{:#?}", out.refs); + assert!( + out.diagnostics + .iter() + .any(|d| d.code == DIAG_REF_UNATTRIBUTABLE && d.detail.contains("npm <= 6")), + "{:?}", + out.diagnostics + ); + } + /// link / inBundle / bundled entries install from somewhere else, so a /// Socket URL written there wires nothing. #[tokio::test] @@ -1606,6 +1880,18 @@ mod tests { ); let out = run(&p).await; assert!(out.refs.is_empty(), "{label}: {:#?}", out.refs); + // Withheld from attestation, but still the wiring of exactly + // this version, so rollback / remove / the takeover see it (#828). + let shadowed: Vec<_> = out + .shadowed + .iter() + .map(|r| (r.purl.as_str(), r.uuid.as_str(), r.mode)) + .collect(); + assert_eq!( + shadowed, + vec![("pkg:npm/left-pad@1.3.0", uuid, mode)], + "{label}" + ); let contested = bundled_contests(&out); assert_eq!(contested.len(), 1, "{label}: {:#?}", out.diagnostics); assert!( @@ -1641,6 +1927,100 @@ mod tests { assert!(bundled_contests(&out).is_empty(), "{:#?}", out.diagnostics); } + /// REGRESSION (#753): npm 7–11 install a copy beneath a `hasShrinkwrap` + /// package from that package's own npm-shrinkwrap.json and ignore the + /// root lock's entry, so a Socket url written there (by a pre-fix scan) + /// is never a ref, and a registry copy there contests the hoisted + /// wired entry of the same version, in either mode. + #[tokio::test] + async fn shrinkwrapped_copy_is_never_attested_and_contests_the_wired_entry() { + let hosted = hosted_url("npm", "left-pad", "1.3.0", UUID_A, "left-pad-1.3.0.tgz"); + let vendored = format!("file:.socket/vendor/npm/{UUID_B}/left-pad-1.3.0.tgz"); + let registry = "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz"; + let sw = serde_json::json!({ + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-SW", + "hasShrinkwrap": true, + }); + for (label, resolved) in [("hosted", &hosted), ("vendored", &vendored)] { + // The only copy is the shrinkwrapped one, rewired in the root lock. + let p = Project::new(); + p.write( + "package-lock.json", + lock_with_packages(serde_json::json!({ + "node_modules/@bh/sw": sw, + "node_modules/@bh/sw/node_modules/left-pad": { + "version": "1.3.0", "resolved": resolved, "integrity": SRI + }, + })), + ); + let out = run(&p).await; + assert!(out.refs.is_empty(), "{label}: {:#?}", out.refs); + assert!( + out.diagnostics + .iter() + .any(|d| d.code == DIAG_REF_UNATTRIBUTABLE + && d.detail.contains("hasShrinkwrap") + && d.detail + .contains("node_modules/@bh/sw/node_modules/left-pad")), + "{label}: {:#?}", + out.diagnostics + ); + // A pre-#753 run's wiring of that copy: still the rewriters' + // own output, so rollback / remove / the takeover unwind it. + assert!( + out.shadowed + .iter() + .any(|r| r.purl == "pkg:npm/left-pad@1.3.0"), + "{label}: {:#?}", + out.shadowed + ); + + // A wired hoisted copy beside a registry shrinkwrapped copy. + let p = Project::new(); + p.write( + "package-lock.json", + lock_with_packages(serde_json::json!({ + "node_modules/left-pad": { "version": "1.3.0", "resolved": resolved, "integrity": SRI }, + "node_modules/@bh/sw": sw, + "node_modules/@bh/sw/node_modules/left-pad": { + "version": "1.3.0", "resolved": registry, "integrity": "sha512-ORIG" + }, + })), + ); + let out = run(&p).await; + assert!(out.refs.is_empty(), "{label}: {:#?}", out.refs); + // The rewriter's own wiring beside the copy it skips + // (`redirect_npm_shrinkwrapped_instance_skipped`) is withheld + // from VEX but not lost (#828). + assert!( + out.shadowed + .iter() + .any(|r| r.purl == "pkg:npm/left-pad@1.3.0"), + "{label}: {:#?}", + out.shadowed + ); + } + + // Without `hasShrinkwrap` the nested wired copy is an ordinary ref. + let p = Project::new(); + p.write( + "package-lock.json", + lock_with_packages(serde_json::json!({ + "node_modules/@bh/sw": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz" }, + "node_modules/@bh/sw/node_modules/left-pad": { + "version": "1.3.0", "resolved": hosted, "integrity": SRI + }, + })), + ); + let out = run(&p).await; + assert_refs( + &out, + &[("pkg:npm/left-pad@1.3.0", UUID_A, WiringMode::Hosted)], + ); + } + /// REGRESSION (#325), lockfileVersion 1: a `bundled: true` copy nested in /// the parent's `dependencies` contests the wired top-level entry the /// same way. @@ -1860,6 +2240,13 @@ mod tests { "{spec} / {wiring}: {:?}", diag_codes(&out) ); + // The non-registry entry itself carries the wiring: no + // Socket rewriter writes that, so it is not shadowed either. + assert!( + out.shadowed.is_empty(), + "{spec} / {wiring}: {:#?}", + out.shadowed + ); } } // Transitive: the hoisted copy is wired, a nested git copy of the @@ -1894,6 +2281,18 @@ mod tests { "{}", diag.detail ); + // The hoisted wiring is the hosted rewriter's own output beside a + // copy it skips: withheld from VEX, still unwound by rollback / + // remove / the takeover (#828). + assert_eq!( + out.shadowed + .iter() + .map(|r| (r.purl.as_str(), r.uuid.as_str())) + .collect::>(), + vec![("pkg:npm/left-pad@1.3.0", UUID_A)], + "{:#?}", + out.shadowed + ); // Control: a registry spec keeps the ref. let p = Project::new(); p.write( diff --git a/crates/socket-patch-core/src/vex/discover/testing/golden.rs b/crates/socket-patch-core/src/vex/discover/testing/golden.rs index b3add1fc1..f0525c4c1 100644 --- a/crates/socket-patch-core/src/vex/discover/testing/golden.rs +++ b/crates/socket-patch-core/src/vex/discover/testing/golden.rs @@ -123,6 +123,7 @@ fn render(out: &Discovery, root: &Path) -> Value { unpatched_copies, unattested, contested, + shadowed, // Bookkeeping of what the vlt extractor read from the store, not a // finding: every copy it found already shows as a contest and a // diagnostic above. @@ -133,32 +134,30 @@ fn render(out: &Discovery, root: &Path) -> Value { // Already folded into `unattested` by the time a run returns. unwired_copies: _, } = out; - let refs: Vec = refs - .iter() - .map(|r| { - let PatchedRef { - purl, - uuid, - mode: m, - source_file, - artifact_rel, - locked_integrity, - integrity_required, - url, - } = r; - json!({ - "purl": purl, - "uuid": uuid, - "mode": mode(*m), - "source_file": path_str(source_file), - "artifact_rel": artifact_rel, - "locked_integrity": locked_integrity.as_ref().map(|i| format!("{i:?}")), - "integrity_required": integrity_required, - "url": url, - "lockfile_basis_ok": r.lockfile_basis_ok(), - }) + let render_ref = |r: &PatchedRef| { + let PatchedRef { + purl, + uuid, + mode: m, + source_file, + artifact_rel, + locked_integrity, + integrity_required, + url, + } = r; + json!({ + "purl": purl, + "uuid": uuid, + "mode": mode(*m), + "source_file": path_str(source_file), + "artifact_rel": artifact_rel, + "locked_integrity": locked_integrity.as_ref().map(|i| format!("{i:?}")), + "integrity_required": integrity_required, + "url": url, + "lockfile_basis_ok": r.lockfile_basis_ok(), }) - .collect(); + }; + let refs: Vec = refs.iter().map(render_ref).collect(); let diagnostics: Vec = diagnostics .iter() .map(|d| { @@ -260,6 +259,9 @@ fn render(out: &Discovery, root: &Path) -> Value { .collect::>() .into(); } + if !shadowed.is_empty() { + rendered["shadowed"] = shadowed.iter().map(render_ref).collect::>().into(); + } if !unpatched_copies.is_empty() { rendered["unpatched_copies"] = unpatched_copies .iter() diff --git a/crates/socket-patch-core/src/vex/discover/vlt.rs b/crates/socket-patch-core/src/vex/discover/vlt.rs index 660bd35c1..c6caa0023 100644 --- a/crates/socket-patch-core/src/vex/discover/vlt.rs +++ b/crates/socket-patch-core/src/vex/discover/vlt.rs @@ -283,6 +283,7 @@ async fn contest_bundled_copies(ctx: &DiscoverCtx<'_>, nodes: &[VltLockNode], ou r.purl, ), ); + out.shadow(r); } for purl in copies.into_keys() { out.resolved_elsewhere(VLT_LOCK, Some(purl)); diff --git a/crates/socket-patch-core/src/vex/discover/yarn.rs b/crates/socket-patch-core/src/vex/discover/yarn.rs index 4be8236c5..c871e772e 100644 --- a/crates/socket-patch-core/src/vex/discover/yarn.rs +++ b/crates/socket-patch-core/src/vex/discover/yarn.rs @@ -1195,6 +1195,18 @@ mod tests { p.write("yarn.lock", classic(&blocks)); let out = run(&p).await; assert!(out.refs.is_empty(), "{case}: {:#?}", out.refs); + // The registry wiring beside the git copy is withheld, not lost: + // rollback / remove / the takeover still unwind it (#828). + let shadowed = out + .shadowed + .iter() + .any(|r| r.purl == "pkg:npm/left-pad@1.3.0"); + assert_eq!( + shadowed, + case.starts_with("registry"), + "{case}: {:#?}", + out.shadowed + ); assert!( out.diagnostics .iter() diff --git a/crates/socket-patch-core/tests/hosted_inventory.rs b/crates/socket-patch-core/tests/hosted_inventory.rs index 1c9c1e65f..504e3ec40 100644 --- a/crates/socket-patch-core/tests/hosted_inventory.rs +++ b/crates/socket-patch-core/tests/hosted_inventory.rs @@ -213,6 +213,144 @@ async fn half_reverted_uv_pair_is_still_contested() { assert!(inv.contested_refusal().is_some(), "{inv:?}"); } +// ── pins withheld from VEX by an unreachable unpatched copy (#828) ─────── +// The hosted rewriters rewire every entry they can and skip a copy no +// rewire reaches (an npm bundled copy, a yarn classic git block). Discovery +// does not attest that pin, but it is still the rewriter's own wiring of +// one package version: management commands restore it, never refuse it. + +#[tokio::test] +async fn npm_pin_beside_a_bundled_copy_is_an_attributable_pin() { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + std::fs::write( + root.join("package.json"), + r#"{"name":"app","version":"1.0.0","dependencies":{"left-pad":"1.3.0","bund":"file:bund-1.0.0.tgz"}}"#, + ) + .unwrap(); + let mut lock: serde_json::Value = + serde_json::from_str(&lock(&hosted_url(), "sha512-patched==")).unwrap(); + let packages = lock["packages"].as_object_mut().unwrap(); + packages.insert( + "node_modules/bund".into(), + serde_json::json!({ "version": "1.0.0", "resolved": "file:bund-1.0.0.tgz" }), + ); + packages.insert( + "node_modules/bund/node_modules/left-pad".into(), + serde_json::json!({ + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-upstream==", + "inBundle": true, + }), + ); + std::fs::write(root.join("package-lock.json"), lock.to_string()).unwrap(); + + let discovery = socket_patch_core::vex::discover_patched_refs(root).await; + assert!(discovery.refs.is_empty(), "never attested: {discovery:#?}"); + let inv = HostedInventory::of(&discovery); + assert_eq!(inv.pins.len(), 1, "{inv:?}"); + assert_eq!(inv.pins[0].purl, "pkg:npm/left-pad@1.3.0"); + assert_eq!(inv.pins[0].uuid, PATCH); + assert_eq!(inv.pins[0].files, vec!["package-lock.json".to_string()]); + assert!(inv.contested_refusal().is_none(), "{inv:?}"); +} + +#[tokio::test] +async fn yarn_classic_pin_beside_a_git_copy_is_an_attributable_pin() { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + std::fs::write( + root.join("package.json"), + r#"{"name":"app","version":"1.0.0","private":true}"#, + ) + .unwrap(); + let url = hosted_url(); + std::fs::write( + root.join("yarn.lock"), + format!( + "# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.\n\ + # yarn lockfile v1\n\n\n\ + \"left-pad@git+https://github.com/stevemao/left-pad.git#v1.3.0\":\n \ + version \"1.3.0\"\n \ + resolved \"git+https://github.com/stevemao/left-pad.git#5e5f1a6e23f6fa2bd1e4a3d0c2bb1c0e1bb0f00a\"\n\n\ + left-pad@^1.3.0:\n \ + version \"1.3.0\"\n \ + resolved \"{url}\"\n \ + integrity sha512-patched==\n" + ), + ) + .unwrap(); + + let discovery = socket_patch_core::vex::discover_patched_refs(root).await; + assert!(discovery.refs.is_empty(), "never attested: {discovery:#?}"); + let inv = HostedInventory::of(&discovery); + assert_eq!(inv.pins.len(), 1, "{inv:?}"); + assert_eq!(inv.pins[0].purl, "pkg:npm/left-pad@1.3.0"); + assert_eq!(inv.pins[0].files, vec!["yarn.lock".to_string()]); + assert!(inv.contested_refusal().is_none(), "{inv:?}"); +} + +/// The twin over a copy beneath a `hasShrinkwrap` package (#753): npm 7–11 +/// install it from that package's own npm-shrinkwrap.json, so the hosted +/// rewriter skips it (`redirect_npm_shrinkwrapped_instance_skipped`), and +/// over a nested git copy (#326). The hoisted pin is still the rewriter's +/// own wiring. +#[tokio::test] +async fn npm_pin_beside_a_shrinkwrapped_or_git_copy_is_an_attributable_pin() { + let shrinkwrapped = ( + serde_json::json!({ + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-sw==", + "hasShrinkwrap": true, + }), + serde_json::json!({ + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-upstream==", + }), + ); + let git = ( + serde_json::json!({ + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@bh/sw/-/sw-1.0.0.tgz", + "integrity": "sha512-sw==", + "dependencies": { "left-pad": "stevemao/left-pad#v1.3.0" }, + }), + serde_json::json!({ + "version": "1.3.0", + "resolved": "git+ssh://git@github.com/stevemao/left-pad.git#ff8e7ba", + }), + ); + for (case, (parent, child)) in [("hasShrinkwrap", shrinkwrapped), ("git", git)] { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + std::fs::write( + root.join("package.json"), + r#"{"name":"app","version":"1.0.0","dependencies":{"left-pad":"1.3.0","@bh/sw":"1.0.0"}}"#, + ) + .unwrap(); + let mut lock: serde_json::Value = + serde_json::from_str(&lock(&hosted_url(), "sha512-patched==")).unwrap(); + let packages = lock["packages"].as_object_mut().unwrap(); + packages.insert("node_modules/@bh/sw".into(), parent); + packages.insert("node_modules/@bh/sw/node_modules/left-pad".into(), child); + std::fs::write(root.join("package-lock.json"), lock.to_string()).unwrap(); + + let discovery = socket_patch_core::vex::discover_patched_refs(root).await; + assert!( + discovery.refs.is_empty(), + "{case}: never attested: {discovery:#?}" + ); + let inv = HostedInventory::of(&discovery); + assert_eq!(inv.pins.len(), 1, "{case}: {inv:?}"); + assert_eq!(inv.pins[0].purl, "pkg:npm/left-pad@1.3.0"); + assert_eq!(inv.pins[0].uuid, PATCH); + assert!(inv.contested_refusal().is_none(), "{case}: {inv:?}"); + } +} + /// A lockless NuGet pin (an exclusive Socket source mapping, no /// `packages.lock.json`: `rewrite_nuget`'s output for most projects) is /// contested wiring nothing can attribute to a version. Its refusal names diff --git a/docs/configuration.md b/docs/configuration.md index 274692dcd..abecba177 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -125,6 +125,13 @@ socket-patch scan 'apps/*' --mode hosted Each PATH in hosted or vendored mode is a project directory. Paths outside the repository are rejected. For JSON output, scan one project per invocation. +An agent-mode scan from the repository root also patches nested projects' +`node_modules`. Each installed copy follows its own project root (the nearest +directory with a lockfile), so these filters skip nested projects too. A package +an included project also installs is patched in every copy, because patches are +recorded per package version; the scan warns `policy_shared_copy` when that reaches +a skipped project. + ### Gradual rollout ```sh diff --git a/docs/ecosystems.md b/docs/ecosystems.md index 1401cbb47..b37e3f608 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -40,8 +40,15 @@ The backticked slug in each row is the value `-e`/`--ecosystems` accepts (e.g. ## npm hosted-mode notes - **npm (package-lock.json / npm-shrinkwrap.json)** — every present npm lock is - rewritten (npm 12 installs from the package-lock.json twin it keeps beside a - committed shrinkwrap). npm 12 defaults `allow-remote=none` and refuses the + rewritten (npm <= 11 installs from a committed shrinkwrap, npm 12 only from + package-lock.json). npm 12 never reads npm-shrinkwrap.json: on a + shrinkwrap-only project it resolves the tree from the registry and writes a + fresh package-lock.json, so the patch reaches npm <= 11 only. Hosted and + vendored runs still rewire the shrinkwrap but warn + (`redirect_npm_shrinkwrap_only` / `vendor_npm_shrinkwrap_only`), and `vex` + omits those patches (`vex_npm_shrinkwrap_only`) while `list`, `rollback` and + `remove` still manage them; rename the lock to package-lock.json (or commit + a copy under that name) and re-run. npm 12 defaults `allow-remote=none` and refuses the redirected tarballs (EALLOWREMOTE) unless the project `.npmrc` sets `allow-remote=all`, so the hosted run writes it (new file, or one appended line; once `rollback` / `remove` / the vendored takeover has restored the last @@ -53,8 +60,18 @@ The backticked slug in each row is the value `-e`/`--ecosystems` accepts (e.g. overridden) — in the project `.npmrc`, the user / global / builtin npm config, or an `npm_config_allow_remote` environment variable — and `--no-npm-allow-remote-config` opts out (install with - `npm ci --allow-remote=all`). Vendored `file:` tarballs are unaffected (npm - gates them by `allow-file`, default `all`). npm 6 ignores `resolved` for registry + `npm ci --allow-remote=all`). Vendored `file:` tarballs are unaffected by + `allow-remote`: npm >= 11.14 gates them by `allow-file` (default `all`). An + explicit `allow-file=none`, or `allow-file=root` while a vendored copy is + transitive, makes every install fail EALLOWFILE; the vendored run keeps the + setting, warns `vendor_npm_allow_file` with the remedy (`allow-file=all` in + `.npmrc`, or `npm ci --allow-file=all`), and `vendor --check` fails. npm >= 8's + `replace-registry-host` set to `always` (or to the hosted patch host) makes npm + rewrite the hosted pins to the configured registry, so every install fails + E404: the hosted run reads it from the same env / project / user / global / + builtin layers and warns `redirect_npm_replace_registry_host` (set + `replace-registry-host=npmjs` in the project `.npmrc`, or use vendored mode). + npm 6 ignores `resolved` for registry dependencies, so a redirected lockfileVersion 1 lock fails closed with EINTEGRITY under npm 6 (`redirect_npm_legacy_client`) and installs under npm >= 7. A lockfileVersion 2 lock's legacy `dependencies` mirror is rewired with @@ -63,7 +80,11 @@ The backticked slug in each row is the value `-e`/`--ecosystems` accepts (e.g. whatever its `resolved` says, so under npm 6 an aliased hosted pin in a v2 lock fails closed with EINTEGRITY too (`redirect_npm_legacy_alias_client`). Lockfile-only `vex` attests nothing for a package whose `packages` entry is - wired while the v2 mirror still resolves it from the registry. Vendoring + wired while the v2 mirror still resolves it from the registry. A mirror + node with no `resolved` but the patched `integrity` (what npm 7–12 + `npm install` leaves on a vendored v2 lock, since npm never writes a + `file:` `resolved` there) still counts as wired for `vex` and + `vendor --check`: npm 6 fails closed on that pin. Vendoring needs a lockfileVersion 2/3 lock (npm 6 still installs a vendored v2 lock from its legacy mirror, alias nodes included) and rewires both locks in npm 12's dual-lock state. Majors 6–12 are measured in @@ -247,8 +268,10 @@ to first-party source: an npm, Yarn, pnpm or Bun workspace member, a `file:` or `link:` directory dependency, or an `npm link` target. That source is not an installed copy of the registry package, and no reinstall restores it, so it is never overwritten. Patch it directly instead (vendored -mode refuses it the same way, with `vendor_workspace_member`). Links into a -store inside a `node_modules` tree, including a workspace member's link +mode refuses it the same way, with `vendor_workspace_member`; when an npm lock +also holds a registry copy of the same `name@version`, that copy is still +vendored and the local source is skipped with `vendor_workspace_member_skipped`). +Links into a store inside a `node_modules` tree, including a workspace member's link into the root `node_modules/.pnpm`, are patched as usual. So are links into Yarn's pnpm-linker store relocated outside `node_modules`, but only for an active Yarn pnpm install (a `yarn.lock`, and `nodeLinker: pnpm` with diff --git a/docs/testing/npm-compatibility.md b/docs/testing/npm-compatibility.md index cd484910a..297a45779 100644 --- a/docs/testing/npm-compatibility.md +++ b/docs/testing/npm-compatibility.md @@ -17,10 +17,19 @@ Measured against the real releases (Node 24.21 on macOS, 2026-09-22): | 6.14.18 | lockfileVersion 1 | renames the lock | **fails closed**: EINTEGRITY — npm 6 fetches registry dependencies from the configured registry and ignores `resolved`, so the patched sha512 pin rejects the registry bytes (`redirect_npm_legacy_client` warns) | a v1 lock is refused (`vendor_lockfile_version_unsupported`), and a hosted → vendored takeover refuses it before restoring the hosted pin, so the package stays hosted; npm 6 DOES install a vendored **v2** lock (written by npm 7+) from its legacy `dependencies` mirror | | 7.0.0, 7.24.2, 8.19.4 | lockfileVersion 2 (+ v1 mirror) | renames the lock | patched (npm 6 installing this v2 lock: patched, except an npm alias, which npm 6 fetches from the registry, so it **fails closed** with EINTEGRITY and `redirect_npm_legacy_alias_client` warns) | patched (npm 6 installing this v2 lock: patched, alias nodes included) | | 9.0.0, 9.9.4, 10.9.9, 11.20.0 | lockfileVersion 3 | renames the lock | patched | patched | -| 12.0.0, 12.1.0 | lockfileVersion 3 | **removed** — a committed shrinkwrap gets a package-lock.json twin on first install, and installs read the twin | patched with a plain `npm ci` — the hosted run writes `allow-remote=all` to the project `.npmrc` (`redirect_npm_allow_remote` warns on every npm hosted run); the same checkout WITHOUT that `.npmrc` is refused EALLOWREMOTE | patched (both locks are rewired in the dual-lock state) | +| 12.0.0, 12.1.0 | lockfileVersion 3 | **removed** — npm 12 never reads npm-shrinkwrap.json: a shrinkwrap-only checkout is resolved from the registry into a fresh package-lock.json (so its patches stay unpatched under npm 12, see below), and installs read package-lock.json | patched with a plain `npm ci` — the hosted run writes `allow-remote=all` to the project `.npmrc` (`redirect_npm_allow_remote` warns on every npm hosted run); the same checkout WITHOUT that `.npmrc` is refused EALLOWREMOTE | patched (both locks are rewired in the dual-lock state) | npm 12 notes: +- `npm patch add` / `npm patch commit` (npm >= 12.1) record the project's own + diff in the root `package.json` `patchedDependencies`, store it under + `patches/`, add a `patched: {integrity, path}` record to the lock entry and + write lockfileVersion 4. Every install extracts the locked tarball and then + applies that diff, failing `EPATCHFAILED` when it no longer applies. Hosted + mode therefore leaves such a package on its registry entry, in every npm + lock, and warns `redirect_npm_patched_dependency_skipped`; other packages in + the lock are still pinned. Vendored mode refuses the v4 lock + (`vendor_lockfile_version_unsupported`, naming `npm patch`) (#711). - `allow-remote` defaults to `none`: any tarball whose `resolved` origin is not the configured registry is refused. `allow-remote=root` admits only direct dependencies. `allow-remote=all` is the setting a hosted redirect needs; it @@ -47,7 +56,11 @@ npm 12 notes: value case-sensitively (`all` admits everything, `none` nothing, anything else — `root`, `All`, a typo — only direct dependencies). - `npm ci` refuses a project whose only lock is npm-shrinkwrap.json; `npm - install` copies it to package-lock.json and installs from the copy. With + install` ignores it, resolves the tree from the registry and writes a fresh + package-lock.json (npm 12.0.0 – 12.2.0, #899), so a shrinkwrap-only + rewrite reaches npm <= 11 only: hosted and vendored runs warn + `redirect_npm_shrinkwrap_only` / `vendor_npm_shrinkwrap_only` and `vex` + omits the patch (`vex_npm_shrinkwrap_only`). With both present, npm 12 installs from package-lock.json while npm <= 11 installs from the shrinkwrap — so hosted and vendored rewrites wire BOTH, and manifest-less VEX refuses to attest a package one lock wires while the other @@ -55,7 +68,21 @@ npm 12 notes: (missing, or only another version that may no longer satisfy `package.json`), which npm can re-resolve from the registry (`patched_ref_unattributable`). -- `allow-file` defaults to `all`: vendored `file:` tarballs install unchanged. +- `allow-file` (npm >= 11.14) defaults to `all`: vendored `file:` tarballs + install unchanged. An explicit `none` (or `root` with a transitive vendored + copy) refuses them with EALLOWFILE; the vendored run warns + `vendor_npm_allow_file` and `vendor --check` fails (#969). + +Dependencies that ship their own `npm-shrinkwrap.json` (#753): the lock marks +such a package `"hasShrinkwrap": true` (`firebase-tools`, `netlify-cli`), and +npm 7–11 install everything beneath it from that package's own shrinkwrap, +ignoring the root lock's entries (npm 12.2.0 honors the root lock). A copy of +the patched `name@version` there is never rewired: hosted and vendored scans +skip it with `redirect_npm_shrinkwrapped_instance_skipped` / +`vendor_shrinkwrapped_instance_skipped` (vendoring refuses with +`vendor_lock_entry_not_rewritable` when it is the only copy), and VEX does not +attest the package while that unpatched copy installs +(`patched_ref_unattributable`); `vendor --check` fails naming that copy. ## Suites @@ -64,7 +91,7 @@ npm 12 notes: | `e2e_redirect_npm_build` (`#[ignore]`) | real | scan / get-uuid / get-ghsa hosted redirects, shrinkwrap flavor, tampered-tarball rejection, fresh-checkout `npm ci`, manifest-less VEX tail | | `e2e_vendor_npm_build` | real | vendor / get-vendored, shrinkwrap flavor, npm 6 × v2 lock, idempotency, byte-exact revert, manifest-less VEX tail | | `e2e_vex_lockfile::npm` | none | tamper / spoof / mismatch / pin cells over lockfileVersion 1, 2, 3, shrinkwrap and dual-lock shapes | -| `redirect_npm_allow_remote` | none | the npm 12 `allow-remote` auto-config: `.npmrc` create / append (BOM, CRLF), explicit values respected (project file, user / global config, env var), unhonored spellings, bare-CR and indented-section files, section-scoped copies, opt-out flag + env, dry run, symlinked `.npmrc`, `--silent`, rollback/removal deleting a standalone setting or warning `npm_allow_remote_left` when other settings remain | +| `redirect_npm_allow_remote` | none | the npm 12 `allow-remote` auto-config: `.npmrc` create / append (BOM, CRLF), explicit values respected (project file, user / global config, env var), unhonored spellings, bare-CR and indented-section files, section-scoped copies, opt-out flag + env, dry run, symlinked `.npmrc`, `--silent`, rollback/removal deleting a standalone setting or warning `npm_allow_remote_left` when other settings remain; `replace-registry-host` rewriting the hosted pin (project file, user config, env var) warned `redirect_npm_replace_registry_host` | | `e2e_hosted_production` / `e2e_vendored_production` (`#[ignore]`) | real (ambient) | the same flows against production, ending in the manifest-less VEX tail | The manifest-less VEX tail (`tests/npm_e2e_common/manifestless.rs`) runs four