diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 6959bc5ee..e5cf877dc 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -962,7 +962,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`); every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). - * **gem** — `Gemfile.lock` / `gems.locked` + `Gemfile` / `gems.rb`: the spec moves back into the upstream `GEM` section (or the Socket remote leaves a merged section), the `source "" do … end` block is undone, the `CHECKSUMS` entry is re-pinned from the rubygems.org compact index (`SOCKET_RUBYGEMS_URL`) and the `DEPENDENCIES` pin loses its `!`. The declaration's original constraint is not recorded, so it comes back as the exact pin `gem "", ""`. A transitive gem (one the manifest never declared) gets an appended block with a blank line before it; the restore removes that block, its blank line and the `DEPENDENCIES` entry, so the pair comes back byte for byte. An appended block with no blank line before it (written by a release before this one) can't be told apart from an in-place rewrite, so it still comes back as the exact pin. Refused: an ambiguous upstream section, an upstream remote other than rubygems.org. + * **gem** — `Gemfile.lock` / `gems.locked` + `Gemfile` / `gems.rb`: the spec moves back into the upstream `GEM` section (or the Socket remote leaves a merged section), the `source "" do … end` block is undone, the `CHECKSUMS` entry is re-pinned from the rubygems.org compact index (`SOCKET_RUBYGEMS_URL`) and the `DEPENDENCIES` pin loses its `!`. The declaration's original constraint is not recorded, so it comes back as the exact pin `gem "", ""`. A transitive gem (one the manifest never declared) gets an appended block with a blank line before it; the restore removes that block, its blank line and the `DEPENDENCIES` entry, so the pair comes back byte for byte. An appended block with no blank line before it (written by a release before this one) can't be told apart from an in-place rewrite, so it still comes back as the exact pin. Refused: an ambiguous upstream section, an upstream remote other than rubygems.org. The manifest pair can't make a committed bundler cache upstream: a `-.gem` left in Bundler's cache dir that isn't the upstream archive is named by `upstream_gem_stale_cache`, never deleted. * **composer** — `composer.lock`: `dist` and the deleted `source` block from packagist's composer v2 metadata (`SOCKET_PACKAGIST_URL`). Refused unless the entry is packagist-sourced and packagist still serves the lock's `dist.reference` for the version. * **maven** — `pom.xml` (the `-socket.` version suffix, the added `` / `` entry) and the `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` lines hosted mode writes: **no network**, so it restores under `--offline` too. `.mvn` files holding anything else keep the resolver lines (`maven_trusted_checksums_left`). * **nuget** — `nuget.config` loses the `socket-patch-` source and its exact-id mapping; every `packages.lock.json` entry of the id gets nuget.org's `contentHash` back (`SOCKET_NUGET_URL`). Refused when the restored config would not resolve the id from nuget.org alone. A config hosted mode created from scratch is kept (`nuget_default_config_left`). @@ -977,7 +977,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem | Key | Shape | Meaning | |---|---|---| | `error` | `{code, message}` | Only on `status: "error"` (v5.0, MAJOR: was a string). Codes: `manifest_not_found`, `manifest_invalid`, `manifest_unreadable`, `patch_not_found`, `path_glob_no_match`, `hosted_wiring_contested`, `vendor_ledger_missing`, `rollback_failed`, `lock_held` / `lock_io`, and `path_glob_invalid` (a usage error, exit 2). Per-result `results[*].error` stays a string. | -| `warnings` | `[{code, detail}]` | Run-level warnings, now populated (previously always empty): `reinstall_required`, `hosted_state_not_preservable`, `out_of_scope_copies_restored`, `vendor_state_unreadable`, `cleanup_failed`, `manifest_write_failed`, `legacy_redirect_ledger_kept`, the upstream-restore advisories (`npm_allow_remote_left`, `pnpm_trust_lockfile_left`, `maven_trusted_checksums_left`, `nuget_default_config_left`, `upstream_uv_override_removed`, `upstream_registry_fallback`, `upstream_pnpm_tarball_setting_guessed`), `ownership_not_restored` (a restored file whose ownership could not be put back — see the apply warnings), `rollback_record_superseded` (a manifest record superseded by a live hosted pin, left to the hosted leg — see Manifest cleanup), plus vendored/hosted leg advisories. New codes are additive (MINOR) | +| `warnings` | `[{code, detail}]` | Run-level warnings, now populated (previously always empty): `reinstall_required`, `hosted_state_not_preservable`, `out_of_scope_copies_restored`, `vendor_state_unreadable`, `cleanup_failed`, `manifest_write_failed`, `legacy_redirect_ledger_kept`, the upstream-restore advisories (`npm_allow_remote_left`, `pnpm_trust_lockfile_left`, `maven_trusted_checksums_left`, `nuget_default_config_left`, `upstream_uv_override_removed`, `upstream_registry_fallback`, `upstream_pnpm_tarball_setting_guessed`, `upstream_gem_stale_cache`), `ownership_not_restored` (a restored file whose ownership could not be put back — see the apply warnings), `rollback_record_superseded` (a manifest record superseded by a live hosted pin, left to the hosted leg — see Manifest cleanup), plus vendored/hosted leg advisories. New codes are additive (MINOR) | | `vendored` | `[purl]` | **Meaning narrowed (MAJOR)**: vendor-owned purls the run did NOT act on — today exactly the corrupt-vendor-ledger skip. | | `vendoredReverted` | `[purl]` | Ledger entries cleanly reverted this run (unwired + artifact deleted + entry dropped; previewed on dry-run) | | `vendoredPreserved` | `[purl]` | `--preserve-state`: unwired with artifact + ledger entry kept | @@ -1280,6 +1280,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `npm_allow_remote_left` / `pnpm_trust_lockfile_left` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): no npm-family lock entry is hosted any more, but the project `.npmrc` keeps a top-level `allow-remote=all` (resp. `pnpm-workspace.yaml` keeps `trustLockfile: true`) in a file that is not exactly what hosted mode creates; the file is left untouched (v5 records no provenance), remove the line if nothing else needs it. A file that is exactly hosted mode's own is deleted silently. | | `maven_trusted_checksums_left` / `nuget_default_config_left` / `upstream_uv_override_removed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): `.mvn` config keeps the trusted-checksums resolver lines because it holds more than hosted mode writes; `nuget.config` now holds only the nuget.org source (delete it if hosted mode created it); a transitive `override-dependencies` entry hosted mode added to `pyproject.toml` was removed. | | `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (`.yarnrc.yml` `npmRegistryServer`, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials), so the default registry's document was used and the restored tarball URL may not be the mirror's. | +| `upstream_gem_stale_cache` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#1260): a restored gem's `-.gem` is still in Bundler's cache dir (`cache_path`, default `vendor/cache`, resolved as for the Gem stale-install guard) and its sha256 is not the upstream one (the restored `CHECKSUMS` entry, else the rubygems.org compact index), or it could not be checked (`--offline`, a registry error). Bundler installs from that dir first, so a `bundle cache` taken while the hosted pin was live makes every later install fail on the upstream checksum (exit 37) or, on bundler < 2.6 frozen installs, keep installing the patched bytes. The detail names the file. Remedy: delete it, then run `bundle cache` to cache the upstream gem in its place (or `bundle install` if the project does not commit its cache; with the cache dir committed, a frozen install reads only the cache). Read-only: the restore never deletes it. Not raised for an archive whose sha256 matches upstream. | | `upstream_pnpm_tarball_setting_guessed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#902): nothing showed which pnpm wrote a hosted `pnpm-lock.yaml` (no unpinned registry entry that shows the setting, no `node_modules/.modules.yaml` install record, no package.json `packageManager` pin; for a Rush lock, no rush.json `pnpmVersion`), so its entries were restored with or without `tarball:` by pnpm 10's reading of `lockfileIncludeTarballUrl` (pnpm-workspace.yaml, else `.npmrc` `lockfile-include-tarball-url`), and pnpm 9 (which reads only `.npmrc`) or pnpm >= 11 (which reads only pnpm-workspace.yaml) would have read it the other way. The detail names the lock, the setting followed, the pnpm that disagrees and the entries. Remedy: pin the pnpm (package.json `packageManager`, or reinstall so the install record names it; rush.json `pnpmVersion` for Rush), or give the two files the same value so every pnpm reads it alike; a rollback or remove can then be redone by restoring the lock from version control and re-running. Not raised when evidence decided, when both files read the same on every pnpm, or when the tarball is one pnpm records regardless. | | `legacy_redirect_ledger_kept` | rollback `warnings[]` (+ remove stderr) | v5.0: a pre-v5 `.socket/vendor/redirect-state.json` could not be deleted once no hosted pin was left; the file is inert (never read for planning). Never flips the exit. | | `vendor_stale_artifact_removed` | `removed` | vendor / scan `--mode vendored`: re-vendor under a newer patch uuid removed the previous uuid's orphaned artifact dir. | diff --git a/crates/socket-patch-cli/tests/e2e_redirect_gem_build.rs b/crates/socket-patch-cli/tests/e2e_redirect_gem_build.rs index 5e7a985e1..e8842634e 100644 --- a/crates/socket-patch-cli/tests/e2e_redirect_gem_build.rs +++ b/crates/socket-patch-cli/tests/e2e_redirect_gem_build.rs @@ -3311,3 +3311,161 @@ async fn gem_hosted_cap_zero_upgrades_a_superseded_gemfile_only_pin() { "the Gemfile must move to the superseding patch:\n{gemfile}" ); } + +/// Copy the committable files of `src` (manifest pair, `.bundle`, the +/// committed bundler cache) into a new dir: a fresh checkout with no +/// installed tree. +fn checkout_with_cache(fx: &RedirectFixture, src: &Path, name: &str) -> PathBuf { + let fresh = fx.tmp.path().join(name); + std::fs::create_dir_all(&fresh).unwrap(); + for file in [fx.gemfile_name, fx.lock_name] { + std::fs::copy(src.join(file), fresh.join(file)).unwrap(); + } + copy_dir_recursive(&src.join(".bundle"), &fresh.join(".bundle")); + copy_dir_recursive(&src.join("vendor/cache"), &fresh.join("vendor/cache")); + fresh +} + +/// #1260 through one unwind `command` (`rollback` / `remove`): a project +/// that ran `bundle cache` while the hosted pin was live commits the +/// PATCHED archive. The unwind puts the manifest pair back on rubygems.org +/// but cannot make that archive upstream, and bundler installs from its +/// cache first: a fresh checkout either fails on the restored upstream +/// checksum (exit 37) or keeps installing the patched bytes. The unwind +/// must name the file, and deleting it must be the whole remedy. +fn unwind_names_the_patched_cached_archive(fx: &RedirectFixture, command: &str) { + let dir = stage_fresh_checkout(fx, &format!("cache-{command}")); + let install = bundle(&dir, &["install"]); + assert!( + install.status.success(), + "{command}: hosted install:\n{}", + String::from_utf8_lossy(&install.stderr) + ); + let cache_cmd = if fx.bundler.at_least(2, 0) { + "cache" + } else { + "package" + }; + let cache = bundle(&dir, &[cache_cmd]); + assert!( + cache.status.success(), + "{command}: bundle {cache_cmd}:\n{}", + String::from_utf8_lossy(&cache.stderr) + ); + let archive = dir + .join("vendor") + .join("cache") + .join(format!("{DEP}-{DEP_VERSION}.gem")); + assert!( + archive.is_file(), + "{command}: bundle {cache_cmd} wrote no archive" + ); + + let api = fx._server.uri(); + let upstream = format!("{api}/upstream"); + let cwd = dir.to_str().expect("utf8 tmp path"); + let mut argv = vec![command, PURL, "--json", "--cwd", cwd]; + if command == "remove" { + argv.push("--yes"); + } + argv.extend([ + "--api-url", + &api, + "--org", + ORG, + "--api-token", + "fake", + "--patch-server-url", + &api, + ]); + let (code, stdout, stderr) = run_socket_env(&dir, &argv, &[("SOCKET_RUBYGEMS_URL", &upstream)]); + let env: serde_json::Value = serde_json::from_str(&stdout).unwrap_or_else(|e| { + panic!("{command}: not JSON: {e}\nstdout:\n{stdout}\nstderr:\n{stderr}") + }); + assert_eq!(code, 0, "{command}: {env}\nstderr:\n{stderr}"); + let lock = std::fs::read_to_string(dir.join(fx.lock_name)).unwrap(); + assert!( + !lock.contains(&fx.index_url), + "{command}: the lock is back on upstream:\n{lock}" + ); + let hits: Vec<&serde_json::Value> = env["warnings"] + .as_array() + .map(|w| { + w.iter() + .filter(|w| w["code"] == "upstream_gem_stale_cache") + .collect() + }) + .unwrap_or_default(); + assert_eq!(hits.len(), 1, "{command}: one stale-cache warning: {env}"); + let detail = hits[0]["detail"].as_str().unwrap(); + assert!( + detail.contains(&archive.display().to_string()), + "{command}: the warning names the cached archive: {detail}" + ); + + // The defect the warning is about: with the archive left in place a + // fresh frozen checkout never installs the upstream bytes. + let stale = checkout_with_cache(fx, &dir, &format!("cache-{command}-stale")); + let install = bundle_env(&stale, &["install"], &[("BUNDLE_FROZEN", "true")]); + let lib = fresh_installed_lib(&stale, &format!("{DEP}-{DEP_VERSION}"), "vuln_gem.rb"); + assert!( + !install.status.success() || std::fs::read(&lib).unwrap() == fx.patched, + "{command}: bundler {} must reuse the cached patched archive (test premise)", + fx.bundler.version + ); + + // The remedy the warning prescribes: delete the archive and re-cache + // (a frozen install with a committed cache dir reads only the cache + // on some bundlers), and a fresh frozen checkout installs upstream. + let recached = checkout_with_cache(fx, &dir, &format!("cache-{command}-recached")); + std::fs::remove_file( + recached + .join("vendor/cache") + .join(format!("{DEP}-{DEP_VERSION}.gem")), + ) + .unwrap(); + let cache = bundle(&recached, &[cache_cmd]); + assert!( + cache.status.success(), + "{command}: bundle {cache_cmd} after deleting the archive:\n{}", + String::from_utf8_lossy(&cache.stderr) + ); + assert_eq!( + std::fs::read_to_string(recached.join(fx.lock_name)).unwrap(), + lock, + "{command}: re-caching leaves the restored lock alone" + ); + let fixed = checkout_with_cache(fx, &recached, &format!("cache-{command}-fixed")); + let install = bundle_env(&fixed, &["install"], &[("BUNDLE_FROZEN", "true")]); + assert!( + install.status.success(), + "{command}: frozen install from the re-cached checkout:\n{}", + String::from_utf8_lossy(&install.stderr) + ); + let lib = fresh_installed_lib(&fixed, &format!("{DEP}-{DEP_VERSION}"), "vuln_gem.rb"); + assert_eq!( + std::fs::read(&lib).unwrap(), + orig_lib().into_bytes(), + "{command}: the upstream bytes are installed" + ); +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "host capstone: shells out to a real ruby/gem/bundler (>= 1.17; CHECKSUMS arm >= 2.6); \ + the unpinned `test` job skips it, an e2e job with a pinned toolchain runs it via --ignored"] +async fn gem_hosted_unwind_names_a_patched_archive_in_vendor_cache() { + let Some(fx) = redirect_scanned_project( + "cache-unwind", + Spelling::Gemfile, + false, + true, + None, + Driver::ScanVex, + ) + .await + else { + return; + }; + unwind_names_the_patched_cached_archive(&fx, "rollback"); + unwind_names_the_patched_cached_archive(&fx, "remove"); +} diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/gem.rs b/crates/socket-patch-core/src/patch/redirect/upstream/gem.rs index c47227d7e..565f63450 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/gem.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/gem.rs @@ -619,6 +619,7 @@ pub(crate) async fn restore( let mut lock: Option = lock_raw.as_deref().map(|t| to_lf(t).into_owned()); let mut manifest = manifest_raw.clone(); let globals = manifest.as_deref().map(global_sources).unwrap_or_default(); + let handled_before = result.handled.clone(); for pin in pins { if result.refused.contains_key(&pin.uuid) { continue; @@ -658,6 +659,20 @@ pub(crate) async fn restore( Err(why) => result.refuse(&pin.uuid, why), } } + // The pins this pair restored, each judged once against its own + // restored lock. + for pin in pins { + if !result.handled.contains(&pin.uuid) + || handled_before.contains(&pin.uuid) + || result.refused.contains_key(&pin.uuid) + { + continue; + } + if let Some(warning) = stale_cache_warning(view.root(), pin, lock.as_deref(), ctx).await + { + result.warnings.push(warning); + } + } if let (Some(next), Some(endings)) = (lock, endings) { let next = endings.restore(&next).into_owned(); if lock_raw.as_deref() != Some(next.as_str()) { @@ -673,6 +688,82 @@ pub(crate) async fn restore( result } +/// The warning code for a patched archive left in Bundler's cache dir by +/// an unwind (the restore-side counterpart of scan's +/// `redirect_gem_stale_install`). +pub(crate) const STALE_CACHE: &str = "upstream_gem_stale_cache"; + +/// The `CHECKSUMS` sha256 the lock pins for `name (version)`, if any. +fn lock_checksum(lock: &str, name: &str, version: &str) -> Option { + let lines: Vec<&str> = lock.split('\n').collect(); + let (s, e) = named_section(&lines, "CHECKSUMS")?; + lines[s..e].iter().find_map(|line| { + let entry = indented(line, 2)?.trim_end(); + let (n, token, tail) = split_checksum_entry(entry)?; + if n != name || token != version { + return None; + } + tail.split_whitespace() + .find_map(|tok| tok.strip_prefix("sha256=")) + .map(str::to_ascii_lowercase) + }) +} + +/// A restored gem whose `-.gem` still sits in Bundler's +/// cache dir (`cache_path`, default `vendor/cache`) and is not proven to +/// be the upstream archive: bundler installs from that dir before +/// fetching, so the archive a `bundle cache` took while the hosted pin was +/// live keeps installing the patched bytes (bundler < 2.6, frozen) or +/// fails every install against the restored upstream checksum (#1260). +/// The upstream sha is the restored lock's `CHECKSUMS` entry, else the +/// rubygems.org compact index (already fetched when the restore re-pinned +/// `CHECKSUMS`). Read-only, like scan's guard: the file is named, never +/// deleted. +async fn stale_cache_warning( + root: &std::path::Path, + pin: &HostedPin, + lock: Option<&str>, + ctx: &Ctx<'_>, +) -> Option<(&'static str, String)> { + let (name, version) = pin.name_version()?; + let cache_dir = crate::crawlers::ruby_crawler::bundler_app_cache_dir(root).await; + let archive = cache_dir.join(format!("{name}-{version}.gem")); + if !archive.is_file() { + return None; + } + let got = crate::vendor::file_sha256_hex(&archive).await; + let upstream = match lock.and_then(|l| lock_checksum(l, &name, &version)) { + Some(sha) => Ok(sha), + None => ctx.client.rubygems_sha256(&name, &version).await, + }; + let fix = "delete it, then run `bundle cache` to cache the upstream gem in its place \ + (or `bundle install` if the project does not commit its cache)"; + let detail = match (got, upstream) { + (Some(got), Ok(want)) if got.eq_ignore_ascii_case(&want) => return None, + (Some(_), Ok(_)) => format!( + "{purl} is back on its upstream registry entry, but Bundler's cache dir still \ + holds a non-upstream (patched) archive at {path}; bundler installs from that \ + dir first, so installs fail on the upstream checksum or keep the patched \ + bytes: {fix}", + purl = pin.purl, + path = archive.display(), + ), + (_, why) => format!( + "{purl} is back on its upstream registry entry, but Bundler's cache dir holds \ + {path}, which could not be checked against the upstream sha256{why}; if it \ + was cached while the hosted patch was live, bundler keeps installing it: \ + {fix}", + purl = pin.purl, + path = archive.display(), + why = match why { + Err(e) => format!(" ({e})"), + Ok(_) => " (the archive is unreadable)".to_string(), + }, + ), + }; + Some((STALE_CACHE, detail)) +} + /// Restore one gem in a lock + manifest pair: `Ok(None)` when neither wires /// it, else the next `(lock, manifest)` texts. async fn restore_one( @@ -1142,6 +1233,282 @@ mod tests { assert_eq!(next_lock.as_deref(), Some(lock.as_str())); } + /// A hosted Gemfile + lock pair for `rails 7.0.0` (a direct gem), from + /// the real forward rewrite: `checksums` picks a converged + /// (bundler 2.6+) or a pre-CHECKSUMS (2.5) lock. + fn hosted_pair(checksums: bool, upstream_sha: &str) -> BTreeMap { + let manifest = "source \"https://rubygems.org\"\n\ngem \"rails\", \"7.0.0\"\n"; + let lock = format!( + "GEM\n remote: https://rubygems.org/\n specs:\n rails (7.0.0)\n\nPLATFORMS\n \ + ruby\n\nDEPENDENCIES\n rails (= 7.0.0)\n\nCHECKSUMS\n rails (7.0.0) \ + sha256={upstream_sha}\n\nBUNDLED WITH\n 2.6.9\n" + ); + let files = BTreeMap::from([ + ("Gemfile".to_string(), manifest.to_string()), + ("Gemfile.lock".to_string(), lock), + ]); + let r = crate::patch::redirect::rewrite_registry_redirect( + &files, + &[crate::patch::redirect::DepOverride { + ecosystem: "gem".into(), + name: "rails".into(), + namespace: None, + version: "7.0.0".into(), + token: "tok".into(), + patch_uuid: UUID.into(), + artifact_url: "https://patch.test/rails-7.0.0.gem".into(), + registry_override: Some(crate::patch::redirect::RegistryOverride { + kind: "rubygems-compact-index".into(), + index_url: IDX.into(), + identifiers: crate::patch::redirect::RegistryOverrideIdentifiers { + name: "rails".into(), + version: "7.0.0".into(), + gem_checksum_sha256: Some("f".repeat(64)), + ..Default::default() + }, + }), + integrity: Default::default(), + }], + ); + let mut out = r.files; + let lock = out.get_mut("Gemfile.lock").expect("the lock is redirected"); + assert!(lock.contains(IDX), "{lock}"); + if !checksums { + // Bundler 2.5 converges the same hosted lock with no CHECKSUMS. + let start = lock.find("CHECKSUMS\n").expect("a converged lock"); + let end = start + lock[start..].find("\n\n").expect("a section end") + 2; + lock.replace_range(start..end, ""); + assert!(!lock.contains("sha256="), "{lock}"); + } + out + } + + fn sha_hex(bytes: &[u8]) -> String { + use sha2::{Digest, Sha256}; + hex::encode(Sha256::digest(bytes)) + } + + /// Restore the hosted pair from `hosted_pair` in a temp project whose + /// `cache_rel` dir holds `rails-7.0.0.gem` (unless `archive` is None), + /// returning the restore's warnings. + async fn restore_with_cache( + checksums: bool, + client: &super::super::UpstreamClient, + upstream_sha: &str, + bundle_config: Option<&str>, + cache_rel: &str, + archive: Option<&[u8]>, + ) -> (tempfile::TempDir, Vec<(&'static str, String)>) { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + for (rel, text) in hosted_pair(checksums, upstream_sha) { + std::fs::write(root.join(rel), text).unwrap(); + } + if let Some(config) = bundle_config { + std::fs::create_dir_all(root.join(".bundle")).unwrap(); + std::fs::write(root.join(".bundle/config"), config).unwrap(); + } + if let Some(bytes) = archive { + std::fs::create_dir_all(root.join(cache_rel)).unwrap(); + std::fs::write(root.join(cache_rel).join("rails-7.0.0.gem"), bytes).unwrap(); + } + let pin = HostedPin { + purl: "pkg:gem/rails@7.0.0".into(), + uuid: UUID.into(), + files: vec!["Gemfile".into(), "Gemfile.lock".into()], + }; + let ctx = ctx_with(client); + let mut view = View::new(root); + let r = restore(&mut view, &[&pin], &pin.files, &ctx).await; + assert!(r.refused.is_empty(), "{:?}", r.refused); + assert!(r.handled.contains(UUID)); + (tmp, r.warnings) + } + + fn stale_cache<'w>(warnings: &'w [(&'static str, String)]) -> Vec<&'w str> { + warnings + .iter() + .filter(|(code, _)| *code == STALE_CACHE) + .map(|(_, d)| d.as_str()) + .collect() + } + + /// #1260: `bundle cache` while the hosted pin was live left the + /// PATCHED archive in vendor/cache. The restored CHECKSUMS pins the + /// upstream sha, so every later install fails (exit 37) until that + /// file goes: the unwind must name it. + #[tokio::test] + async fn restore_warns_about_a_patched_archive_in_vendor_cache() { + let upstream_sha = sha_hex(b"upstream gem"); + let client = super::super::UpstreamClient::new(true); + client + .seed_rubygems_sha256("rails", "7.0.0", &upstream_sha) + .await; + let (tmp, warnings) = restore_with_cache( + true, + &client, + &upstream_sha, + None, + "vendor/cache", + Some(b"patched gem"), + ) + .await; + let hits = stale_cache(&warnings); + assert_eq!(hits.len(), 1, "{warnings:?}"); + let path = tmp + .path() + .join("vendor") + .join("cache") + .join("rails-7.0.0.gem"); + assert!(hits[0].contains(&path.display().to_string()), "{}", hits[0]); + assert!(hits[0].contains("pkg:gem/rails@7.0.0"), "{}", hits[0]); + assert!(hits[0].contains("non-upstream"), "{}", hits[0]); + } + + /// #1260: an upstream archive in the cache (cached before the hosted + /// scan, or re-cached after) is healthy: no warning. + #[tokio::test] + async fn restore_keeps_quiet_about_an_upstream_archive_in_vendor_cache() { + let upstream_sha = sha_hex(b"upstream gem"); + let client = super::super::UpstreamClient::new(true); + client + .seed_rubygems_sha256("rails", "7.0.0", &upstream_sha) + .await; + let (_tmp, warnings) = restore_with_cache( + true, + &client, + &upstream_sha, + None, + "vendor/cache", + Some(b"upstream gem"), + ) + .await; + assert!(stale_cache(&warnings).is_empty(), "{warnings:?}"); + // No archive at all: nothing to say either. + let (_tmp, warnings) = + restore_with_cache(true, &client, &upstream_sha, None, "vendor/cache", None).await; + assert!(stale_cache(&warnings).is_empty(), "{warnings:?}"); + } + + /// #1260, `cache_path gems/cache`: the dir bundler reads is the + /// configured one, so that's the archive the warning names. + #[tokio::test] + async fn restore_follows_the_configured_bundle_cache_path() { + let upstream_sha = sha_hex(b"upstream gem"); + let client = super::super::UpstreamClient::new(true); + client + .seed_rubygems_sha256("rails", "7.0.0", &upstream_sha) + .await; + let (tmp, warnings) = restore_with_cache( + true, + &client, + &upstream_sha, + Some("---\nBUNDLE_CACHE_PATH: \"gems/cache\"\n"), + "gems/cache", + Some(b"patched gem"), + ) + .await; + let hits = stale_cache(&warnings); + assert_eq!(hits.len(), 1, "{warnings:?}"); + let path = tmp + .path() + .join("gems") + .join("cache") + .join("rails-7.0.0.gem"); + assert!(hits[0].contains(&path.display().to_string()), "{}", hits[0]); + } + + /// #1260 on bundler 2.5 (no CHECKSUMS): a frozen install takes the + /// cached patched archive without complaint, so the rollback silently + /// doesn't take effect. The upstream sha comes from rubygems.org; when + /// it can't be fetched (`--offline`) the warning still names the file. + #[tokio::test] + async fn restore_warns_about_a_cached_archive_without_checksums() { + let upstream_sha = sha_hex(b"upstream gem"); + let client = super::super::UpstreamClient::new(true); + client + .seed_rubygems_sha256("rails", "7.0.0", &upstream_sha) + .await; + let (_tmp, warnings) = restore_with_cache( + false, + &client, + &upstream_sha, + None, + "vendor/cache", + Some(b"patched gem"), + ) + .await; + let hits = stale_cache(&warnings); + assert_eq!(hits.len(), 1, "{warnings:?}"); + assert!(hits[0].contains("non-upstream"), "{}", hits[0]); + + let (_tmp, warnings) = restore_with_cache( + false, + &client, + &upstream_sha, + None, + "vendor/cache", + Some(b"upstream gem"), + ) + .await; + assert!(stale_cache(&warnings).is_empty(), "{warnings:?}"); + + let offline = super::super::UpstreamClient::new(true); + let (tmp, warnings) = restore_with_cache( + false, + &offline, + &upstream_sha, + None, + "vendor/cache", + Some(b"patched gem"), + ) + .await; + let hits = stale_cache(&warnings); + assert_eq!(hits.len(), 1, "{warnings:?}"); + let path = tmp + .path() + .join("vendor") + .join("cache") + .join("rails-7.0.0.gem"); + assert!(hits[0].contains(&path.display().to_string()), "{}", hits[0]); + assert!(hits[0].contains("could not be checked"), "{}", hits[0]); + } + + /// A gem restored in both a `Gemfile` and a `gems.rb` pair shares one + /// cache dir: one warning, not one per pair. + #[tokio::test] + async fn restore_warns_once_for_both_lock_spellings() { + let upstream_sha = sha_hex(b"upstream gem"); + let client = super::super::UpstreamClient::new(true); + client + .seed_rubygems_sha256("rails", "7.0.0", &upstream_sha) + .await; + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + let pair = hosted_pair(true, &upstream_sha); + std::fs::write(root.join("Gemfile"), &pair["Gemfile"]).unwrap(); + std::fs::write(root.join("Gemfile.lock"), &pair["Gemfile.lock"]).unwrap(); + std::fs::write(root.join("gems.rb"), &pair["Gemfile"]).unwrap(); + std::fs::write(root.join("gems.locked"), &pair["Gemfile.lock"]).unwrap(); + std::fs::create_dir_all(root.join("vendor/cache")).unwrap(); + std::fs::write(root.join("vendor/cache/rails-7.0.0.gem"), b"patched gem").unwrap(); + let pin = HostedPin { + purl: "pkg:gem/rails@7.0.0".into(), + uuid: UUID.into(), + files: vec![ + "Gemfile".into(), + "Gemfile.lock".into(), + "gems.locked".into(), + "gems.rb".into(), + ], + }; + let ctx = ctx_with(&client); + let mut view = View::new(root); + let r = restore(&mut view, &[&pin], &pin.files, &ctx).await; + assert!(r.refused.is_empty(), "{:?}", r.refused); + assert_eq!(stale_cache(&r.warnings).len(), 1, "{:?}", r.warnings); + } + #[test] fn global_sources_skip_blocks() { let m = format!("source 'https://rubygems.org'\nsource(\"https://b.example\")\nsource \"{IDX}\" do\nend\n");