Vendored Pipenv re-serializes the whole Pipfile.lock, so a non-ASCII lock is rewritten throughout and its revert is not byte-identical #1128

[agent] Filed by the scheduled architecture audit routine (ecosystems and formats). Register: register comment.

Kind: bug (a duplicated writer whose behavior has drifted, proven by execution). Source: review Part 5.4 ("Pipfile.lock is written two ways"); register E14.

Problem

Pipfile.lock has two writers:

• Hosted mode (and upstream restore) splices single entries by byte span and leaves every other byte alone: redirect/pipenv.rs entries / format_entry.``
• Vendored mode parses the lock into a serde_json::Value, edits it, and writes the whole document back through to_canonical_json.`` It does this on both wire (#L498) and revert (`#L686`).

to_canonical_json claims to be byte-identical to Pipenv's json.dumps(indent=4, sort_keys=True) "for the ASCII content pipenv locks carry". That holds only for ASCII. Pipenv's encoder uses Python's default ensure_ascii=True, so it writes any non-ASCII character as \uXXXX, while serde_json writes raw UTF-8. A Pipenv lock can carry non-ASCII text, for example a local path dependency in a directory named café, or a source name.

The writers have also drifted on a leading BOM. Hosted mode accepts one and keeps it (bom_prefixed_lock_is_rewritten_with_the_bom_intact).`` Vendored mode passes the raw text to serde_json::from_str and refuses it.

The Composer backend fixed this exact class of bug. vendor/composer_lock/lock_text.rs splices one entry and renders it in the lock's own slash and unicode escaping ([`escapes_unicode`](https://github.com/SocketDev/socket-patch/blob/e2d96336fbc449a87a20faae57afbed597e8788f/crates/socket-patch-core/src/vendor/composer_lock/lock_text.rs#L132-L160)),`` because "a whole-lock re-serialization rewrote what serde_json spells differently from the lock's writer". Pipenv never got the same fix.

A layering detail: vendored revert imports its relock test from the hosted engine (crate::patch::redirect::pipenv_reserialized_around_reference),`` which is a vendor → redirect edge.

Proof. A throwaway test in pypi_pipenv.rs's test module, run twice on e2d9633. It used the module's LOCK_DIRECT_REGISTRY fixture with one extra develop entry, "mylib": {"editable": true, "path": "./libs/café"}, written as Pipenv writes it:

So vendor --revert reports a clean revert while leaving a diff in Pipfile.lock, and vendor rewrites lines the user never asked it to touch. The test revert_round_trip_restores_lock_byte_identically passes only because every fixture is ASCII.

Symptoms

None filed. Related: #905 (BOM helper; its pending-files list doesn't name this missing strip) and #937 (the shared Poetry/PDM/Pipenv wire and revert envelope).

Impact: low severity, but it breaks a contract. Pipenv still reads the rewritten lock. The cost is a noisy diff on vendor, and a "byte-identical" revert that isn't one. The vendor --check and takeover drift checks compare parsed values, so they are not misled. Size: about 120 production lines change, and to_canonical_json and with_line_ending are deleted.

Proposed change

1. Move the hosted span reader (entries, properties, format_entry) and reserialized_around_reference from patch/redirect/pipenv.rs into a new formats/pipenv.rs, as a pure move. Hosted, upstream restore and vendored all import it from there, which removes the vendor → redirect edge.
2. Vendored wire and revert splice only the entries they change, through that module, as hosted does. Read the lock through formats::text::strip_bom and re-add the BOM on write.
3. Delete to_canonical_json, with_line_ending and the pipenv_reserialized_around_reference re-export in redirect/mod.rs.
4. Render a spliced entry in the lock's own escaping. Either share Composer's escapes_unicode / escape_non_ascii (moving them to formats::json or similar), or have format_entry emit \uXXXX whenever the surrounding text does.

Size and scope

• Files: vendor/pypi_pipenv.rs, patch/redirect/pipenv.rs, patch/redirect/mod.rs (re-export), a new formats/pipenv.rs, and possibly vendor/composer_lock/lock_text.rs (if the escaping helpers move).
• About 250–400 changed lines. The move (step 1) can land as its own commit.
• Out of scope: the Poetry/PDM/Pipenv envelope (Share the single-lock wire and revert envelope across the Poetry, PDM and Pipenv vendored backends #937), and Pipenv's choice of lock (Vendored mode in a Pipenv project wires only Pipfile.lock and silently leaves a sibling requirements.txt unpatched, and the hosted → vendored takeover reverts that file's hosted pin to plain PyPI #612, Vendored scan of a Pipenv use_pylock = true project wires only pylock.toml, but Pipenv installs from Pipfile.lock, so pipenv sync / install --deploy install the unpatched release after a "success" run #1122).

Acceptance criteria

• One Pipfile.lock entry reader/splicer, in formats/, used by hosted, upstream restore and vendored.
• to_canonical_json and with_line_ending are deleted, and vendor/ no longer imports patch::redirect::pipenv_*.
• Regression test: a lock carrying é in an unrelated entry. Vendored wire changes only the target entry's bytes, and wire + revert is byte-identical.
• Regression test: a BOM-prefixed lock vendors and reverts with the BOM kept, matching hosted.
• wiring_matches_fixtures_byte_identically, revert_round_trip_restores_lock_byte_identically and the redirect::pipenv tests stay green, along with cargo test -p socket-patch-core and the Pipenv CLI suites.

Dependencies

• Blocked by: nothing.
• Coordinate with Share the single-lock wire and revert envelope across the Poetry, PDM and Pipenv vendored backends #937 (shared envelope) and Route inline BOM strips through formats::text (#905) #1117 (BOM routing): whichever lands second rebases.
• Blocks: E23's Pipenv slice.

Backlog review — 2026-10-08

Priority: unassigned → P3. Keep the concrete Pipenv writer/BOM compatibility follow-up. Non-ASCII reserialization changes formatting while parsed values and installs remain correct; a BOM-prefixed file is refused safely. This is low-priority formatting and input compatibility, not P1 patch or attestation correctness.