Upstream restore sends every public-registry lookup at once; the registry_concurrency cap has no caller #1220

[agent] Filed by the scheduled architecture audit routine (CLI and core). Register: discussion #560 register.

Kind: bug. Source: new finding; register C78. Related: C38 (#614), the other API-concurrency cap that bypasses utils::concurrent, and E33, the upstream restore.

Problem (main @ f3c6313)

#1039 added a cap for requests to the public package registries, utils/concurrent.rs#L64-L83:

/// In-flight cap for pristine downloads from the PUBLIC package registries
/// (npmjs.org, PyPI, crates.io, RubyGems, the Go proxy, Maven Central). ...
/// the fetcher has no 429/`Retry-After` handling ... 4 keeps the
/// lockfile-only ladder latency-flat without bursting at anyone.
pub const REGISTRY_CONCURRENCY: usize = 4;
pub fn registry_concurrency() -> usize {
    if crate::crawlers::walk_pool::fd_limit_is_tight() { return 1; }
    REGISTRY_CONCURRENCY
}

Nothing calls registry_concurrency() or reads REGISTRY_CONCURRENCY, in production or in tests.

The only code that sends public-registry lookups concurrently is the hosted upstream restore. It runs every lookup at once with futures_util::future::join_all, in five places:

• npm-family and vlt: upstream/npm.rs#L66-L112 (fetch_dists_on);
• PyPI: upstream/pypi.rs#L89-L103;``
• cargo: upstream/cargo.rs#L98;``
• Go: upstream/golang.rs#L68;``
• Composer: upstream/composer.rs#L253.``

restore_upstream runs these for rollback and remove of hosted pins (hosted_unwind.rs#L76), and for the hosted → vendored takeover and eject (vendor.rs#L1754, vendor.rs#L1881). The UpstreamClient holds a plain build_registry_client() with no semaphore.

Proof by execution. I ran this twice on f3c6313, with identical results. A temporary unit test in upstream/npm.rs called fetch_dists_on with 40 pins against a local HTTP server. The server counts open requests and answers each one after 300 ms.

PROBE lookups=40 resolved=40 requests=40 peak_in_flight=40 registry_concurrency()=4 wall_ms=322
PROBE lookups=40 resolved=40 requests=40 peak_in_flight=40 registry_concurrency()=4 wall_ms=327

Every lookup is in flight at once: 40, where the cap says 4, and where a tight RLIMIT_NOFILE should mean 1.

Symptoms

I found no open issue for this. Impact:

• A rollback or remove of N hosted pins opens N connections at once to registry.npmjs.org, PyPI, crates.io, proxy.golang.org or Packagist. The registry client has no 429 handling, so a rate-limited lookup fails and its pin is refused (the restore's per-pin refused path).
• With a tight descriptor limit, the burst can fail with EMFILE. A serial loop would not, and fd_limit_is_tight exists to prevent exactly that.
• This is the second network cap that sits outside utils::concurrent (the first is The public-proxy per-package fallback runs 10 requests in flight, ignoring SOCKET_API_CONCURRENCY and the proxy cap of 4 #614). Any new fan-out site will repeat the pattern until the cap is applied in one place.

Proposed change

• Replace the five join_all(lookups) calls with utils::concurrent::ordered_concurrent(lookups, registry_concurrency(), …), collected in input order. The BTreeMaps that fold the results stay unchanged, so the output stays byte-identical.
• An alternative is a semaphore inside UpstreamClient::get_json (or the shared registry client) sized by registry_concurrency(). That would cover every lookup in the restore, including any future fan-out, at one site. Pick one; don't do both.
• What gets deleted: the five unbounded join_all fan-outs. registry_concurrency() gets its first caller. If instead the maintainers decide public registries need no cap, delete REGISTRY_CONCURRENCY and registry_concurrency() as dead code.

Size and scope

• Files: patch/redirect/upstream/{npm,pypi,cargo,golang,composer}.rs (or upstream/client.rs for the semaphore variant). About 30 changed production lines.
• Out of scope: 429/Retry-After handling for the registry client (C15, Tracking: one retry primitive for the patch API client (JSON, vendor service, blob and diff) #676), the public-proxy cap (The public-proxy per-package fallback runs 10 requests in flight, ignoring SOCKET_API_CONCURRENCY and the proxy cap of 4 #614), and the restore's design (E33).

Acceptance criteria

• Add a regression test: a local server counts peak in-flight requests, and fetch_dists_on with 40 pins never exceeds registry_concurrency(). Cover at least one non-npm format the same way, or test the client-level semaphore once.
• Under a forced tight fd limit (the existing fd_limit_is_tight test hook, if there is one), the peak is 1.
• Restore output is unchanged: the existing upstream unit tests and hosted_unwind/takeover e2e suites stay green.
• cargo clippy shows no dead-code allowance for registry_concurrency.

Dependencies

None. It doesn't block anything.