Skip to content

aprender#5751: keep crates.io and the build-base fetch off the merge path (seeded registry, local build base) - #5754

Open
noahgift wants to merge 1 commit into
mainfrom
p4-cratesio-brownout
Open

noahgift wants to merge 1 commit into
mainfrom
p4-cratesio-brownout

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Fixes #5751. During a WAN brownout, merge jobs failed on crates.io timeouts and on pin_build_base "cannot fetch the build base". This PR removes both network dependencies from the common merge path.

Root cause. The cargo registry mounted into each job is keyed per PR × shard (cargo-ci/registry/<PR_OR_REF>-sN). Every new PR and every merge-queue ref therefore starts with an empty registry and downloads its full dependency set from crates.io before it compiles. Separately, pin_build_base.sh always re-fetched the build base from github.com, even though the base is the merge commit's first parent. The existing P6 retry only retries, which does not survive a brownout lasting minutes.

Changes:

  • New scripts/ci/seed_registry.sh:
    • Seed: an empty registry is filled from a host-local seed at /mnt/nvme-raid0/cargo-ci/registry/_seed. This step never fails the job.
    • Publish: a green shard refreshes the seed atomically. It refuses to publish an empty registry.
  • ci/sections.yml:
    • calls the seed in the create-registry step;
    • adds a publish step that runs on success and only warns on failure;
    • uses fetch-depth: 2 on the two guard-job checkouts.
  • pin_build_base.sh: uses the base commit locally when cat-file finds it. Otherwise it falls back to the unchanged P6 fetch.

Tests. scripts/tests/brownout_test.sh has 6 red results before the fix and 7 ok, 0 red after. pin_build_base.sh --self-test gives 30 ok. fetch_p6.sh --check-wiring passes. bashrs lint reports 0 errors.

Caveats:

  • The seed is empty until the first green shard publishes it, so the first runs after this lands are still cold.
  • Each host has its own seed; it is not shared across the fleet.
  • Reaper: the intel reaper only enumerates numeric and merge-queue registry keys, so it never deletes _seed. paiml/infra PMAT-1880 adds an explicit guard and a log line, so a later, wider enumerator still cannot reach it.

🤖 Generated with Claude Code

…path

Registry was keyed per PR x shard, so every new PR and merge-queue ref cold-fetched all crates
from crates.io; pin_build_base re-fetched a base the checkout could already hold. Seed a cold
registry from a host-local seed (published by green shards); pin reads a local base first and
checkout depth is 2 in the guard jobs. brownout_test.sh is RED without the fix.

Pmat-Ticket: PMAT-5751
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=5754 head=94804c4c577f93ac1c41a8403f7f41c6cf5478dd verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: x86-main security section fails on crates.io timeouts and pin_build_base cannot fetch the build base (run 38363869044)

1 participant