Skip to content

🤖 ci: retry bun teardown crashes and npm registry publication skew - #3856

Merged
ibetitsmike merged 2 commits into
mainfrom
mike/ci-flake-retries
Aug 14, 2026
Merged

ibetitsmike merged 2 commits into
mainfrom
mike/ci-flake-retries

Conversation

@ibetitsmike

Copy link
Copy Markdown
Contributor

Summary

Fixes the two CI flake classes that dequeued #3854 from the merge queue twice on 2026-08-14 (runs 31829008751 and 31831156227).

Background

  1. Bun teardown crash: in Test / Unit, bun 1.3.5 crashed (Illegal instruction, exit 132) at process teardown in an isolated WorkflowRunner.test.ts run after every test passed ("0 fail", panic during shutdown). The same signature dequeued another PR on Aug 13.
  2. Registry publication skew: during an AWS SDK publish wave, the lockfile-free resolves in Static Checks (check-bench-agent.sh) and Smoke / Server (npm install of the packed tarball) requested @aws-sdk/credential-provider-ini@^3.973.14 seconds before it was visible on the registry (ETARGET / "No version matching ... (but package exists)").

Implementation

  • pr.yml: isolated unit-test files retry up to 3 times, but only on signal exits (>= 128). Genuine test failures (exit 1) still fail immediately on the first attempt, so no flaky-test masking is possible. The existing scripts/retry.sh was not reused here because it retries every nonzero exit.
  • check-bench-agent.sh, smoke-test.sh, generate-npm-shrinkwrap.sh: the three registry-facing installs (all resolve fresh from npm) are wrapped in the existing scripts/retry.sh with 5 attempts x 60s, enough to ride out a multi-minute publish wave.

Considered and rejected: scoping the smoke-test retry to SKIP_SHRINKWRAP=1 only. The observed incident hit the shrinkwrapped Smoke / Server install, so that scoping would leave the observed flake unfixed. Also considered a 3x30 retry budget to match dist-mac's; kept 5x60 because publish waves span minutes and the extra delay only applies to genuinely broken resolution, which lockfile-based jobs never hit.

Validation

  • Behavioral test of the crash-retry loop with a fake bun (crash-then-pass recovers; exit 1 fails immediately with a single invocation; persistent crash exits 132 after 3 attempts; clean pass runs once).
  • make static-check, shellcheck on all three scripts, actionlint, zizmor: all green.
  • check-bench-agent.sh and generate-npm-shrinkwrap.sh run end-to-end locally through the retry wrapper.

Risks

Low. The unit-test retry cannot mask real failures (exit-code gated). The install retries add up to 4 minutes before surfacing a genuine resolution failure in Static Checks / Smoke / npm publish; publication skew is far more common than genuine breakage on those lockfile-free paths.


Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh

@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 1aa03a8a35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike added this pull request to the merge queue Aug 14, 2026
Merged via the queue into main with commit bfb1a38 Aug 14, 2026
22 checks passed
@ibetitsmike
ibetitsmike deleted the mike/ci-flake-retries branch August 14, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant