feat(stack): fall back to the S3 and ECR Public slim mirrors by avallete · Pull Request #6716 · supabase/cli

avallete · 2026-09-23T09:07:15Z

Summary

Slim native artifacts and images now fall back to mirrors. The primary source is always tried first. When every candidate fails, the user still sees the primary's original error. A fallback failure, including a checksum mismatch, is logged as a warning, and a fallback that succeeds is logged.

Natives, downloads: archives and manifests are tried from SLIM_NATIVE_GITHUB_RELEASES first, then SLIM_NATIVE_SUPABASE_S3_MIRROR (the public supabase-cli-artifacts bucket).
Natives, checksums: checksum sources are listed separately from download mirrors. The checksum comes from the GitHub Release SHA256SUMS, or else from the archive layer digest of ghcr.io/supabase/cli/<service>:<version>-native-<target>, read with an anonymous pull token. The bucket is never a checksum source, and an archive from any host is only accepted if it hashes to that checksum.
Images: the container runtime pulls from SLIM_IMAGE_GHCR_REGISTRY first. If that fails, it pulls the same tag and digest from SLIM_IMAGE_SUPABASE_ECR_MIRROR, and launches then use the reference that was pulled.
More mirrors later: checksum sources, native mirrors and image mirrors (slimImageMirrors) are all ordered lists, so adding another host is one more entry.
Mirror workflow: upload-natives-s3 in mirror-slim-image.yml no longer waits for the ECR mirror job, so an ECR Public failure no longer keeps natives out of S3. The job now checks the source image digest itself before uploading.
Native Postgres as root: startup fails with PostgreSQL cannot be run as root before any artifact download or process launch. Docker and Podman are unchanged.
Fresh init config: an unset env(NAME) literal is ignored only for experimental.s3_*. Other unresolved values, including auth keys, still fail the experimental stack.
A service that exits before it is ready keeps that service's error, including a native Postgres stderr tail.

Catalog pins that have no public mirror still fail on the primary download. Mirroring those pins is a separate release step.

ADR 0026 is updated: the earlier decision against a public S3 bucket is superseded for native archives. S3 is not a checksum authority.

Linked issue

None; maintainer change. The sender side is supabase/slim-services#318.

🤖 Generated with Claude Code

https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

`upload-natives-s3` needed the `mirror` job to succeed, so any ECR Public failure (missing `cli/*` repositories or push permissions) also kept every native out of the S3 bucket. The job now runs alongside the ECR copy and verifies the source image digest itself, so nothing reaches the bucket from a dispatch whose claimed image digest does not match GHCR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

Hosts that block GitHub release downloads, such as agent sandboxes that only allow `*.amazonaws.com`, could not prepare native service artifacts. Each artifact now lists its hosts in order, the GitHub Release first and the public S3 bucket second. The checksum comes from the first host that serves one, and the archive from the first host whose copy hashes to that checksum, so falling back never accepts different bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

…lure The container runtime now pulls a catalog image from its ECR Public mirror when the GHCR pull fails, and launches the mirror reference it pulled. Both this and the S3 fallback for natives now report the primary's original error when the mirror fails too, so behavior only changes when a mirror rescues a failed download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

Mirror constants now say which host they point at (`SLIM_NATIVE_SUPABASE_S3_MIRROR`, `SLIM_IMAGE_SUPABASE_ECR_MIRROR`, and the GHCR / GitHub Releases primaries), and images take an ordered mirror list like natives do, so another host can be added next to the existing ones without renaming or reshaping anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

The S3 fallback also served the checksum for its own archive, making the bucket a second trust root. Checksum authorities are now listed apart from download mirrors: the GitHub Release SHA256SUMS first, then the archive layer digest of the native OCI artifact on GHCR, read through an anonymous pull token. The S3 bucket only serves the archive and manifest, which must match that checksum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

github-actions

Superseded by a newer AI review

🤖 AI Review

Claude reported six findings; Codex’s independent review completed with zero findings. Five Claude findings are confirmed (two with factual narrowing) and one is refuted. No critical or major defects were verified; the substantive concerns are stale native fallback selection, repeated container pulls, ADR inconsistency, observability, and missing helper coverage.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	`docs/adr/0026-slim-artifact-mirrors.md:25`	`documentation`	claude	ADR 0026 contradicts itself by retaining decisions that reject public S3 and client fail-through while its updated follow-up and consequences sections describe both as shipped.
🟡 MINOR	`packages/stack/src/preparation/SlimServicesSource.ts:227`	`correctness`	claude	Checksum authority and archive host are selected independently, so a fallback GHCR digest can validate and persist an older S3 archive during a same-version overwrite even when GitHub already serves newer bytes.
🟡 MINOR	`packages/stack/src/runtime/Container.ts:173`	`performance`	claude	After a successful mirror rescue, every later prepare for the same image retries the unavailable primary instead of reusing the recorded local mirror, potentially repeating the five-minute pull timeout.
⚪ NIT	`packages/stack/src/preparation/SlimServicesSource.ts:244`	`observability`	claude	A successful fallback leaves no normal-run indication that the primary failed or which alternate source was used, and a manifest failure is described using the mirror’s download URL rather than its manifest URL.
⚪ NIT	`packages/stack/src/Artifacts.ts:175`	`test-coverage`	claude	The production slimImageMirrors mapping is untested; container mirror tests inject stubs and do not exercise the GHCR-to-ECR rewrite or the empty result for unrelated images.

Refuted findings (kept for transparency, not posted as review comments)

packages/stack/src/Artifacts.ts:199 (security): The S3 mirror independently supplies both native bytes and their validating checksum, making it a new self-certifying trust anchor.
Refuted: The checked-out code provides concrete counter-evidence: S3 is only an archive/manifest mirror, and its archive must match a digest obtained independently from GitHub or GHCR.

Stats

Claude findings: 6 · Codex findings: 0 · Confirmed: 5 · Refuted: 1 · Uncertain: 0

Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

…supabase#6717) ## Summary In `mirror-slim-image.yml`, `upload-natives-s3` needed the `mirror` job to succeed. The ECR Public `cli/*` repositories don't exist yet, so every dispatch fails at "Ensure ECR Public repository exists" and no native ever reaches the S3 bucket. The S3 job now runs alongside the ECR copy. It checks the source image digest on GHCR itself, so nothing reaches the bucket from a dispatch whose claimed image digest doesn't match. The ECR job is unchanged. This is split out of supabase#6716 so the S3 mirror can start filling before the CLI fallback lands. Once this merges, supabase#6716 carries the same change with no remaining diff. ## Linked issue None; maintainer change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p --- _Generated by [Claude Code](https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p)_ Co-authored-by: Claude <noreply@anthropic.com>

Native Postgres cannot initialize as root, so startup fails before any artifact download or process launch. A failed mirror fallback, including a checksum mismatch, is logged while the returned error stays the primary's. Unresolved env() placeholders are ignored only for the experimental S3 fields.

…esilience-s3-fallback-8nm0qz Resolve the container prepare conflict by using develop's `image inspect` presence check for both the primary image and its mirrors, keep develop's Docker database storage alongside the native root check, and take develop's container-free DatabaseSnapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

avallete · 2026-09-23T14:55:08Z

/ai-review

github-actions

🤖 AI Review

Both independent reviews were available. After checking the cited code and trusted ADR, I merged two overlapping pairs into 11 findings. The findings hold, with two qualifications: six catalog images lack digest pins, not seven; and native process-group cleanup makes the PostgreSQL exit hang conditional on a process retaining stderr outside that cleanup. No additional finding emerged from the independent pass.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	`packages/stack/src/Artifacts.ts:176`	`integrity`	claude	The ECR fallback is not content-pinned for six catalog images. For those images, rewriting the registry prefix pulls a mutable tag without verifying that it matches the GHCR image.
🟡 MINOR	`apps/cli/src/command-internal/stack-config.ts:721`	`correctness`	claude	A lowercase unresolved env placeholder can be rejected as an unsupported experimental S3 setting even though the config loader accepts that placeholder form.
🟡 MINOR	`apps/cli/src/commands/experimental/stack/stack-config.integration.test.ts:175`	`test-quality`	claude	The S3 placeholder test depends on four ambient S3 environment variables being unset or empty; a populated variable resolves its placeholder and can make the test fail.
🟡 MINOR	`packages/stack/src/services/Database.ts:158`	`lifecycle`	claude+codex	Native PostgreSQL's automatic exit observation can wait indefinitely after the launcher exits if a surviving process keeps its stderr pipe open.
🟡 MINOR	`packages/stack/src/preparation/SlimServicesSource.ts:296`	`compatibility`	claude	The OCI checksum reader rejects native artifact forms that the mirror pipeline accepts, potentially disabling the S3 fallback when GitHub checksums are unavailable.
🟡 MINOR	`packages/stack/src/Service.ts:557`	`correctness`	claude	A readiness waiter can receive another launch's error because diedBeforeReady reads the current observation error without checking which launch produced it.
🟡 MINOR	`packages/stack/src/services/Database.ts:353`	`diagnostics`	claude+codex	The stderr tail can replace a non-ASCII character with replacement characters when its UTF-8 bytes span stream chunks.
⚪ NIT	`apps/cli/src/commands/experimental/stack/start/start.handler.ts:212`	`ux`	claude	The new root-user failure gives no actionable suggestion for starting with Docker or as a non-root user.
⚪ NIT	`packages/stack/src/services/Database.ts:129`	`type-safety`	claude	The root guard accepts any string as a runtime, allowing a mistyped runtime literal to silently bypass the check.
⚪ NIT	`apps/cli/src/commands/experimental/stack/start/start.handler.ts:211`	`testability`	claude	The handler's root guard reads the global process uid, so tests cannot control it through the RuntimeInfo service already injected into the handler.
⚪ NIT	`docs/adr/0026-slim-artifact-mirrors.md:20`	`documentation`	claude	The updated ADR describes shipped mirror behavior as superseded and still lists client consumption as follow-up work.

Stats

Claude findings: 11 · Codex findings: 2 · Confirmed: 11 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5 + gpt-6-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

github-actions · 2026-09-23T15:27:34Z

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@882b5f9ba4f4d80265bdba543c18090ce4c98d8f

Preview package for commit 882b5f9.

A held stderr pipe no longer stalls PostgreSQL shutdown, and a readiness waiter only reports the error from its own launch. Unresolved env() placeholders use the config loader's matcher.

The S3 placeholder test blanked its env vars by writing `process.env` directly, which the Effect lint rejects in CI. `vi.stubEnv` with an `afterEach` `vi.unstubAllEnvs()` keeps the same isolation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

…nm0qz

jgoux

I found a repeat-start failure after mirror recovery and a smaller native stderr timing issue. Details are inline.

After a mirror rescue, a later prepare that found or pulled the GHCR image kept the remembered ECR reference, so a pruned mirror image failed the launch. Prepare now forgets the mirror whenever the primary is ready. The native PostgreSQL stderr-drain guard ran alongside the exit wait, so it had already expired for any process that lived past one second. It now starts after the exit code resolves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

avallete · 2026-09-23T16:23:29Z

Run CLI end-to-end tests (shard 1/3) failed on 86a72a0 in serve-main-offline.e2e.test.ts. That test belongs to functions serve, not to this PR's changes.

The edge-runtime container (supabase-serve-kong-e2e-…-runtime, edge-runtime 1.76.2) exited with code 135 (SIGBUS) while starting its workers, so Kong returned 502.
The test drives Docker directly with the edge-runtime and Kong images and doesn't import packages/stack. The only changes in 86a72a0 are in packages/stack/src/runtime/Container.ts and services/Database.ts.
The same test passed on this PR's previous two commits, ad7bf02 and 58de213.

No fix exists in this PR, since it doesn't touch that path. I'll re-run the failed job once when the run finishes.

Generated by Claude Code

…nm0qz

avallete · 2026-09-23T17:35:56Z

Run stack end-to-end tests (docker) failed on 882b5f9, the develop merge. This PR didn't cause it. In "default database eager lifecycle and reopen", the database storage helper pull (public.ecr.aws/docker/library/debian:bookworm-slim, in DockerDatabaseStorage.ts, which this PR doesn't change) got toomanyrequests: Rate exceeded from ECR Public. That image has no mirror, so it takes the same path as on develop. The other three docker scenarios passed on the same run. No fix exists to port. I'll re-run the failed job once when the run finishes.

Generated by Claude Code

avallete · 2026-09-23T17:48:47Z

The merge queue dropped this PR twice because Run integration tests failed on the queue commits 800fba7 and f895443. This PR didn't cause it. Both times the service tests failed on their first GHCR pull with toomanyrequests: retry-after: …, allowed: 44000/minute, which is GHCR's shared rate limit. This PR sends the same primary pull as develop. After a failed pull it only tries the ECR Public mirror, which doesn't exist yet, and then reports the original GHCR error.

The limit is hitting the whole repo right now. The unrelated #6764, running at the same time, failed its integration, CLI e2e and docker stack e2e jobs with the same GHCR toomanyrequests errors. The same integration job passed on this PR's head 882b5f9, which has the same tree. No fix exists to port. I re-queued once already, so I'm not re-queuing again until GHCR stops throttling.

Generated by Claude Code

…ase#6776) ## Summary `supabase stack start --runtime native --eager` failed with `functions: ServiceStaleLaunch` and `studio: Prerequisite <functions> is not healthy` whenever a parent directory of the stack's data root, such as `$HOME`, contained a `package.json`. The error looked like a launch-id race, but functions launched exactly once. Edge Runtime crashed about a second after starting, during readiness. ### What went wrong ```mermaid flowchart TD A["stack start --runtime native"] --> B["edge-runtime --main-service= ~/.supabase/stacks/<id>/…/runtime/functions"] B --> C{"Deno config discovery walks up parent directories"} C -->|"no deno.json / package.json"| C C -->|"finds ~/package.json"| D["$HOME adopted as workspace root"] D --> E["scans $HOME ~/.Trash → Operation not permitted"] E --> F["💥 edge-runtime exits with 1 while readiness is pending"] F --> G["functions never becomes healthy"] G --> H["studio: Prerequisite functions is not healthy"] classDef bad fill:#fde2e2,stroke:#c0392b,color:#000 class E,F,G,H bad ``` There were two problems stacked on top of each other: | | Problem | Where it is fixed | |---|---|---| | **1. Cause** | Deno adopts an ancestor config as its workspace and crashes on unreadable entries | **This PR:** make the bootstrap directory its own empty Deno workspace root | | **2. Symptom** | A crash during readiness was reported as `ServiceStaleLaunch`, which hid `Process exited with 1` | Already on `develop` via supabase#6716 (`launchError` / `diedBeforeReady` in `Service.ts`) | ### The fix: stop config discovery at the bootstrap The bootstrap directory now contains `deno.json` = `{"workspace":[]}`. Deno treats it as a workspace root, so discovery stops there before it reaches any ancestor `package.json` or Deno workspace. ```mermaid flowchart LR subgraph Before B1["runtime/functions/ index.ts"] -->|walk up| B2["…"] -->|walk up| B3["~/package.json 💥"] end subgraph After A1["runtime/functions/ index.ts deno.json {workspace: []} ✅"] -.->|discovery stops here| A1 end ``` The bootstrap directory is owned by the stack and removed with its data. Docker already mounts it without ancestors at `/__supabase_bootstrap`, so native now behaves the same way. User function workers start from their own entrypoints in the project and resolve their config as before, including a shared `supabase/functions/deno.json` or a function's own `package.json`. ### Why the readiness change alone is not enough ```mermaid flowchart LR subgraph "develop today" D1["edge-runtime crashes"] --> D2["ready(): Process exited with 1 ✅ honest"] --> D3["stack start fails ❌"] end subgraph "with this PR" P1["edge-runtime boots"] --> P2["ready(): healthy"] --> P3["stack start succeeds ✅"] end ``` ### Alternatives considered and abandoned `edge-runtime start` has no `--config` or `--no-config` flag, so the parent-directory walk can only be influenced by a marker file or an environment variable. | Approach | Parent `package.json` + unreadable dir | Parent Deno workspace | User functions | Outcome | |---|---|---|---|---| | `DENO_NO_PACKAGE_JSON=1` on the edge-runtime process | ✅ | ✅ | ❌ Every user worker inherits the variable, so a function importing a dependency through its own `package.json` fails with `Relative import path "…" not prefixed with / or ./ or ../` | Abandoned | | `deno.json` = `{}` | ✅ | ❌ `Config file must be a member of the workspace.` Deno keeps walking to find a workspace root | ✅ | Abandoned; this was the first version of this PR | | `package.json` marker | ✅ | ❌ same membership error | ✅ | Abandoned | | **`deno.json` = `{"workspace":[]}`** | ✅ | ✅ | ✅ | **Chosen** | Other options were rejected without implementation: - **Moving the bootstrap to a directory without config ancestors:** no location can guarantee that, and the stack data root normally lives under `$HOME`. - **Shipping the bootstrap as an eszip:** a larger build change for the same outcome. ### Tests - A native Functions integration test boots the bootstrap below two ancestor setups: a `package.json` with an unreadable sibling directory, and a Deno workspace that includes the project's functions. In both cases a user function still resolves imports from a shared `supabase/functions/deno.json`. - Without a marker, the `package.json` case fails with `Process exited with 1`. - With a plain `{}` marker, the workspace case fails with the membership error. - The Service tests lock the readiness contract: a crash during health returns the crash `ServiceError`, and a requested stop returns `ServiceStaleLaunch`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

claude added 3 commits September 23, 2026 08:41

avallete requested a review from a team as a code owner September 23, 2026 09:07

avallete mentioned this pull request Sep 23, 2026

Mirror native artifacts to S3 and check all destinations independently supabase/slim-services#318

Merged

depthfirst-app Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread packages/stack/src/Artifacts.ts Outdated

github-actions Bot reviewed Sep 23, 2026

View reviewed changes

avallete mentioned this pull request Sep 23, 2026

ci(repo): upload slim natives to S3 without waiting on the ECR mirror #6717

Merged

avallete and others added 2 commits September 23, 2026 16:26

github-actions Bot reviewed Sep 23, 2026

View reviewed changes

avallete added the run-preview-packages Publish pkg.pr.new preview CLI packages for this PR label Sep 23, 2026

avallete and others added 3 commits September 23, 2026 17:37

fix(stack): bound native exit and keep launch errors together

4b3f12a

A held stderr pipe no longer stalls PostgreSQL shutdown, and a readiness waiter only reports the error from its own launch. Unresolved env() placeholders use the config loader's matcher.

Merge branch 'develop' into claude/mirroring-resilience-s3-fallback-8…

58de213

…nm0qz

jgoux requested changes Sep 23, 2026

View reviewed changes

Comment thread packages/stack/src/runtime/Container.ts Outdated

Comment thread packages/stack/src/services/Database.ts Outdated

avallete requested a review from jgoux September 23, 2026 16:17

jgoux approved these changes Sep 23, 2026

View reviewed changes

Merge branch 'develop' into claude/mirroring-resilience-s3-fallback-8…

882b5f9

…nm0qz

avallete enabled auto-merge September 23, 2026 17:27

avallete added this pull request to the merge queue Sep 23, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026

avallete added this pull request to the merge queue Sep 23, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026

avallete added this pull request to the merge queue Sep 24, 2026

Merged via the queue into develop with commit 1725770 Sep 24, 2026
34 of 36 checks passed

avallete deleted the claude/mirroring-resilience-s3-fallback-8nm0qz branch September 24, 2026 08:38

avallete mentioned this pull request Sep 24, 2026

fix(stack): anchor native Functions bootstrap config discovery #6776

Merged

Conversation

avallete commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Linked issue

Uh oh!

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avallete commented Sep 23, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Supabase CLI preview

Uh oh!

jgoux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

avallete commented Sep 23, 2026

Uh oh!

avallete commented Sep 23, 2026

Uh oh!

Uh oh!

avallete commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

avallete commented Sep 23, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

github-actions Bot commented Sep 23, 2026 •

edited

Loading

avallete commented Sep 23, 2026 •

edited

Loading