Repository navigation
feat(stack): pin slim artifacts by revision and make the catalog the single version table - #6883
Merged
jgoux merged 47 commits intoSep 30, 2026
Conversation
4 tasks done
jgoux
force-pushed
the
juliengoux/cli-2512-slim-services-unify-integrity-hotfix-model-for-native
branch
from
September 30, 2026 11:22
3d68cf9 to
1f8af88
Compare
jgoux
marked this pull request as ready for review
September 30, 2026 11:23
Manual sync mode located an additional (release-line-keyed) pin's span by content, but only ever rewrote the pin literal's own upstreamVersion field, leaving the entry's string key behind on the old upstream version. That desyncs resolveArtifact/postgresVersion lookups, which key off the object's literal keys. Track each additional pin's key span and rewrite it alongside the pin when the upstream version moves.
…isions slim-services has published immutable -r0 releases for every service. Replace every placeholder pin in Artifacts.ts with the real content pin (image digest, archive and manifest sha256 per native target) at the newest -r0, using the sync script's manual mode so each pin's S3 staleness check actually ran. Drop PLACEHOLDER_PINS/placeholderPin now that nothing needs them, and turn the "catalog has no placeholder pins" test from it.fails into a normal passing assertion.
Vector 0.58 (up from 0.53) disabled \${VAR} interpolation in config
files by default; the API config's address, sourced from our own
VECTOR_API_ADDRESS env var, was left as a literal unexpanded string,
so vector rejected it with "api.address: invalid socket address
syntax" and exited before it was ready. Pass
--dangerously-allow-env-var-interpolation: the interpolated value is
stack-controlled, not user-supplied config.
Several stack and CLI integration tests hardcoded upstream versions (or, for auth, derived the real catalog's pinned version rather than hardcoding it) that matched the old placeholder catalog. Now that the catalog resolves real -r0 pins at newer upstream versions, update those fixtures so the tests exercise real artifact resolution again instead of failing on "Unsupported <service> artifact version" or a stale saved-stack-binding/readiness-marker mismatch.
dockerImageTag used lastIndexOf(":") directly on the image string, so a
slim ref's @sha256:<digest> suffix made it return the raw digest hex
instead of the tag; a slim ref's own <upstream>-r<N> revision suffix
then also kept a docker.io tag at the bare upstream version from ever
comparing equal to it. Both defeated the "tags match but the image
family doesn't" staleness check. Strip the digest unconditionally and
the -r<N> suffix for a slim ref before comparing.
Confirmed against origin/develop with a throwaway worktree: this
specific pgdelta.seam.integration.test.ts case passes there, so this
branch caused the regression. Root-caused per dockerImageTag's own
digest-suffix bug, plus the catalog's real (non-placeholder) postgres
pin now differing from the Dockerfile's own pinned version, which the
test's fixture catalog mock (mirroring slim-images.unit.test.ts's
pattern) now accounts for.
…terpolation
--dangerously-allow-env-var-interpolation applies process-wide, covering
both --config files vector loads: the recipe's own api config and a
caller-suppliable pipeline config (Config.configPath). The prior fix
only needed it for the API address, so it was broader than justified.
Template the real address into the stack-owned api config directly at
write time (now done in `args`, once the launch's real endpoint is
known, instead of `prepare`, before it is). The default pipeline config
has no `${VAR}` usage, so it needs no rendering. `configPath` is not
reachable from user CLI config today (grepped both apps/cli and the
stack; `stack-config.ts` never sets it), but the stack's own
`tests/whole-stack` fixture does supply one that references
`${LOGFLARE_URL}`/`${LOGFLARE_PRIVATE_ACCESS_TOKEN}`, relying on
Vector's interpolation. Rather than pass the flag for that case (which
would expose the whole process environment, the exact over-broad
behavior this fix removes), render only the recipe's own closed set of
placeholders into a recipe-owned copy of a caller's pipeline before
Vector loads it. The recipe never passes the interpolation flag now,
for either config.
Supersedes 8014870, which framed this as "the address is our own env
var, so opting back in is safe" — that justification covered only the
api config, not the caller-suppliable pipeline config the same flag
also applied to.
…se image
Two issues in ensureLocalPostgresImageCurrent's dockerImageTag:
1. It re-implemented digest stripping (split("@")[0]) and revision
stripping (its own REVISION_SUFFIX), one commit after a139af1
unified the digest-aware split into slim-images.ts's imageTag and
the revision split into services.shared.ts's upstreamVersionFromTag.
Compose those exported helpers instead of a third parser.
2. Revision stripping was unconditional, so a container on
postgres:17.11.0.002-r0 compared equal to a catalog pin on
-r1 — masking exactly the hotfix-revision drift this staleness
check exists to catch. Only strip the revision across families
(docker.io vs slim, where a `-r<N>` has no docker.io equivalent);
within the same family, compare the digest when both refs carry
one, or the full release version otherwise.
Added tests for same-family-different-revision (stale, generic
remediation), same-family-same-revision (current), alongside the
existing docker.io-vs-slim-matching-upstream case (family-mismatch
remediation, unchanged).
…ution prepareNativeArtifact's only remaining call site in this test was replaced by the develop flake fix's nativeRestArtifact helper during the rebase 3 conflict resolution, leaving the import unused.
jgoux
force-pushed
the
juliengoux/cli-2512-slim-services-unify-integrity-hotfix-model-for-native
branch
from
September 30, 2026 11:37
1f8af88 to
130d060
Compare
With SUPABASE_USE_SLIM_IMAGES, the older slim path only uses a slim image when the Dockerfile tag matches a catalog pin; otherwise it keeps the upstream image with the upstream healthcheck. Today that's auth, pgmeta, edge-runtime and pooler — not every service, as the tests assumed. realtime is on Dockerfile v2.135.3 against catalog v2.140.3, so its upstream (Elixir, no wget) container failed the BusyBox wget probe the old test always expected. - The BusyBox wget test now iterates catalogPins() directly (the pinned digest refs) for the services that use the wget healthcheck, independent of whether the Dockerfile currently matches. It no longer depends on the Dockerfile at all. - The start test derives each service's expected image the same way slim mode does (toSlimImage(alias, dockerfileTag) ?? upstream), then asserts the container runs that image and its healthcheck matches that image's family (slim BusyBox wget vs. each service's own upstream argv) — never assuming a fixed family. auth naturally exercises the slim branch today; realtime and storage exercise whichever branch the Dockerfile/catalog alignment puts them in. - Renamed both tests: "latest slim images" no longer describes a path that depends on Dockerfile/catalog alignment.
avallete
requested changes
Sep 30, 2026
avallete
left a comment
Member
There was a problem hiding this comment.
The runtime integrity path looks solid: expected hashes come only from the catalog pin, the manifest is verified before it is parsed, a mirror serving the wrong bytes falls through to the next one, and the per-revision cache key is right. Tag and version parsing handles digest-suffixed refs and Studio-style versions.
One blocker and one real workflow bug, plus some cleanup (inline).
Blocker: slim-images.ts now imports @supabase/stack/internal/artifacts, and .github/scripts/sync-artifacts-catalog.ts imports slim-images.ts. Neither slim-release-published.yml nor sync-artifacts-catalog.yml installs dependencies, so both would fail to load the script after merge. Reproduced on a clean git archive HEAD (no node_modules): bun .github/scripts/sync-artifacts-catalog.ts validate-payload ... exits 1 with Cannot find module '@supabase/stack/internal/artifacts'. GitHub Scripts CI misses it because it installs the whole workspace.
Non-blocking: because the catalog is ahead of the Dockerfile, SUPABASE_USE_SLIM_IMAGES starts currently fall back to docker.io for most services. That matches the description, just flagging that the opt-in path is mostly inert until Dependabot catches up.
Adds `upstreamImage` to each catalog `ArtifactPin` (the exact upstream image slim-services built or mirrored each pin from), backfilled through the sync script's manual mode. A new generator (apps/cli/scripts/render-service-dockerfile.ts) rewrites only the slim-capable `FROM ... AS <alias>` lines of the Dockerfile in place from that field, asserting the repository stays the one the Dockerfile already uses (imgproxy is the one documented exception); everything else (kong, Dependabot-managed job images) passes through untouched. Its drift test runs as part of the required apps/cli unit suite. Adds the `pg15` (generated) and `pg14` (hand-pinned, no slim build) Dockerfile stages as the single version table for legacy Postgres resolution. `postgresImageForDbMajorVersion` now reads them instead of three hardcoded fallback constants; the Go CLI's own `pg14`/`pg15` constants are replaced with the same Dockerfile-derived fields, closing the second version table. One-time regeneration bumps every slim-capable service to its currently pinned upstream version.
Replaces literals that happened to equal the current catalog/Dockerfile pin with values derived from the catalog (`catalogPins()`/`postgresVersion`) or the Dockerfile (`dockerfileServiceImageRaw`), so a routine version bump no longer turns these integration tests red. Mocked, self-contained fixture catalogs (services.shared/slim-images unit tests) are unaffected: they only ever compare against their own literals.
sync-artifacts-catalog.ts writes upstreamImage straight into
Artifacts.ts as a TypeScript string literal. Validate the resolved
value against an anchored image-reference pattern before returning it
from resolveUpstreamImage, and serialize every field in serializePin
with JSON.stringify instead of manual "${...}" interpolation, so a
release manifest or provenance file carrying a quote or expression
can't inject code into the generated catalog.
start.integration.test.ts's stack-backend database fixture derives its
version from the real catalog (postgresVersion("17")) instead of a
hardcoded pin, so a routine Postgres bump can't desync it from the
version the code under test resolves. render-service-dockerfile's
hand-managed-line preservation test no longer freezes kong/differ/
migra/pgprove's literal tags; it now tampers them with a synthetic
marker and asserts the generator leaves them untouched, so a
Dependabot bump of any of those can't fail this required check.
slim-release-published now reconciles a service against every committed slim-services release (planSlimUpdates) and opens or updates one PR per hotfix and/or upgrade it finds, instead of only matching the release named in the dispatch. Plan and apply run from the same checkout in a single job so a re-run always recomputes from develop. Manual mode gains --release to pin an exact committed revision.
The stack catalog is now the single version table; the Dockerfile's slim-capable lines are generated from it and updated by slim-release-published, not by syncing a Dependabot bump back into the catalog. Dependabot now ignores every slim-capable image and keeps bumping only the upstream-only ones.
The comment said the fallback "only happens for an alias with no slim build", but toSlimImage also returns undefined for an OrioleDB tag (an alias that does have a slim build) and for a tag the catalog simply doesn't pin. State the general contract instead: undefined whenever no catalog pin matches the alias and tag, with all three cases as examples. Comment-only; no behavior change.
Guards slim-release-published against a releases API that hasn't caught up with the release that triggered the dispatch: plan-updates now requires --expect-release to be listed before planning, retrying a bounded number of times first. Also drops the dead placeholderPin parsing path now that the catalog only ever holds resolved pins, and stops embedding raw dispatch values in error messages so a crafted payload can't inject workflow commands.
Several slim-image tests computed their expected value with toSlimImage itself, so they passed whether or not the catalog lookup actually resolved. Under the single version table, a default Dockerfile tag always matches a catalog pin, so these now read the expected ghcr.io image straight from catalogPins() and assert its shape, catching a regression to docker.io. Also has services.shared.ts reuse slim-images.ts's imageTag/imageRepository instead of re-implementing the @-aware tag split.
Six test files each carried a near-identical "look up the catalog's pinned image for this alias" helper, plus three copies of the ghcr.io shape regex. Consolidates both into slim-images.test-support.ts, next to slim-images.ts.
Routes every ::error :: and ::warning :: line this script emits through one encoder (workflow-command data encoding: %, then \r/\n), so an external value reaching the top-level catch or a planner warning can never inject an extra workflow command via an embedded newline. Also validates --service in plan-updates before any listing or wait, and makes the expect-release wait interval reject a malformed SLIM_UPDATES_EXPECT_RELEASE_WAIT_MS instead of silently coercing it to NaN, 0, or a negative delay.
avallete
requested changes
Sep 30, 2026
avallete
left a comment
Member
There was a problem hiding this comment.
Re-reviewed at 89fb625. The single-table redesign holds up: plan and apply run from one checkout, each branch pins exactly one --release, validation is anchored, and the rendered Dockerfiles match the Go copy. Almost everything from the first round is fixed (replies on those threads).
One small must-fix and one reliability gap (inline), plus non-blocking notes:
- A same-date Studio build (
2026.09.28-sha-avs2026.09.28-sha-b) compares equal, so a later same-day build never gets an automatic upgrade PR. The comment says that's intended and the manual--releasepath covers it; flagging it in case it's not. upstreamImageand thefetchManifest/fetchProvenancefetchers are optional only so tests can skip stubbing them, butArtifactPin.upstreamImageis required.refreshCatalogPinwith such an io rewrites the pin without it, producing anArtifacts.tsthat doesn't typecheck (the test nearsync-artifacts-catalog.test.ts:1174locks that in). Consider making them required.- A hotfix and an upgrade planned on the same line open two PRs editing the same span; after the upgrade merges, the
slim-hotfix/<svc>PR is left conflicting and never revisited. Skipping the hotfix when an upgrade is planned on that line would avoid it. - Unlinked PG13/15 local projects move to
15.19.0.002, and every later catalog upgrade on the 15 line moves existing local volumes with it. Probably fine within a major, but worth a release-note line. - The load-hardening changes to
upload-release-assetsandsweep-live-projectstests look unrelated and would be easier to review separately. - Comment density is still well above the repo's quarter guideline in
sync-artifacts-catalog.tsand the workflow; the mode docs could move into the ADR.
avallete
self-requested a review
September 30, 2026 17:52
avallete
approved these changes
Sep 30, 2026
avallete
reviewed
Sep 30, 2026
avallete
left a comment
Member
There was a problem hiding this comment.
Default supabase start fails on Vector 0.58. stack start does not.
Dogfooded d0574ea. The legacy entrypoint is unchanged on this head.
packages/stack/src/services/Vector.ts writes the pipeline on the host and bind-mounts it, so stack start --runtime docker and --runtime native come up on the new pins.
Legacy Compose does not. buildVectorEntrypointScript in apps/cli/src/commands/start/services/vector.service.ts still does:
cat <<'EOF' > /etc/vector/vector.yamlVector 0.58 (slim 0.58.0-r0 and timberio/vector:0.58.0-alpine) has no /etc/vector. Both slim and flag-off supabase start crash-loop Vector and the health check times out:
sh: can't create /etc/vector/vector.yaml: nonexistent directory
ERROR vector::config::loading: Config file not found in path. path="/etc/vector/vector.yaml"
HealthCheckTimeoutError: supabase_vector_* container is not ready: unhealthy
Vector starts by default (analytics.enabled). --exclude vector skips it. Catalog digests matched on GHCR and the ECR mirror, and both native mirrors matched the pinned sha256. The rest of the slim service smoke did not run, because start aborted on Vector.
This blocks the default local start. It does not block stack start.
The S3 native-archive mirror uploads in parallel with the dispatch, best- effort, so a freshly committed release's S3 copy can lag GitHub by the time the sync script pins it. Waits, bounded, while an object is missing (reusing the same retry helper --expect-release's visibility wait uses), but fails immediately on a digest mismatch, since that's corruption rather than lag.
The apply loop ran bun x oxfmt (fetching ~46 packages from the public registry) and every sync/generator call in the same step that held the app token in its env, and persisted the token into .git/config via git remote set-url. Formats with the locked pnpm install instead, pushes with the token only in an explicit URL computed once (never written to disk), and strips it from every bun/pnpm command with env -u. gh now also gets the token inline per call, and filters gh pr list to same-repository PRs (isCrossRepository) so a fork PR with a matching branch name can't be selected and edited with a trusted-looking title and body.
Vector 0.58's images (upstream and slim alike) ship no /etc/vector, so the legacy start entrypoint's heredoc write failed and crash-looped the container in both flag-off and slim supabase start. Create the directory before writing the config.
Contributor
Author
|
Thanks for the re-review and the dogfooding. Vector on legacy The two inline items (the oxfmt install path and the S3 mirror wait) and the fork-PR lookup are fixed too; details are on those threads. On the summary notes:
|
jgoux
enabled auto-merge
September 30, 2026 18:41
jgoux
deleted the
juliengoux/cli-2512-slim-services-unify-integrity-hotfix-model-for-native
branch
September 30, 2026 18:54
Member
Dogfood — conditional goTested flowchart LR
pins[Catalog pins] --> slim[Legacy slim start]
pins --> off[Flag-off start]
pins --> native[Native cache]
native --> tests[Retry contract tests]
tests --> gap[Live GitHub to S3 not injected]
What ran
Kong stayed on ECR GapLive native fallthrough (GitHub release 404, then S3) was not proven on the real CLI. A A live GHCR→ECR pull failure was not injected: that needs the daemon to fail Soft, not product failures
|
TFSebben
pushed a commit
to TFSebben/supabase_cli
that referenced
this pull request
Oct 1, 2026
…hotfixes (supabase#6926) Follows up the non-blocking review notes on supabase#6883. - **Required fetchers.** `RevisionIo.fetchManifest` and `fetchProvenance` are now required, and `refreshCatalogPin` always resolves `upstreamImage`. The serializer only accepts a pin type where that field is required, so a rewritten `Artifacts.ts` always typechecks. - **One update per release line.** When a line upgrades, `planSlimUpdates` skips that line's hotfix. It emits a warning that `--service <svc> --release <U>-r<N>` can pin the hotfix alone. Otherwise the `slim-hotfix/<svc>[-<line>]` PR would conflict once the upgrade merges. Separate lines, such as postgres 17 and 15, stay independent. - **Docs over comments.** The sync script's mode docs and the `slim-release-published.yml` header move into ADR 0026: - a new "Invocations" section; - "Hotfix and upgrade pickup" now covers payload trust, the per-line rule, line assignment and app-token handling. Provenance, narration and over-long comments are trimmed across the files supabase#6883 changed. - **Listing tests.** `services.shared.unit.test.ts` covers two `supabase services` cases against the real catalog again: - the default slim listing; - a catalog-pinned postgres 15 override. The auth-only fixture stays the default for the other tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
slim-services now publishes immutable
<upstream>-r<N>releases (supabase/slim-services#326, #328). This PR makes the CLI consume them safely and from one place:packages/stack/src/Artifacts.ts) is the only place slim-capable service versions are written down. The new stack, legacysupabase start, and legacy slim mode (SUPABASE_USE_SLIM_IMAGES) all derive from it.Pinning
ArtifactPins:upstreamVersionrevisionimage: …:<U>-r<N>@sha256:…upstreamImage: the upstream image slim-services built from or mirrored, as recorded in the releasenatives: sha256s per targetSHA256SUMSand GHCR checksum lookups are gone. The GitHub release and S3 serve bytes only.slim-services/<svc>/<R>/<target>, so a hotfix revision gets its own cache entry.One version table
FROMlines ofapps/cli/src/shared/services/Dockerfile, and its Go copy, are generated from the catalog byapps/cli/scripts/render-service-dockerfile.ts.darthsim/imgproxy.pg14, and the job images (migra, pg_prove, pgadmin-schema-diff) are still maintained by hand or by Dependabot.apps/cliunit project fails when the Dockerfile and the catalog disagree.pg15(generated) andpg14(hand-pinned) stages replace the hardcoded constants, in both the TS and the Go CLI.pg15through the catalog, like every other alias.Updates
slim-release-published.ymlreconciles the dispatched service against every committed slim-services release. For each release line it opens, or rewrites in place:slim-hotfix/<svc>[-<line>]), when the pinned upstream has a newer revision;slim-bump/<svc>[-<line>]), when a newer upstream exists on that line.jgoux.::error ::/::warning ::line uses workflow-command data encoding. The app token reaches onlygit pushandgh: it is never written to.git/config, and everybun/pnpmcommand (the formatter runs from the locked install) runs without it. The PR lookup ignores fork PRs. Payload and release-tag values are validated against anchored patterns, and recorded image references are validated and escaped before they are written into the catalog.sync-artifacts-catalog.yml) is deleted.Artifacts.tsand the two Dockerfiles.Other changes needed by the new versions
${VAR}in its config.LOGFLARE_URLandLOGFLARE_PRIVATE_ACCESS_TOKEN.--dangerously-allow-env-var-interpolation.supabase start. Vector 0.58's images don't ship/etc/vector, so the entrypoint creates it before writing the config. Astart.lifecyclee2e scenario starts Vector against a real Logflare.pgdelta.seam.layer.tscompares slim images by revision or digest, so a container on-r0is reported stale once the catalog moves to-r1.upload-release-assets: the retry that must succeed had to fit in the same 500ms as the attempt that hangs on purpose.stack-shadow: a real-Docker test ran under vitest's 5s default timeout.sweep-live-projects: a 10s hard kill inside the test's budget.Docs
ADR 0026 is rewritten for this model, including the tradeoffs:
pg14are bumped by hand;infra/cli-artifacts/README.mdis updated to match.Release notes
supabase/postgres:15.19.0.002instead of15.8.1.085. Later catalog upgrades on the 15 line move them within the same major.supabase startruns the catalog's upstream versions (table below), and Storage is pinned to thev1.79.28-r1hotfix.Versions
Every catalog entry pins its newest committed revision. Legacy
supabase startnow runs the same upstream versions:17.6.1.173→17.11.0.002-r017.6.1.171→17.11.0.00215.14.1.173→15.19.0.002-r015.8.1.085(constant) →15.19.0.002v16.2→v16.4-r0v16.3→v16.4v2.196.0→v2.197.0-r0v2.197.0(unchanged)v2.134.5→v2.140.3-r0v2.135.3→v2.140.3v1.73.0→v1.79.28-r1v1.77.0→v1.79.28v3.8.0→v3.26.0-r0v3.8.0→v3.26.0v1.77.1→v1.77.1-r0v1.77.1(unchanged)2026.09.04-sha-5a67366→2026.09.28-sha-5e59b60-r02026.09.14-sha-4dd8a95→2026.09.28-sha-5e59b60v0.99.0→v0.99.0-r0v0.99.0(unchanged)v1.30.2→v1.31.3-r0v1.30.2→v1.31.3v1.50.9→v1.50.15-r01.50.12→1.50.150.53.0→0.58.0-r00.53.0-alpine→0.58.0-alpinev2.9.12→v2.9.13-r02.9.13(unchanged)