feat(supervisor): stage gateway configuration snapshot delivery by pimlock · Pull Request #3244 · NVIDIA/OpenShell

pimlock · 2026-09-09T19:39:58Z

Summary

TL;DR Supervisors currently poll the gateway for configuration every 10 seconds. This PR adds an opt-in push path. Polling remains authoritative in 0.1.x.

Stage 1 sends full snapshots when the gateway setting is push and the supervisor advertises snapshot support. Each update reaches the gateway that owns the affected supervisor session. The supervisor ignores the snapshot. Its existing polling path still applies configuration.

The setting defaults to poll in 0.1.x. The plan makes push the default in 0.2.0, deprecates polling there, and removes polling APIs in 0.3.0. Helm users can set this field through gatewayConfig after #3384 lands.

Tip

Overview of changes.

Related Issue

Part of #1731. Stage 2 (#3265) will apply and acknowledge snapshots. Stage 3 (#3273) will track completion of global updates.

Changes

Add bootstrap, snapshot, and acknowledgement contracts. Negotiate snapshot support through SupervisorHello.supports_config_snapshots so older supervisors keep polling.
Add the opt-in gateway setting while keeping polling and older supervisors working.
Route updates to the owning HA gateway and schedule them without drops, coalescing repeats.
Send provider updates only to sandboxes attached to that provider.
Initialize policy history without overwriting existing apply results.

Testing

mise run pre-commit; Rust workspace and server tests; mise run go:ci; mise run sdk:ts:ci
Scheduler unit tests cover coalescing, fanout cursors, build-slot fairness, provider filtering, recipient epochs, and lost-update regressions
Helm lint and rendering; verified that feat(helm): migrate gateway configuration to gatewayConfig #3384 renders config_delivery_mode = "push" from gatewayConfig
Local Kubernetes HA smoke test in push mode (kind, two replicas, PostgreSQL, Envoy), with feat(helm): migrate gateway configuration to gatewayConfig #3384 merged locally. Conformance passed, and both replicas ran scheduler fanouts. sandbox_file_sync_survives_gateway_pod_rolls failed in 2 of 3 push runs and 1 of 3 poll runs with the same "sandbox is not ready" retry error, so it is an existing flake, not a push regression.

Checklist

Conventional Commits and DCO sign-off
Existing polling APIs remain available in 0.1.x

github-actions · 2026-09-09T19:40:25Z

🌿 Preview your docs: https://nvidia-preview-pr-3244.docs.buildwithfern.com/openshell

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

The startup repair that creates version-one policy history for legacy sandboxes propagated validation failures, so a single stored policy that no longer passes current validation rules prevented the gateway from starting. Skip such sandboxes with a warning and a completion summary so they keep the pre-repair behavior where only their own configuration reads report the failure. Store errors remain fatal. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Fleet-wide configuration changes spawned one snapshot build per connected sandbox and component with no concurrency limit, so a global setting or provider change issued every store query and credential-driver call at once. Gate builds behind a semaphore sized from the database pool and start the build deadline only once a permit is held. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Sandboxes keep their supervisor binary until they are recreated, so a gateway upgrade meets supervisors that predate the handshake and report revision zero. Rejecting them severs every running sandbox with no automatic recovery. Accept revision zero for one release, log a warning per session, and count them in openshell_supervisor_protocol_legacy_sessions_total. The supervisor mirrors the allowance for gateways that predate the handshake. Add a shared ConnectSupervisor test harness and handler-level tests for legacy acceptance and unknown-revision rejection. Move the skill troubleshooting paragraph out of the numbered deployment list so the list renders. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock

gator-agent

PR Review Status

I reconciled the current head against the full Gator feedback ledger and reviewed the author-only delta since bd9f91e5 in critical-only mode. The range-diff shows all ten PR-authored commits are patch-identical after the merge from main, the earlier fanout-bounding obligation remains resolved, and no new Critical defect was found. Current-head Branch Checks and E2E workflows are running.

Blocking findings:

No blocking findings remain

Carried findings:

None

Gator metadata

Validation: Project-valid Stage 1 of accepted issue #1731; the effective PR patch remains within the reviewed gateway/supervisor configuration-delivery scope.
Docs: Architecture and Fern gateway-reference documentation remain included for the current behavior.
Checks: OpenShell Helm Lint and Trivy Changes are green; Branch Checks and E2E are pending on the current head.
E2E: test:e2e and test:e2e-kubernetes are applied; the current-head Branch E2E Checks workflow is running.
Head SHA: 8dfe13b575a931586a5bb26238230227bdc40d82
Base SHA: fa0bfa490e42c87a74a70be6ebb40faee7fb8faa
Merge base SHA: fa0bfa490e42c87a74a70be6ebb40faee7fb8faa
Patch ID: ca184a2ffbb6b7e1d73cbd0fc2a97533eca331e3
Gator payload: 9
Review mode: critical_only
Previous reviewed SHA: bd9f91e593c2096542599aebaad43a2c179426e9
Review budget exhausted: yes
Maintainer decision required: no
Next state: gator:watch-pipeline

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Replace the bounded delivery queue with a synchronous scheduler that never rejects admission. Every publication takes a sequence number and a key is rebuilt only while its last build predates the publication, so repeated publications coalesce. Fanouts are cursors over sorted recipients; direct and fanout builds share build slots with a direct reserve and a fanout floor, and a separate route lane handles owner lookups and peer hints. Provider instance changes and credential refreshes publish a targeted provider scope that fans out only to sandboxes attached to that provider, filtered by a monotonic attachment cache. Add the PeerConfigProviderTarget hint scope and remove the unreleased queue_full field. Push-mode sessions get one latest-wins slot per component, drained by an outbound stream that sends control frames first. Sessions register with the scheduler only after setup succeeds and only while still current, and recipient epochs fence stale tickets. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Resolve the failed-create cleanup against the version-guarded delete from main: remove the initial policy revision only after the sandbox row is deleted. Regenerate Go bindings, update the public schema fingerprint, and add PeerNotifyConfigUpdate to the new CLI exec streaming mock gateway. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Replace the exact-match protocol_revision handshake with a SupervisorHello.supports_config_snapshots capability, matching the existing supports_provider_readiness pattern. Peers no longer reject unknown revisions, so gateway rollback and mixed-version HA replicas keep supervisor sessions alive. SessionAccepted no longer carries a revision. Document SandboxConfigSnapshot and ProviderEnvironmentSnapshot as the canonical configuration shapes, with the GetSandboxConfig and GetSandboxProviderEnvironment responses as polling projections. State ProviderEnvironmentValue invariants and number its expiration_time field 3 instead of reserving a field that never shipped. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Replace the shared 45-second build deadline with one per component. Sandbox configuration builds only read the store, so 5 seconds is ample. Provider environment builds also resolve credentials; 20 seconds covers a cold Vault Kubernetes-auth login plus a read at the default request timeout. A stalled build no longer holds a build slot for 45 seconds. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Take main's failed-create path, which compensates through the normal delete and already removes policy revisions. Compensate the same way when initial policy history fails, so a prepared SSH identity is cleaned up. Update the public schema fingerprint. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 9, 2026 19:39

pimlock mentioned this pull request Sep 9, 2026

feat(supervisor): push gateway desired state #2967

Closed

6 tasks

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

pimlock added gator:in-review Gator is reviewing or awaiting PR review feedback gator:blocked Gator is blocked by process or repository gates and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Sep 10, 2026

This comment has been minimized.

Sign in to view

pimlock added the test:e2e Requires end-to-end coverage label Sep 10, 2026

This comment was marked as outdated.

Sign in to view

pimlock mentioned this pull request Sep 10, 2026

feat(supervisor)!: apply streamed configuration snapshots #3265

Open

5 of 6 tasks

pimlock added this pull request to stack #3266 September 10, 2026 23:38

pimlock added 8 commits September 10, 2026 18:58

feat(supervisor): stage gateway configuration snapshot delivery

ea6c689

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore(sdk): regenerate Go proto bindings for protocol revision comment

fe2fa4e

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(supervisor): bound optional bootstrap latency on reconnect

3053b2f

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(server): bound configuration delivery workers

86773ee

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(server): update configuration tests for workspace selector

70773d3

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock force-pushed the 1731-config-update-stage-1/pimlock branch from e7729ec to 70773d3 Compare September 11, 2026 02:08

pimlock added 2 commits September 10, 2026 19:49

fix(proto): refresh public schema inventory

c645112

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into configuration snapshot branch

36bd16d

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock mentioned this pull request Sep 11, 2026

Push gateway-owned desired state to supervisors #1731

Open

pimlock added 4 commits September 17, 2026 20:42

chore: merge latest main into config activation

33e8449

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

test(server): assert repairable invalid policy snapshot

1e49913

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into stage 1 configuration updates

99d99cc

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into stage 1 configuration updates

bd9f91e

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

This comment was marked as outdated.

Sign in to view

pimlock added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label Sep 19, 2026

This comment was marked as outdated.

Sign in to view

pimlock added gator:blocked Gator is blocked by process or repository gates and removed gator:approval-needed Gator completed review; maintainer approval needed labels Sep 19, 2026

chore: merge main into config snapshot stack

8dfe13b

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock commented Sep 19, 2026

View reviewed changes

pimlock added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 19, 2026

pimlock added 15 commits September 21, 2026 09:44

docs(architecture): correct configuration snapshot boundaries

7f88f5d

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

2091c38

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

f56e7f6

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

6344aad

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot stack

9d5a3f2

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore(config): merge main into gateway snapshot branch

d7cbee6

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

feat(config): route shadow push through HA owners behind gateway mode

c013645

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(config): coalesce direct and fleet updates per component

0e3d78b

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(config): use generic Helm gateway configuration

b04c20f

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

docs(config): remove public guidance for internal push rollout

f1e8503

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Navigation Menu

Repository navigation

feat(supervisor): stage gateway configuration snapshot delivery - #3244

feat(supervisor): stage gateway configuration snapshot delivery#3244
pimlock wants to merge 36 commits into
mainfrom
1731-config-update-stage-1/pimlock

pimlock commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

This comment was marked as outdated.

This comment has been minimized.

This comment was marked as outdated.

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

pimlock left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pimlock commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

This comment was marked as outdated.

This comment has been minimized.

This comment was marked as outdated.

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

pimlock left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pimlock commented Sep 9, 2026 •

edited

Loading