Push gateway-owned desired state to supervisors

[pimlock]

Summary

Replace supervisor polling for gateway-owned configuration with bootstrap and live snapshot delivery over the existing reverse ConnectSupervisor session. The rollout is staged and opt-in: polling keeps working until a migration removes it in 0.3.0. Add durable completion tracking so callers can return after commit, or wait for the exact sandbox configuration revision to reach a terminal state.

The persistent supervisor session and its SSH/exec relay substrate landed in PR #867. Warm-pool registration in PR #2460 remains a pre-session identity and activation step. Managed inference routing was removed in PR #3195 and is no longer part of this work.

Resolved design

• Rollout setting: a gateway-owned config_delivery_mode = "poll" | "push" setting controls the rollout.
  • An omitted setting means poll in 0.1.x and push from 0.2.0. An explicit poll stays valid through 0.2.x.
  • 0.3.0 removes polling after a recreate or drain migration.
  • In poll mode the push path is dormant.
• Protocol compatibility: supervisors advertise a protocol revision.
  • Legacy supervisors are accepted, keep polling, and never receive pushed snapshots.
  • Stage 2 negotiates the applied mode in the handshake instead of inferring it from the revision.
  • Old and new gateways and supervisors keep working together through 0.2.x.
• Bootstrap: in push mode, push-capable supervisors receive complete sandbox configuration and provider-environment snapshots in SessionAccepted.
  • Stage 1 sends them as a bounded shadow, and polling stays authoritative.
  • From Stage 2, supervisors apply the bootstrap before starting gateway-owned runtime state and report the result.
• Live changes: each change is a complete, one-component ConfigUpdate snapshot, built from authoritative state at delivery time, with session-scoped correlation and ordering. One delivery scheduler bounds the work:
  • Admission never drops updates. Repeated publications coalesce, and only the latest state is built and sent.
  • Fleet-wide updates (all connected sessions, a workspace, or a provider) walk the connected sandboxes like a cursor. They cannot starve sandbox-scoped updates, which have reserved build capacity.
  • Provider changes and credential refreshes rebuild only the sandboxes that attach that provider.
  • Each session holds at most one unsent snapshot per component, and heartbeats and relay control go first.
  • Delivery starts from a fresh bootstrap after reconnect.
• Standalone and local policy: standalone file-backed operation remains independent. Explicit local policy keeps precedence and returns an observable retained-local-override outcome.
• Cross-replica delivery: the database remains authoritative, and the gateway that owns the live session delivers to it.
  • Other replicas send it secret-free hints, and the owner rebuilds from shared state.
  • Owner reconciliation and durable fanout intent repair missed hints, owner handoff, and a crash between commit and publication.
• Durable operations: sandbox-scoped policy and settings mutations commit desired state and a durable, non-secret operation atomically.
  • COMMIT_ONLY returns after commit.
  • WAIT_FOR_APPLY waits for an exact correlated result or an authoritative terminal lifecycle state.
  • Idempotency keys make retries stable.
  • A newer delivered revision supersedes older pending operations.
• Recovery: pending operations survive client timeout and gateway restart. Reconciliation rebuilds current snapshots and republishes them with bounded backoff. No database transaction stays open during delivery or waiting.
• Retained and removed RPCs: GetSandboxConfig, ReportPolicyStatus, SubmitPolicyAnalysis, PushSandboxLogs, and RelayStream keep their existing roles. 0.3.0 removes supervisor polling and the supervisor-only provider-environment fetch RPC after the migration.

Scope

• Define bootstrap, snapshot, update, result, revision, and durable operation contracts.
• Add the gateway rollout setting, and keep legacy supervisors and mixed versions working through 0.2.x.
• Move gateway-backed initialization and live configuration changes to the supervisor stream.
• Bound, coalesce, and prioritize delivery so fleet-wide updates neither drop nor starve sandbox-scoped updates. Rebuild only the sandboxes a change affects.
• Deliver through the session-owning gateway across replicas, with reconciliation for missed hints.
• Add atomic operation persistence, lookup, idempotency, terminal states, bounded waits, and crash-safe reconciliation for sandbox-scoped policy and settings mutations.
• Preserve local source ownership, last-known-good behavior, heartbeat and relay priority, and secret-safe logging.
• Update the CLI, SDKs, tests, architecture documentation, operator documentation, and public troubleshooting skill.

Non-goals

• Removing polling before the 0.3.0 migration.
• Persisting configuration snapshots, credentials, or an event-by-event delivery log.
• Moving status, analysis, logs, or relay bytes onto the shared session protocol.
• A gateway-initiated supervisor RPC server, endpoint discovery, inbound auth or mTLS, Kubernetes networking changes, or a generic transport abstraction.
• In-place supervisor replacement, or changes to multi-replica session ownership itself. Delivery uses the existing owner index.
• Changing supervisor image defaults. Deployment-default cleanup is separate work.

Acceptance criteria

• Compatibility: in poll mode, and for legacy supervisors, configuration behaves as it does with 0.1.0 polling. Old and new gateways, supervisors, and clients keep working together through 0.2.x.
• Bootstrap: in push mode, gateway-backed supervisors apply a complete bootstrap before the gateway marks them initialized. Missing, failed, oversized, timed-out, or revision-mismatched exchanges fail clearly within a bound.
• Live changes: in push mode, live policy, settings, middleware, and provider changes arrive without supervisor polling.
  • Duplicate, stale, failed, coalesced, timeout, and reconnect cases preserve correct runtime state.
  • Fleet-wide updates never drop or starve sandbox-scoped updates.
  • Provider changes rebuild only the sandboxes that attach the provider.
• Durable operations: COMMIT_ONLY and WAIT_FOR_APPLY expose durable outcomes for the exact policy and settings revision tuple, without holding database locks across network work.
  • Operations resolve as applied, inactive, failed, superseded, or cancelled, from correlated results or authoritative lifecycle state.
  • Client cancellation does not erase the operation.
• Cross-replica: a write through another gateway replica reaches the session owner through a hint. Reconciliation repairs missed hints and owner handoff.
• Standalone and local policy: standalone supervisors require no gateway session, and explicit local policy remains authoritative.
• RPCs and documentation:
  • Retained RPCs remain compatible.
  • Supervisor polling and fetch paths are removed only in 0.3.0, after the migration.
  • Documentation describes the rollout setting, source ownership, reconciliation, and the operation lifecycle.

Implementation

• Stage 1, PR feat(supervisor): stage gateway configuration snapshot delivery #3244: opt-in shadow push. Includes the delivery scheduler, targeted provider fanout, and owner-directed hints between gateways.
• Stage 2, PR feat(supervisor)!: apply streamed configuration snapshots #3265: supervisors apply and acknowledge streamed snapshots, with reconciliation and retries.
• Stage 3, PR feat(config)!: add durable configuration completion #3273: durable completion operations.
• 0.2.0: push becomes the default. 0.3.0: polling is removed after the migration.

References

• Canonical implementation plan: issue comment
• Stage 3 transaction and state model: design gist
• Session and relay foundation: PR feat(server,sandbox): supervisor-initiated SSH connect and exec over gRPC-multiplexed relay #867
• Managed inference removal: PR refactor(inference): remove managed inference routes #3195
• Supervisor-session token expiry: feat(supervisor): enforce supervisor session token expiry with proactive reconnect #1954

[pimlock]

🏗️ build-plan

Implementation plan

Issue type: feat
Complexity: High
Confidence: High. The staged implementation defines the contract and verification boundaries.

Goal

Move supervisor configuration delivery from polling to gateway-pushed bootstrap and live snapshots over ConnectSupervisor. Roll it out behind a gateway setting so polling keeps working until the migration completes. Add durable completion semantics for sandbox-scoped policy and settings mutations.

Plan

1. Rollout control. Add a gateway-owned config_delivery_mode = "poll" | "push" setting (gateway TOML, CLI flag, environment variable).
   • An omitted setting means poll in 0.1.x and push in 0.2.x.
   • An explicit poll stays valid through 0.2.x.
   • In poll mode the push path is dormant.
2. Protocol compatibility. Supervisors advertise a protocol revision. Legacy supervisors are accepted, keep polling, and never receive pushed snapshots. Stage 2 negotiates the applied mode in the handshake instead of inferring it from the revision.
3. Bootstrap. SessionAccepted carries complete sandbox configuration and provider-environment snapshots for push-capable sessions.
   • Stage 1 sends it as a bounded, optional shadow.
   • Stage 2 applies it before gateway-owned runtime state starts and reports the result.
4. Live delivery. Later changes are complete, one-component snapshots built from authoritative state at delivery time. A single delivery scheduler, introduced in Stage 1, bounds the work:
   • Admission never drops. Repeated publications coalesce per sandbox component.
   • Every publication gets a sequence number. A component is rebuilt only if its last build started before that publication, so only the latest state is built and sent.
   • Fleet-wide updates (all connected sessions, a workspace, or a provider) walk the connected sandboxes like a cursor. A newer publication during a pass costs at most one more pass.
   • Sandbox-scoped updates have reserved build capacity and are never starved by fleet updates. Fleet updates always keep at least one build slot.
   • Provider changes and credential refreshes rebuild only the sandboxes that attach that provider.
   • Each session holds at most one unsent snapshot per component. A newer snapshot replaces it, and heartbeats and relay control are sent first.
5. HA routing. The database is authoritative, and the gateway that owns the live session delivers to it.
   • A non-owning gateway sends a secret-free hint to the owner, which rebuilds from shared state.
   • Fleet-wide hints go to every peer once per publication.
   • Single-replica gateways skip peer work.
6. Reconciliation (Stage 2).
   • Owner reconciliation and a durable fanout watermark repair three cases: missed hints, owner handoff, and a crash between commit and publication.
   • Failed builds retry with bounded backoff.
   • Database locks are never held while delivering or waiting.
7. Durable operations (Stage 3). Each sandbox-scoped policy or settings mutation commits a durable, non-secret operation in the same transaction. Operations support:
   • commit-only responses, server-side wait-for-apply, idempotent retries, and later lookup;
   • terminal states: applied, inactive, failed, superseded, or cancelled. A newer delivered revision supersedes older pending operations.
8. Standalone and local policy. Preserve standalone file-backed startup and explicit local policy ownership. Keep the last-known-good runtime state when a live update fails, according to the configured failure mode.
9. Polling removal. After a recreate or drain migration, 0.3.0 removes supervisor polling and the supervisor-only provider-environment fetch. Public configuration reads and supervisor-triggered status, analysis, log, and relay RPCs stay separate.
10. Docs and SDKs. Update the CLI, Go and TypeScript SDKs, tests, architecture and operator documentation, and the public troubleshooting skill.

Pull request sequence

• PR feat(supervisor): stage gateway configuration snapshot delivery #3244 (Stage 1): opt-in shadow push. Bootstrap and live snapshots are sent, but supervisors keep polling and ignore them. Includes:

  • the final delivery scheduler and per-session latest-wins slots;
  • targeted provider fanout;
  • owner-directed HA hints, including a provider scope.

• PR feat(supervisor)!: apply streamed configuration snapshots #3265 (Stage 2): supervisors apply and acknowledge bootstrap and live snapshots. It builds on the Stage 1 scheduler instead of replacing it:

  • acknowledgement gating holds the session slot;
  • failed builds retry with backoff;
  • fanout intent becomes durable.

  Still to decide: whether push stays opt-in in Stage 2.

• PR feat(config)!: add durable configuration completion #3273 (Stage 3): durable completion operations.

• 0.2.0: an omitted mode defaults to push, and polling is deprecated.

• 0.3.0: polling is removed after the migration.

Managed inference routing was removed in PR #3195, so inference snapshots are no longer part of this issue. Supervisor image default cleanup also remains separate.

Verification

• Bootstrap ordering, mixed old and new gateways and supervisors, poll-mode dormancy, bounded failure, reconnect, and standalone startup.
• Delivery scheduling:
  • no dropped updates, and coalescing to the latest state;
  • sandbox updates during fleet fanouts;
  • provider-scoped fanout, including attach races and out-of-order builds;
  • session replacement;
  • heartbeat and relay priority over configuration.
• Exact result correlation, duplicate and stale inputs, local override, last-known-good behavior, and ordering.
• HA: owner-directed hints, the owner moving after a build, peer failure, and cross-gateway reconciliation.
• Atomic desired-state and operation persistence, idempotent retries, terminal lifecycle states, timeouts, supersession, and restart recovery.
• Public API compatibility, SDK behavior, secret-safe persistence and logging, documentation, and Docker-backed and Kubernetes HA end-to-end coverage.

Run mise run pre-commit, mise run ci, SDK CI, and the relevant sandbox end-to-end lanes for each stage.

Out of scope

• Removing polling before the 0.3.0 migration.
• Persisted snapshot payloads, credentials, or an event-by-event delivery log.
• Moving status, analysis, logs, or relay bytes onto ConnectSupervisor.
• Direct supervisor RPC transport, discovery, inbound auth or mTLS, Kubernetes networking, transport abstraction, or in-place supervisor replacement.
• Supervisor image default changes.

---

Revision 7: staged, opt-in rollout with polling kept through 0.2.x and owner-directed HA hints. Stage 1 now carries the final delivery scheduler (lossless coalescing, sequence-based latest-wins, cursor fanouts, reserved capacity for sandbox updates, per-session latest-wins slots) and targeted provider fanout, so later stages build on it instead of replacing it.
Revision 6: aligned the canonical plan with the staged implementation, including exact protocol matching, durable apply operations, stream-only delivery, and the removal of managed inference routing.
Revision 5: consolidated the preparation and protocol work into draft PR #2967; supervisor-image cleanup remained separate.
Revision 4: selected ConfigUpdate, fingerprint-plus-sequence ordering, explicit component outcomes, bounded failure semantics, and a two-PR implementation split.