Repository navigation
bug(gateway): finalized ephemeral sandboxes remain Completed while supervisor stays connected #3938
Description
User Story
As an operator running short-lived agent jobs, I want an ephemeral sandbox to be removed after its main process finishes and its terminal output has been delivered, so that completed jobs do not leave compute resources running.
Problem Statement
A sandbox created with --no-keep --detach receives the openshell.nvidia.com/retention: ephemeral annotation, but remains in Completed after its canonical process exits successfully. Its supervisor stays connected to retain the control-mode access plane, so the sandbox and its Docker containers remain present indefinitely until manually deleted.
Impact / Why This Matters
Unattended job launchers such as Gator can accumulate completed sandboxes and running supervisor/compute containers despite requesting self-deletion. The current workaround is to list and manually delete completed ephemeral sandboxes, which requires ongoing operator attention and does not prevent resource buildup between cleanups.
Acceptance Criteria
- After the canonical process result and terminal delivery are finalized, an ephemeral sandbox is deleted even while its supervisor session remains connected.
- Cleanup does not begin before terminal output delivery is finalized.
- Retained sandboxes remain available after their canonical process exits.
- A gateway regression test covers a finalized ephemeral sandbox with a still-connected supervisor session and verifies that cleanup starts without a disconnect.
- A real end-to-end test runs
--no-keep --detachagainst the Docker, Podman, Kubernetes, and VM driver lanes, then verifies that the sandbox record and driver resources disappear without a CLI delete. Cover both successful and failed canonical-process exits. Add equivalent coverage to the separate MXC e2e lane where that driver supports this lifecycle.
Reproduction Steps
- Start the local Docker gateway (
mise run gateway:docker). - Run
openshell --gateway docker-dev sandbox create --name eph-check --no-keep --detach -- /bin/true. - Run
openshell --gateway docker-dev sandbox get eph-check -o jsonanddocker psafter the command exits. - Observe
phase: Completed,exit_code: 0, andopenshell.nvidia.com/retention: ephemeral. The sandbox and supervisor containers remain running. The supervisor logsCanonical process exited; retaining control-mode access plane. - Manually delete the sandbox with
openshell --gateway docker-dev sandbox delete eph-check.
Environment
- OpenShell:
0.1.3-dev.28+g798500ccd - OS: macOS 26.7, arm64
- Runtime: Docker Desktop engine 29.7.2; local
docker-devgateway - Reproduced on 2026-09-29 local time
Logs
$ openshell --gateway docker-dev sandbox list
NAME CREATED PHASE
eph-check 2026-09-30 06:20:08 Completed
supervisor: Canonical process exited; retaining control-mode access plane
Agent Investigation
The gateway schedules ephemeral deletion only in the supervisor-session disconnect path when terminal_delivery_finalized is true (crates/openshell-server/src/compute/mod.rs, around line 4456). The supervisor intentionally retains control-mode access after reporting and finalizing the process exit (crates/openshell-supervisor/src/lib.rs, around line 1293). The existing test ephemeral_cleanup_waits_for_terminal_finalization explicitly waits for a disconnect before expecting deletion (crates/openshell-server/src/compute/mod.rs, around line 9265). The finalize RPC records terminal delivery while the session is still connected (crates/openshell-server/src/supervisor_session.rs, around line 2055).
The shared Rust lifecycle e2e suite already tests attached --no-keep commands, but attached CLI cleanup can mask the gateway bug (e2e/rust/tests/sandbox_lifecycle.rs, around line 1345). Its detached canonical-process tests use persistent sandboxes (e2e/rust/tests/sandbox_lifecycle.rs, around line 760). Extend that shared suite with detached ephemeral cleanup and run it in each applicable driver lane; MXC has a separate Windows e2e runner.
Activity
johntmyers commentedon Sep 30, 2026
Implementation Plan
Issue type:
fixComplexity: Medium
Confidence: High on the gateway cause; Medium on cross-driver execution
Summary
After the gateway validates and finalizes an ephemeral sandbox's terminal result, start the existing deletion path without waiting for its supervisor session to disconnect. Keep disconnect-triggered cleanup as a fallback for ordering races. Add unit and detached end-to-end coverage that cannot be satisfied by the CLI's attached-session cleanup.
Scope
crates/openshell-server/src/supervisor_session.rsandcrates/openshell-server/src/compute/mod.rs: trigger deletion after finalization/session bookkeeping for a terminal ephemeral result, preserving instance validation, restart transitions, and retained sandboxes.--no-keepsandboxes with successful and failed commands; verify the gateway record and compute resources disappear without a CLI delete in Docker, Podman, Kubernetes, and VM lanes. Ensure the Podman CI selection and VM wrapper actually run the test.Implementation Steps
Test Plan
--no-keepsuccess and failure in Docker, Podman, Kubernetes, and VM, checking both gateway absence and driver resource removal. Run available lanes locally and report any environment-limited lanes precisely.Risks & Open Questions
compute.finalize_main_process_exitwould race the session registry mark and could make the finalize RPC fail. The trigger belongs after the registry mark, with a safe fallback if the session disappears between those operations.ConnectSupervisoror the supervisor finalization RPC. This gateway trigger cannot exercise MXC; equivalent MXC cleanup coverage requires its separate lifecycle path and is outside this fix.Documentation Impact
No gateway TOML, driver options, Helm rendering, public API, or documented workflow changes. This restores the existing
--no-keeppromise. No LSM-sensitive process identity or/procbehavior changes.Revision 1 — initial plan
johntmyers commentedon Sep 30, 2026
Implementation Complete
PR: #3984
The gateway starts ephemeral driver deletion once the canonical main-process result is finalized, even while the supervisor control session remains connected. Existing disconnect cleanup remains as a fallback.
Three focused unit tests cover connected cleanup, failure cleanup, and retained/restarting exclusions. Two shared detached e2e tests check both gateway record deletion and driver resource removal across the supervisor-based driver suites. Local Docker execution passed both tests and all six conformance scenarios. Final full local CI and pre-commit checks passed; the earlier unrelated supervisor-network timeout did not recur in the final CI run. Live Podman, Kubernetes, and VM verification remains unperformed locally.
Updated the Gator skill's cleanup description; the published sandbox docs already describe the intended behavior. This issue will close when the PR merges.
fix(gateway): delete finalized ephemeral sandboxes while connected (N…