Stop→start leaves sandbox stuck at Starting on Agent Sandbox v1beta1 (stale Suspended condition read as Stopped)

[russellb]

User Story

As an operator running OpenShell on Kubernetes with the Agent Sandbox v1beta1 API,
I want openshell sandbox stop followed by openshell sandbox start to return the sandbox to service,
so that I can pause and resume sandboxes without recreating them.

Problem Statement

After a stop→start cycle, a resumed sandbox never reaches Ready. Both the workload
pod and (for proxy-pod topology) the supervisor pod are Running 1/1, and the
Sandbox CR reports Ready=True (DependenciesReady), yet the gateway keeps the
sandbox at phase Starting indefinitely and sandbox start fails with a
300s readiness timeout.

The gateway derives phase from the Sandbox CR conditions. On the Agent Sandbox
v1beta1 API, the controller sets Suspended=True (PodTerminated) when a sandbox
is stopped and does not clear it when the sandbox is resumed. After start the
CR therefore carries both Ready=True and a stale Suspended=True. The gateway's
phase derivation treats a present Suspended=True condition as Stopped before
it ever consults the Ready condition, so a resumed-but-Ready sandbox is read as
not-running and the Starting lifecycle phase is never allowed to advance to
Ready.

This is independent of sandbox topology (it stems from the shared gateway phase
derivation) and is not present with the v1alpha1 API, which encodes desired state
as spec.replicas and does not emit a lingering Suspended condition.

Impact / Why This Matters

When this happens, openshell sandbox start blocks for 300s and then errors with
timed out after 300s waiting for sandbox to reach Ready, even though the
workload is actually running and healthy. The sandbox is stuck at Starting in
sandbox list/get and cannot be used through the gateway.

The only workaround is to delete and recreate the sandbox, which defeats the
purpose of stop/start (pausing to save resources while preserving the sandbox)
and discards any workspace/session state the operator intended to keep. Any
automation that relies on stop/start for cost control or scheduling is broken on
v1beta1 clusters.

Acceptance Criteria

• On the Agent Sandbox v1beta1 API, openshell sandbox stop then
  openshell sandbox start returns the sandbox to phase Ready.
• openshell sandbox start completes without timing out when the workload
  and supervisor pods are Running and the CR reports Ready=True.
• A stale Suspended=True condition left on a resumed CR does not force the
  gateway phase to Stopped/Starting when the sandbox is otherwise Ready
  and its desired operating mode is Running.
• Behavior remains correct on the v1alpha1 API (no regression).

Reproduction Steps

1. Deploy an OpenShell gateway on a Kubernetes cluster using the Agent Sandbox
   v1beta1 API (Agent Sandbox v0.5.0).
2. Create a sandbox with a long-running entrypoint and wait for Ready, e.g.
   openshell sandbox create --name demo --driver-config-json '{"kubernetes":{"containers":{"agent":{"command":["sleep","infinity"]}}}}'
3. openshell sandbox stop demo and confirm the workload pod terminates.
4. openshell sandbox start demo.
5. Observe: the workload pod returns to Running 1/1, the CR shows
   Ready=True (DependenciesReady) alongside a stale Suspended=True (PodTerminated),
   but sandbox list shows phase Starting and the start command times out
   after 300s.

Environment

• OpenShell: gateway built from main (derive_phase unchanged through
  fb6610df); observed with 0.0.112-dev gateway image.
• Runtime: Kubernetes with Agent Sandbox v1beta1 (v0.5.0); reproduced on
  OpenShift 4.x with OVN-Kubernetes.
• Deployment: Helm chart, shared workspace mode. Observed with proxy-pod
  topology but the phase derivation is topology-independent.

Logs

# CR after `sandbox start` — Ready but stale Suspended:
$ kubectl get sandbox default--demo -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'
Ready=True (DependenciesReady)
Suspended=True (PodTerminated)
$ kubectl get sandbox default--demo -o jsonpath='spec.operatingMode={.spec.operatingMode}{"\n"}'
spec.operatingMode=Running

# CLI:
$ openshell sandbox start demo
Error:   × timed out after 300s waiting for sandbox to reach Ready

# Code pointer (evidence, not a prescribed fix):
# crates/openshell-server/src/compute/mod.rs — derive_phase() returns
# SandboxPhase::Stopped on any Suspended=True condition before it checks Ready.

[russellb]

📋 triage-agent

Triage Assessment

Classification: validated-bug

Summary

Confirmed real defect with high confidence. The report's diagnosis is accurate: on the Agent Sandbox v1beta1 API, a resumed sandbox carries a stale Suspended=True condition alongside Ready=True, and the gateway's phase derivation reads the stale Suspended as Stopped, pinning the sandbox at Starting and timing out sandbox start after 300s. Root cause verified in code and reproduced live on a v1beta1 cluster.

Investigation

Gateway mechanism (crates/openshell-server/src/compute/mod.rs, current main):

1. derive_phase returns SandboxPhase::Stopped on any Suspended=True condition before it consults the Ready condition (the Suspended check precedes the Ready loop).
2. ComposedPhase::new maps a backend phase of Stopped straight through even when a supervisor session is connected, bypassing the session-based Ready promotion.
3. The reconcile transition keeps old_phase == Starting unless the derived phase is Ready/Error; since it derives Stopped, the sandbox stays Starting indefinitely and start hits its 300s readiness timeout.

The Kubernetes driver's status_from_object (crates/openshell-driver-kubernetes/src/driver.rs) forwards CR conditions verbatim and does not consult spec.operatingMode, even though it already writes operatingMode on stop/start.

Live reproduction of the trigger (Agent Sandbox v0.5.0, v1beta1, raw CR only — no OpenShell gateway involved, in an isolated scratch namespace):

• create (operatingMode: Running) → Ready=True (DependenciesReady), no Suspended
• patch → Suspended → Ready=False (SandboxSuspended), Suspended=True (PodTerminated), pod terminating
• patch → Running → Ready=True (DependenciesReady) and stale Suspended=True (PodTerminated), pod Running 1/1, spec.operatingMode=Running

This confirms the stale condition is an upstream Agent Sandbox controller behavior (it does not clear Suspended on resume), which OpenShell's v1beta1-agnostic phase derivation then mishandles. The report's claim that v1alpha1 is unaffected is consistent with v1alpha1 encoding desired state as spec.replicas with no lingering Suspended condition.

Affected components: openshell-server (gateway phase derivation) and openshell-driver-kubernetes (status mapping).

Duplicates: none found.

Impact Signals

• Affected users/scope: All operators running the gateway against Agent Sandbox v1beta1 (v0.5.0). Topology-independent (shared phase derivation). Any automation relying on stop/start for cost control or scheduling is broken on v1beta1.
• Regression: No — this is a pre-existing limitation exposed by adopting the v1beta1 API; v1alpha1 is unaffected.
• Workaround: Delete and recreate the sandbox, which discards preserved workspace/session state and defeats the purpose of stop/start. Effectively unavailable for the intended use case.
• Evidence quality: High — root cause confirmed by code trace and the external trigger reproduced live at the CR level.

Human Decision Required

Decide whether OpenShell should address this issue. If yes, apply state:accepted, associate it with a roadmap item, or do both, and decide whether the work remains human-owned. Either action records acceptance; roadmap placement additionally records sequencing. To queue investigation or planning for an unattended agent, also apply agent:plan-requested. You can instead directly ask an agent to use create-spike or build-from-issue on this issue; the agent will warn about missing expected workflow labels and continue without changing them. If no, close it as not planned and record the rationale.

[russellb]

Root cause traced to an upstream dependency bug

The stale Suspended condition originates in the Agent Sandbox controller (kubernetes-sigs/agent-sandbox), not in OpenShell.

• In controllers/sandbox_controller.go (v0.5.0), condition handling is asymmetric. meta.SetStatusCondition only adds/updates — it never removes a condition absent from the computed set. The controller explicitly clears the Finished condition when it no longer applies, but does nothing equivalent for Suspended: on resume, computeSuspendedCondition early-returns nil, so the previously-set Suspended=True (PodTerminated) persists alongside Ready=True. That violates metav1.Condition semantics, which require conditions to reflect current observed state.
• This is fixed upstream by feat: implement persistent Suspended conditions for Sandbox kubernetes-sigs/agent-sandbox#1150 ("implement persistent Suspended conditions"), which makes Suspended transition to False (NotSuspended) on resume instead of being left stale.
• Verified against the released controller source: the fix is absent in v0.5.0 (deployed when this was reported) and present in v0.5.6 (latest). Operators running a controller release with fix(server): pre-register prometheus metrics at startup #1150 will not produce the stale condition.

OpenShell fix

Even with the upstream fix available, the gateway should not depend on a particular controller version and its "Ready=True means usable" contract should be robust to contradictory/stale driver conditions. PR #2933 makes derive_phase treat Ready=True as authoritative — a Suspended=True condition only resolves to Stopped when Ready is not True. A genuine stop leaves Ready unset/False, so it still resolves to Stopped, and v1alpha1 is unaffected.

Reproduction (deterministic): starting from the Starting phase that start sets, a resumed snapshot with Ready=True (DependenciesReady) + stale Suspended=True (PodTerminated) reconciled to Starting (never Ready) on unfixed code; the fix advances it to Ready. Also reproduced live at the raw CR level on an Agent Sandbox v0.5.0 (v1beta1) cluster.

Recommendation for operators: upgrade the Agent Sandbox controller to a release containing kubernetes-sigs/agent-sandbox#1150; the OpenShell gateway fix in #2933 resolves the symptom regardless of controller version.