bug(vm-driver): a sandbox that was never stopped ends in unrecoverable `Error` after a gateway restart

[ACodingfreak]

User Story

I use OpenShell 0.1.2 with a local gateway and the VM compute driver (libkrun microVMs) on Ubuntu 24.04. I directly encountered this: after systemctl --user restart openshell-gateway, two of my three VM sandboxes ended in Error, and the only way out was to delete them.

This affects anyone who restarts or upgrades a local gateway that uses the VM driver: they can lose the sandbox and its /sandbox workspace.

Problem Statement

After the gateway restarts, a VM sandbox that has never been stopped and started since creation ends in Phase: Error, with this condition:

Ready: False (StartFailed) - Failed to start sandbox during gateway startup: VM sandbox is already running generation <64 hex characters>

The sandbox's VM is running again after the restart, but exec and SSH refuse the sandbox (sandbox '<name>' is not ready (phase: Error)). A sandbox that has been stopped and started at least once comes back Ready after the same restart.

Impact / Why This Matters

Any gateway restart does: systemctl --user restart openshell-gateway, a package upgrade, a gateway.toml change. A host reboot is expected to trigger it too.

• There's no recovery path. openshell sandbox stop refuses (sandbox must be Ready to stop), and so does openshell sandbox start (sandbox must be Stopped, Completed, or a failed main-process Error to start). Later gateway restarts don't retry the sandbox. The only way out is sandbox delete, which discards /sandbox. Getting files back means copying the sandbox's overlay.ext4 and extracting from it by hand.
• The workaround: stop and start every new VM sandbox once, right after creating it. That works (tested), but nothing in the CLI or docs tells users to do it.

Acceptance Criteria

• A VM sandbox that was created and never stopped returns to Ready after a gateway restart, and openshell sandbox exec works in it.
• A sandbox that a gateway restart has already put in Error with reason StartFailed can be recovered without deleting it and without losing /sandbox

Reproduction Steps

Prerequisites: the OpenShell 0.1.2 .deb, a local gateway with OPENSHELL_COMPUTE_DRIVER=vm running as the systemd user unit openshell-gateway, and KVM.

1. Create two sandboxes and wait for both to be Ready:

   openshell sandbox create --name a --detach -- sleep infinity
   openshell sandbox create --name b --detach -- sleep infinity
   openshell sandbox list

2. Stop and start only b:

   openshell sandbox stop b && openshell sandbox start b

3. Restart the gateway:

   systemctl --user restart openshell-gateway

4. After about 30 seconds, a is Error and b is Ready:

   openshell sandbox list
   openshell sandbox get a | grep StartFailed

5. a can't be recovered:

   openshell sandbox stop a     # sandbox must be Ready to stop (current phase: Error)
   openshell sandbox start a    # sandbox must be Stopped, Completed, or a failed main-process Error to start

This also reproduces every time on a separate, throwaway gateway instance, for example a second VM-driver gateway on another bind_address with its own XDG_CONFIG_HOME and XDG_STATE_HOME, run under systemd-run --user and restarted with systemctl --user stop/start.

Suggested UX (if applicable)

No response

Environment

• OpenShell: 0.1.2 (openshell --version: openshell 0.1.2; Debian package openshell 0.1.2-1).
• OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-139-generic, x86_64, KVM.
• Deployment: local gateway installed from the .deb, running as the systemd user unit openshell-gateway (KillMode=control-group) on 127.0.0.1:17670. Compute driver vm (/usr/libexec/openshell/openshell-driver-vm, libkrun).

Logs

Gateway journal around the restart (`journalctl --user -u openshell-gateway`):


WARN openshell_server::compute: Failed to stop sandbox during gateway shutdown sandbox_id="fe76c66f-…" sandbox_name="vm1" error=code: 'The service is currently unavailable', …
WARN openshell_server::compute: Failed to start sandbox during gateway startup sandbox_id=fe76c66f-… sandbox_name=vm1 error=code: 'The system is not in a state required for the operation's execution', …
INFO openshell_server::compute: Started sandbox during gateway startup sandbox_id=… sandbox_name=dev phase=Provisioning recovered=false



$ openshell sandbox get vm1
  Phase: Error
  Conditions:
    - Ready: False (StartFailed) - Failed to start sandbox during gateway startup: VM sandbox is already running generation d925a7260e8c96f01c9c9f277c456f0a97316c83b567d479785042d4a03309c2
$ openshell sandbox stop vm1
Error:   × … message: "sandbox must be Ready to stop (current phase: Error)"
$ openshell sandbox start vm1
Error:   × … message: "sandbox must be Stopped, Completed, or a failed main-process Error to start (current phase: Error)"

[nkch1k]

Reproduced on 0.1.2.

Cause, from reading the source: on first launch the VM driver records a marker that the gateway's later StartSandbox can never match.

• First launch. provision_sandbox_inner finds no boundary-generation file, falls back to random_boundary_token() (64 hex characters) and persists that: driver.rs#L1381-L1393, #L1608-L1613.
• Gateway restart. The driver relaunches the VM from persisted state. The gateway's start sweep then sends StartSandbox with the sandbox's runtime generation, which for a sandbox that was never restarted is still the UUID assigned at creation: compute/mod.rs#L3187, sandbox_session.rs#L104-L107.
• The rejection. start_sandbox sees the sandbox running, compares the marker with the requested generation and fails: driver.rs#L1912-L1926. The 64-hex "generation" in the error above is the driver's token, so it can never equal a gateway generation.
• Why one stop/start cures it. An explicit start overwrites the marker with the gateway's generation: driver.rs#L1956-L1958.

No recovery follows because the sweep marks the sandbox StartFailed (compute/mod.rs#L3361-L3371), is_recoverable_error_reason only retries ContainerRuntimeRestart and ContainerStopped (#L6811-L6818), and sandbox start rejects that state.

The same code is in v0.1.2 and on main at be7af99ff. Both the fallback and the comparison came in with #2942.

Proposed fix, in the driver:

1. On first launch, record the generation from the launch authentication instead of a random token. The boundary spec a few lines below already uses it (driver.rs#L1578-L1581).
2. In start_sandbox, when the marker does not match, also accept the generation recorded in the persisted launch authentication. This covers sandboxes created before the fix, which matter most because an upgrade is itself a gateway restart.

Both changes are on a local branch with unit tests: nkch1k@9ec8a59. The new test as expected fails on main with the error from this issue.

I have a vouch request open (#4211), so there is no PR yet, I'd be happy to open it when my vouch will be approved

Open question for the second acceptance criterion: StartFailed is only ever set by the startup sweep, so adding it to is_recoverable_error_reason would let the next gateway start retry those sandboxes. That changes behavior for every driver, so I would leave it out unless you want it in the same fix.