improve external compute-driver startup API and test coverage #2987

User Story

As an OpenShell operator or contributor, I want standalone compute drivers to share a clear startup contract, so that I can run gateway-external drivers consistently across Docker, Podman, VM, and CI environments.

Problem Statement

OpenShell supports external compute drivers through the gateway remote driver path: the gateway selects a configured driver name and connects to a Unix socket. However, the startup UX for standalone driver processes is not consistent across drivers.

The gateway-side contract is generic:

• select one compute driver
• configure [openshell.drivers.<driver>].socket_path
• connect to the external driver over the socket

The driver-side startup contract is not yet generic:

• Docker uses --bind-socket, --config, and --gateway-bind
• Podman uses --bind-socket or OPENSHELL_COMPUTE_DRIVER_SOCKET plus many individual flags/env vars
• VM has its own set of startup flags
• Kubernetes external-driver mode is expressed through Helm/image composition

This makes external-driver testing and CI artifact reuse harder than it should be. It also makes it unclear which options are part of the common external-driver API versus driver-specific runtime configuration.

Impact / Why This Matters

Today, each E2E harness has to know driver-specific launch details. That increases duplication and makes it easy for coverage to drift. In particular, restoring rootless Podman through Nix VM tooling exposed that the gateway-side external-driver model is generic, but the standalone driver launch UX is not.

This blocks a clean CI workflow where prepared artifacts can be composed predictably:

• driver-free gateway artifact
• standalone compute-driver artifact
• supervisor/sandbox artifact
• Nix VM guest runtime

This also leaves a narrower coverage gap after #2981 removed the original rootless Podman lane: rootless Podman with pasta, a driver-free gateway, and an external openshell-driver-podman process.

Related PR: #2981

Proposed Design

Define and implement a consistent startup UX for standalone compute drivers.

Externally observable behavior should include:

• Every standalone compute driver can bind its public gateway socket through the same option, for example --bind-socket.
• Every standalone compute driver can consume driver runtime configuration through a consistent mechanism, for example --config.
• Gateway callback/listener inputs have a consistent name and meaning across drivers.
• The gateway continues to select external mode through [openshell.drivers.<driver>].socket_path.
• E2E tooling can start an external driver without embedding one-off launch logic for each driver.

Use VM-backed rootless Podman testing as one validation path:

• Build or download a driver-free gateway artifact.
• Build or download openshell-driver-podman.
• Start both inside the Ubuntu 26.04 Nix test guest.
• Run the rootless Podman E2E suite against the external driver.

Acceptance Criteria

• Document the intended external compute-driver startup contract.
• Align standalone driver CLI options where practical.
• Preserve backward-compatible env vars or document any intentional migration.
• Add tests or launch checks proving the common contract works for at least Docker and Podman.
• Add CI artifact support for a driver-free gateway if needed.
• Add CI artifact support for standalone compute-driver binaries if needed.
• Add or update E2E coverage for rootless Podman using an external openshell-driver-podman inside the Nix VM.
• Update E2E docs with the external-driver workflow.

Alternatives Considered

Keep adding driver-specific branches to each E2E helper. This is simpler short-term, but it preserves the API ambiguity and makes future external-driver coverage harder to compose.

Treat rootless Podman external-driver coverage as only a test gap. That restores one lost CI scenario, but it does not address the underlying inconsistency exposed by the work.