Repository navigation
fix(gpu): select one CDI GPU by default for Docker and Podman #1477
Description
Activity
- added a parent issue
on May 20, 2026 🏗️ build-plan
Implementation Plan
Issue type:
fix
Complexity: Medium
Confidence: High - Docker uses daemon inventory and Podman local Linux
inventory is derived directly from/dev/nvidiaN; remote Podman inventory is
explicitly out of scope.Summary
Update local-container GPU selection so a bare
--gpurequest resolves to one
CDI GPU device by default instead ofnvidia.com/gpu=all, using
driver-owned round-robin selection to rotate default requests across available
IDs without tracking active ownership. Docker and Podman share CDI selection
helpers, while inventory collection stays driver-specific: Docker uses
daemon-reported CDI inventory, and local Linux Podman maps/dev/nvidiaN
nodes tonvidia.com/gpu=N.nvidia.com/gpu=allremains an intentional
compatibility path for all-only runtimes. Docker indexed IDs and UUID-style
IDs are treated as alternate naming families for the same device pool and are
not mixed.Scope
crates/openshell-core/src/gpu.rs: Add shared CDI GPU inventory
normalization, fallible default-selection, and concurrency-safe round-robin
cursor helpers.crates/openshell-driver-docker/src/lib.rs: Capture Docker daemon CDI GPU
inventory fromdocker.info()and use it when building CDI device requests.
Treat missing, null, empty, or non-GPU Docker inventory as a clear
precondition error for bare--gpuwhile preserving explicit
--gpu-devicepass-through when Docker CDI support is enabled. Normalize
mixed indexed/UUID-style CDI IDs to one available naming family, preferring
indexed IDs when present. Add a Docker-owned round-robin cursor and ensure
validation/preview paths do not advance it.crates/openshell-driver-docker/src/tests.rs: Cover individual default
selection, missing/null inventory failure, all-only fallback, known-empty
failure, mixed indexed/UUID-style normalization, UUID-style-only selection,
cursor advancement/wrapping, validation no-advance behavior, and explicit
device pass-through.crates/openshell-driver-podman/src/driver.rsand
crates/openshell-driver-podman/src/container.rs: Enumerate local
/dev/nvidiaNnodes, map them tonvidia.com/gpu=N, carry that inventory
into a driver-owned round-robin cursor and a fallible inventory-aware
container spec builder. The inventory helper should be unit-testable with an
injectable device root and ignore non-numeric NVIDIA control/helper nodes.crates/openshell-driver-podman/src/container.rs/driver tests: Cover
local/dev/nvidiaNinventory mapping, empty-inventory rejection, default
selection, cursor advancement/wrapping, validation no-advance behavior, and
explicit pass-through.e2e/rust/tests/gpu_device_selection.rs: Update optional GPU e2e
expectations so Docker uses daemon CDI inventory and Podman uses local
/dev/nvidiaNinventory for the default device.- Docker/Podman docs and READMEs: Document the default, all-only
compatibility path, missing-inventory failure behavior, and--gpu-device
override behavior.
Implementation Steps
- Add shared CDI GPU selection, round-robin cursor, and tests in
openshell-core. - Wire Docker daemon discovered-device inventory into Docker device request
construction, normalizing indexed and UUID-style aliases into one available
naming family, and add Docker driver-owned round-robin selection. - Add Podman local
/dev/nvidiaNinventory enumeration and Podman
driver-owned round-robin selection. - Update Docker and Podman unit tests for defaults, all-only compatibility,
Docker mixed-name-family normalization, explicit pass-through,
cursor advancement/wrapping, validation no-advance behavior, Docker's
missing/empty inventory failure paths, and Podman's local empty-inventory
rejection. - Update optional GPU e2e expectations for Docker daemon inventory and
Podman local/dev/nvidiaNinventory. - Update docs.
Test Plan
- Unit tests:
openshell-core, Docker driver tests, and Podman
driver/container tests. - Integration tests: N/A - behavior is contained in driver spec/request
builders. - E2E tests: Update existing GPU e2e tests. Run
mise run e2e:docker:gpuandmise run e2e:podman:gpuonly in a
CDI-capable GPU environment; otherwise note the hardware gate.
Risks & Constraints
- Remote Podman does not currently report available CDI devices. For this
task, Podman inventory is local Linux only via/dev/nvidiaN; future Podman
remote CDI reporting can replace or augment that later. - Missing Docker
DiscoveredDevices, known-empty Docker inventory, and empty
local Podman inventory fail clearly for bare default requests instead of
falling back. - All-only CDI runtimes must keep working through the compatibility fallback.
GetCapabilitiesResponse.gpu_countstays0; this task does not implement
public count reporting.- This is not allocation. The cursor can wrap to a CDI ID still used by an
existing sandbox, and separate gateway/driver processes can choose the same
ID.
Documentation Impact
- Update Docker/Podman GPU docs and driver READMEs.
- No
docs/reference/gateway-config.mdxupdate expected. - LSM compatibility: no expected SELinux/AppArmor behavior change.
Revision 1 - initial plan
🏗️ build-from-issue-agent
Implementation Complete
PR: #1675
What was built
Docker and Podman now use driver-owned CDI GPU inventory to select one default NVIDIA CDI GPU for bare
--gpurequests. Explicit--gpu-devicevalues continue to pass through unchanged.Tests
- Unit: Core GPU inventory/round-robin tests; Docker inventory/default selection tests; Podman local inventory/default selection tests.
- E2E: GPU device selection expectations updated and the GPU e2e target compiled with
--no-run. - Pre-commit:
mise run pre-commitpassed.
GPU e2e execution was environment-blocked locally: Docker reports only
docker.com/gpu=webgpu, Podman is not installed, and/dev/nvidia0is absent.Docs updated
docs/sandboxes/manage-sandboxes.mdxdocs/reference/sandbox-compute-drivers.mdxcrates/openshell-driver-docker/README.mdcrates/openshell-driver-podman/README.md
The issue will auto-close when the PR is merged.
Metadata
Metadata
Assignees
Type
Projects
- Status
Description
Update Docker and Podman GPU sandbox defaults so
--gpuprefers one CDI GPU device instead of defaulting tonvidia.com/gpu=all.This is part of the GPU roadmap in #1444.
--gpumeans the active driver's default GPU behavior, and for GPU-enabled drivers that default should inject or allocate one suitable GPU when the runtime supports individual device selection.Context
Parent roadmap: #1444
Current local-container behavior maps a GPU request with no explicit
gpu_devicetonvidia.com/gpu=allthrough the shared CDI helper. That makes Docker and Podman inconsistent with Kubernetes and VM behavior, where a default GPU request maps to one GPU.Docker has priority for implementation because OpenShell's Docker GPU path and CDI discovery are more mature today. Podman should be handled in the same task, but may require additional runtime support or an out-of-band CDI device discovery path. Upstream Podman behavior such as podman-container-tools/podman#28712 may be relevant.
Proposed Scope
--gpubehavior to prefer one CDI GPU device instead ofnvidia.com/gpu=all.--gpubehavior to prefer one CDI GPU device instead ofnvidia.com/gpu=all.--gpu-devicebehavior as a driver-native advanced option.Target Behavior
Default GPU selection should use this order:
nvidia.com/gpu=0.nvidia.com/gpu=all, such as some WSL2-based setups, usenvidia.com/gpu=allas a compatibility fallback.Additional behavior:
openshell sandbox create --gpu ...on Docker injects one CDI GPU device when individual device selection is available.openshell sandbox create --gpu ...on Podman injects one CDI GPU device when individual device selection is available.openshell sandbox create --gpu --gpu-device nvidia.com/gpu=0 ...continues to pass the explicit CDI device ID through.nvidia.com/gpu=allshould be intentional and documented, not the default for platforms with individual device selection.gpu_countremains unsupported unless a driver explicitly implements count-based allocation.Out of Scope
This task fixes default GPU device selection cardinality. It does not require OpenShell to track active GPU assignments or prevent two OpenShell sandboxes from selecting the same default GPU.
If multiple sandboxes are created concurrently, selecting the same default fallback device is acceptable until a separate allocation/exclusivity task is implemented.
Open Questions
Definition of Done
--gpuprefers one individual CDI GPU device when available.--gpuprefers one individual CDI GPU device when available.nvidia.com/gpu=0.nvidia.com/gpu=allremains available as a documented compatibility fallback.--gpu-devicepass-through behavior is preserved for Docker and Podman.--gpu-deviceas an advanced driver-native option.