Repository navigation
feat: Use CDI for GPU injection instead of nvidia-container-cli #398
Description
Activity
Thanks @klueska. This is definitely the direction we would want to move this so as to not depend on a specific runtime / k8s implementation. Using CDI as the mechanism for defining what
-
As of the NVIDIA Container Toolkit v1.18.0 the CDI specs for available devices SHOULD be generated automatically when installing the NVIDIA Container Toolkit. This should remove the need for explicit CDI spec generation. With that said, we may want to consider a situation where a sandbox is only given access to a subset of the GPUs so as to allow at least minimal isolation between multiple agents. Here the important thing is to separate CDI spec generation from consumption to allow for flexibility when defining new use cases. For example, if we are starting a REMOTE sandbox.
-
Although CDI support is enabled in docker as of Docker 25, it was an opt-in feature up to 28.2.0 -- where it was enabled by default. The logic for injection was not changed significantly between these versions.
-
The containerd config option that needs to be enabled is the
enable_cdiplugin option. The exact config path depends on the containerd version / CRI plugin used. Furthermore, it is not available in v1.6 of container, is disabled by default in v1.7, and enabled by default as of v2.0. The option is also marked as deprecated in the current HEAD. -
The default paths for CDI specs are
/etc/cdiand/var/run/cdi, with specifications in/var/run/cditaking precedence. As such we would also have to mount/var/run/cdi` to get access to the specs there. Also note that the device plugin (when CDI is activated) generates CDI specs itself -- assuming a containerized node these would have to be scoped to the container and not the host.
-
Thanks @elezar for the clarifications.
Regarding your first bullet -- This sounds like a host-level dependency that OpenShell would need to clarify in its README. From my understanding of how OpenShell works, I think the k3s container would still want access to all GPUs, and then each of the agent sandboxes it orchestrates would (potentially) want limited access to a subset of the GPUs (which would be made possible via the device plugin).
Regarding your fourth bullet -- I guess this means we don't actually need to inject these folders at all. Putting the device plugin into CDI mode would generate the specs on the fly so that these folders (and any generated specs) would be created on the fly.
Depending on the version of k8s in use, it might also make sense to use the DRA driver for GPUs instead of the device plugin so that controlled GPU sharing could be introduced more easily in the future. Also, it uses CDI natively, so no need to "configure" it in CDI mode.
Regarding your fourth bullet -- I guess this means we don't actually need to inject these folders at all. Putting the device plugin into CDI mode would generate the specs on the fly so that these folders (and any generated specs) would be created on the fly.
Yes, I think we should separate giving the containerized node access to the driver and GPUs from ensuring that individual containers running on this containerized node (which I assume is each as sandbox).
Depending on the version of k8s in use, it might also make sense to use the DRA driver for GPUs instead of the device plugin so that controlled GPU sharing could be introduced more easily in the future. Also, it uses CDI natively, so no need to "configure" it in CDI mode.
I was also thinking of DRA in this context -- especially considering that the DRA driver doesn't have a dependency on the NVIDIA Container Toolkit. This would mean that it SHOULD be possible to simplify the node image and have the DRA driver generate CDI specs and provide the CDI hook as it does today.
Closing this as superseded.
This issue was originally filed against the older k3s-based GPU architecture. The GPU path has since moved to specific compute driver implementations, with CDI support handled in the relevant driver/runtime paths rather than as one broad k3s migration item.
The remaining CDI-related work is now tracked more specifically in:
- GPU-accelerated Sandbox Support #1444 for the GPU-accelerated sandbox roadmap
- feat(gpu): derive sandbox access requirements from CDI specs #1606 for deriving sandbox access requirements from selected CDI specs
Those issues better capture the current architecture and remaining work.
Reacted by Tyeth Gundry
Problem Statement
GPU access currently relies on the legacy
nvidia-container-runtime+nvidia-container-clistack at two layers: once when Docker injects GPUsinto the k3s cluster container, and again when the nvidia-device-plugin +
nvidia-container-runtimeinject them into individual sandbox pods.Proposed Design
Both layers should be migrated to CDI instead. The general idea:
nvidia-ctk cdi generateinto the k3s container:
--device nvidia.com/gpu=all/etc/cdiinto the k3s container, enableenable_cdi_devices = truein the containerd config, and configure the nvidia-device-plugin to use CDI
device IDs so containerd handles injection natively
CDI is the canonical way NVIDIA supports GPU access in containerized
environments going forward. Some platforms require CDI and are incompatible
with the legacy runtime stack, so this would also broaden the set of platforms
OpenShell can run on. It also makes what gets injected explicit and
auditable via the CDI spec rather than delegating to a CLI with broad host
access.
/cc @elezar @jgehrcke
Alternatives Considered
None
Agent Investigation
No response
Checklist