Repository navigation
bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343
Description
Activity
📋 triage-agent
Triage Assessment
Classification: needs-investigation
Summary
The report has substantive diagnostics and strong controls: host CUDA and plain Docker CDI pass, while an OpenShell GPU sandbox using the same injected driver library reaches
cuDriverGetVersionbut segfaults incuInit(0). I do not see this as a duplicate of #1486 because #1486 covered the generic/procand static GPU baseline failure; this report says the/proc/.../commwrite and Ubuntu control path now pass.Investigation
crates/openshell-driver-docker/src/lib.rsselects Docker CDI devices and submitsDeviceRequest { driver: "cdi" }; it does not derive inner sandbox policy from the selected CDI spec.crates/openshell-sandbox/src/lib.rscurrently detects GPU presence via/dev/nvidiactlor/dev/dxg, adds static GPU paths, enumerates/dev/nvidia[0-9]*, and promotes/procto read-write for CUDA compatibility.architecture/security-policy.mddocuments that GPU baseline enrichment is path-based and startup-time, andarchitecture/plans/cdi-derived-gpu-filesystem-policy-issue.mdalready identifies the design gap: non-standard CDI layouts can inject correct devices, mounts, libraries, or groups into the outer container while the OpenShell inner workload still lacks the corresponding Landlock or process setup.- Existing GPU e2e coverage (
e2e/gpu/README.md,e2e/rust/tests/gpu/workloads.rs) validates the genericcuda-basicpath, but not this DGX OS 7.5 / GB300 / ARM64 stack or CDI-derived platform shapes.
Recommendation
Route as a spike for platform-specific GPU sandbox investigation. The spike should collect the selected CDI spec/container edits from DGX OS 7.5, compare the OpenShell effective filesystem/process setup against a passing plain Docker CDI container, and decide whether the fix belongs in CDI-derived sandbox requirement enrichment, CUDA runtime diagnostics, or a separate DGX OS compatibility path. Do not apply
state:agent-readyuntil a human approves the investigation or a concrete build issue.spike-investigation
Investigation Summary
#2343 looks like a concrete DGX OS 7.5 / GB300 reproducer for the broader CDI-derived sandbox requirement gap tracked in #1606, plus a smaller sandbox exec diagnostic issue.
The #1486 / #1522
/procfailure mode does not appear to be the whole explanation here. The reporter proves/proc/<pid>/task/<tid>/commwrites work,libcuda.so.1loads, andcuDriverGetVersionsucceeds beforecuInit(0)segfaults. The host and plain Docker CDI controls also pass with the same driver library and UID/GID, so the remaining boundary is the OpenShell inner sandbox setup on this DGX kernel/driver/CDI stack.Code Findings
crates/openshell-driver-docker/src/lib.rsdetects Docker CDI support fromInfo.CDISpecDirs, resolves opaque CDI GPU IDs throughresolve_gpu_cdi_devices, and passes them only to Docker asDeviceRequest { driver: "cdi" }.- The Docker create/start flow does not pass the selected CDI IDs or daemon
CDISpecDirsto the supervisor. After Docker applies CDI to the outer container, OpenShell has no supervisor-side source of truth for CDIcontainerEdits. crates/openshell-sandbox/src/lib.rsenriches the inner Landlock policy from a static GPU baseline:/dev/nvidiactl,/dev/nvidia-uvm,/dev/nvidia-uvm-tools,/dev/nvidia-modeset,/dev/dxg,/proc, plus/dev/nvidia[0-9]*discovered byenumerate_gpu_device_nodes.- That static baseline does not derive CDI
deviceNodes,mounts[].containerPath, oradditionalGIDs. It also does not enumerate paths such as/dev/nvidia-caps/*; the VM GPU init script does create/dev/nvidia-caps/nvidia-cap1andnvidia-cap2, which is a useful local signal that GPU platforms can need device nodes outside the container baseline. crates/openshell-supervisor-process/src/process.rsdrops privileges withinitgroups,setgid, andsetuid, but there is no path to merge CDIadditionalGIDsinto the sandbox workload credentials.crates/openshell-supervisor-process/src/ssh.rscollapses signaled SSH exec children to exit status1because both PTY and no-PTY waiters usestatus.and_then(|s| s.code()).unwrap_or(1). The entrypoint path preserves signals throughProcessStatus::code()as128 + signal, soopenshell sandbox execis currently less diagnostic for a SIGSEGV than the entrypoint path.
Most Likely Failure Class
The leading hypothesis is that DGX OS 7.5 CDI injects one or more required CUDA initialization resources that are present in the outer Docker CDI container but absent from the OpenShell inner policy or credentials. Candidate resource classes are:
- platform-specific device nodes, for example
nvidia-capsor DGX/NVSwitch/fabric-related nodes; - per-file driver/library bind mounts whose container destinations are not covered by the existing static baseline under Landlock mount-boundary behavior;
- CDI
additionalGIDs, if the selected spec requires supplemental groups not present after OpenShell drops to the sandbox user.
This fits the observed call boundary:
cuDriverGetVersioncan succeed with onlylibcuda.so.1, whilecuInit(0)can touch more devices, helper libraries, sockets, or driver paths.Recommended Next Diagnostics On A DGX Host
To confirm the exact missing resource, capture the selected CDI metadata and compare it with the OpenShell effective sandbox view:
docker info --format "{{json .CDISpecDirs}}" docker info --format "{{json .DiscoveredDevices}}" grep -R "nvidia.com/gpu" /etc/cdi /var/run/cdi 2>/dev/null
For the selected CDI spec, inspect only the relevant
containerEdits.deviceNodes,containerEdits.mounts, andcontainerEdits.additionalGIDsentries. Then compare plain Docker CDI versusopenshell sandbox execfor:find /dev -maxdepth 2 \( -name "nvidia*" -o -name "nv*" \) -ls 2>/dev/null LD_DEBUG=libs python3 -c "import ctypes; lib=ctypes.CDLL(\"libcuda.so.1\"); print(lib.cuDriverGetVersion); lib.cuInit(0)"
If available, a file-focused trace around
cuInitwould be even better:strace -f -o /tmp/cuinit.strace -e trace=file,process python3 -c "import ctypes; lib=ctypes.CDLL(\"libcuda.so.1\"); lib.cuInit(0)"Recommended Engineering Direction
- Treat bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343 as a hardware/platform validation case for feat(gpu): derive sandbox access requirements from CDI specs #1606: Docker should preserve
Info.CDISpecDirs, mount CDI specs supervisor-only, upload a versionedcdi-context.jsonaftercreate_containerand beforestart_container, and let the supervisor resolve selected CDI IDs into Landlock paths and supplemental groups. - The supervisor should derive device nodes as read-write paths, mount destinations as read-only by default, and
additionalGIDsas supplemental groups before privilege drop, while keeping fail-closed diagnostics for missing specs, unsafe paths, writable directory mounts, and custom policy conflicts. - Add a separate small fix in the supervisor SSH exec path so signal deaths are surfaced as
128 + signaland ideally as an SSHexit-signalor clear diagnostic. This would make a CUDA SIGSEGV show up as SIGSEGV/139 instead of a generic exit 1. - Extend hardware-free tests from feat(gpu): derive sandbox access requirements from CDI specs #1606 with a fixture that includes
nvidia-capsand per-file library mounts, then use this DGX host as real hardware validation before closing bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343.
Scope Assessment
- Root-cause fix complexity: High, already covered by feat(gpu): derive sandbox access requirements from CDI specs #1606 and currently
state:in-progress. - Diagnostic fix complexity: Low to medium, likely localized to
crates/openshell-supervisor-process/src/ssh.rsplus CLI/e2e expectations. - Confidence: Medium-high for the CDI-derived requirement class; low for the exact missing DGX resource until the selected CDI spec or trace is captured.
- LSM sensitivity: Yes. Changes touch Landlock filesystem rules,
/procbehavior, bind-mounted CDI specs, and supplemental groups. SELinux/AppArmor hosts may need explicit validation around Docker bind labels, procfs visibility, and group-mediated device access.
Recommendation: keep this issue open and review-ready as the DGX validation target for #1606, with the SSH exec signal-reporting fix either split out or included as the diagnostic half of this issue. Do not apply
state:agent-readyhere unless maintainers want a separate implementation thread from #1606.local-validation
I validated the reported CUDA init behavior on this local host and could not reproduce it here.
Host/control details:
- Host kernel/arch:
Linux 6.17.0-1017-aws,x86_64 - GPU through Docker CDI: NVIDIA L4,
nvidia.com/gpu=0,nvidia.com/gpu=GPU-a6d5342f-d3d5-a14a-24d0-e92cad8530e8,nvidia.com/gpu=all - Docker CDI spec dirs:
/etc/cdi,/run/cdi - Plain Docker CDI CUDA workload passed with
OPENSHELL_GPU_WORKLOAD_SUCCESS cuda-basic. mise run e2e:docker:gpupassed: 10/10 tests, includingworkloads::gpu_workload_manifest_runs_expected_workloads.
Exact Python/ctypes probe results:
# plain Docker CDI, OpenShell base image cuDriverGetVersion 0 13000 cuInit(0)=0 # OpenShell Docker-backed GPU sandbox initial command cuDriverGetVersion 0 13000 cuInit(0)=0 # OpenShell Docker-backed GPU sandbox via sandbox exec cuDriverGetVersion 0 13000 cuInit(0)=0This confirms the failure is not reproducible on the local AWS x86_64 / NVIDIA L4 / driver API 13000 path. The issue still looks DGX OS 7.5 / GB300 / ARM64 specific, or at least specific to that kernel/driver/CDI platform shape.
- Host kernel/arch:
New root-cause evidence from NVIDIA/NemoClaw#7108 narrows this from the broader CDI hypotheses to sysfs visibility through Landlock.
On a physical DGX OS 7.5 / GB300 Station, released OpenShell 0.0.85 passes
cuInit(0), CUDA device count, and GB300 identification when the generated NemoClaw direct-GPU filesystem policy adds/systoread_only. No patched or custom OpenShell binary is required.At the current PR head, the change is applied by
buildDirectGpuPolicyYaml()only when the agent sandbox has direct GPU enabled; CPU-only sandboxes are unchanged:
https://github.com/NVIDIA/NemoClaw/blob/58c11e9d687231baad2debb8e48dd1e6d9c281c8/src/lib/onboard/initial-policy.ts#L38-L66An equivalent GPU-only, read-only
/sysrule in the OpenShell default GPU policy enrichment could therefore benefit all OpenShell GPU consumers, not just NemoClaw. We are also preparing an independent clean-host repetition of the final PR head.- added a commit that references this issue
on Jul 17, 2026 @elezar can i work on this
Additional context from the merged NemoClaw PR NVIDIA/NemoClaw#7108: this now looks like a sysfs visibility gap in the OpenShell inner Landlock policy, not a CDI-spec parsing issue.
The important nuance is that NVIDIA CDI specs do not generally enumerate the sysfs topology paths that CUDA/NVML may read during initialization. Docker CDI can still pass because the ordinary container view leaves enough of
/sysreadable. OpenShell then applies a stricter inner Landlock policy, so those implicit host sysfs reads become explicit sandbox policy requirements.The merged NemoClaw fix narrowed the workaround from broad read-only
/systo a Station GB300 direct-GPU allowlist:- exact read-only
/sys/bus/pci/devices/<BDF>entries for discovered NVIDIA display-class PCI devices; - existing read-only
/sys/devices/system/cpu,/sys/devices/system/memory, and/sys/devices/system/node; - existing read-only
/sys/module/nvidia/initstateand/sys/module/nvidia_uvm/initstate.
They explicitly avoid adding broad
/sys,/sys/bus/pci/devices, PCI parent subtrees, or any sysfs write access.NemoClaw reports an A/B validation on physical DGX OS / GB300: without the sysfs exception, sandbox CUDA init failed; with the exact BDF plus CPU/memory/NUMA/module-init read-only paths,
cuInit(0)passed, while unrelated sysfs paths such as/sys/fs/cgroup/cgroup.controllersand/sys/class/net/lo/addressremained denied.This suggests the OpenShell-side fix should be a separate GPU runtime/sysfs enrichment, not a relaxation of CDI path validation and not an expectation that
/syswill appear in CDI. A conservative first implementation could mirror the NemoClaw allowlist for selected NVIDIA GPU PCI BDFs and the small shared topology/module paths, then validate on this DGX OS 7.5 / GB300 reproducer.- exact read-only
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
Description
On a DGX Station GB300 running DGX OS 7.5, a GPU-enabled OpenShell 0.0.85
sandbox can load the injected driver library and call
cuDriverGetVersion, butcuInit(0)segfaults. The same library and GPU pass on the host and in ordinaryDocker CDI containers, including as the sandbox UID/GID.
This blocks qualification of DGX OS 7.5 for the NemoClaw Station golden path.
The existing generic Ubuntu path remains the supported control.
Related: #1486
Environment
7.5.0-GB300ws-GB200ws/ Ubuntu 24.04.46.17.0-1014-nvidia-64k595.58.03cuDriverGetVersion=1302029.2.11.19.00.0.85b47c81e0cc2b3dc22e07921167e48ad0a4ff7c1cReproduction
In a GPU-enabled OpenShell sandbox:
Observed during fresh onboarding and in a separate bounded diagnostic:
The isolated diagnostic produced:
Controls
All of the following passed with
cuInit(0)=0:998:998;commit and OpenShell 0.0.85 (kernel
6.8.0-124-generic, driver610.43.02).The DGX OS host and sandbox used the same injected driver file:
The managed vLLM container, direct inference, routed agent inference, exact
response, tool-use artifact, restart/reuse, and ECC checks all passed. This
narrows the failure to the OpenShell sandbox boundary on the DGX OS
kernel/driver stack; the current evidence does not isolate kernel versus driver.
Expected Behavior
A GPU-enabled sandbox should complete
cuInit(0)when the host and equivalentplain CDI container pass. At minimum, the sandbox must fail deterministically
with an actionable diagnostic rather than continuing with CUDA marked
unverified after a segmentation fault.