[joaopaulosr95]

Summary

driver-validation probes the host for a pre-installed driver exactly once, before entering the driver-container wait loop. On a node where the driver ships with the OS image, that probe can run before the kernel modules have finished loading — kubelet starts the validator while the node's driver install is still in progress. The single failed probe then commits the validator to waiting for a driver container, and when the driver is not managed by the operator no such container is ever going to exist, so the pod stays in Init forever even though the host driver became usable seconds later.

In Driver.runValidation, validateHostDriver is called once and validateDriverContainer owns an unconditional retry loop, so there is no path back to the host check once the first probe has failed. Deleting the pod is the only recovery: the replacement re-runs the host probe against a by-then-loaded driver and proceeds normally.

How it shows up

Observed on EKS AL2023 NVIDIA AMIs, where nvidia-kmod-load.service takes 57–86s (it installs the kmod and regenerates the initramfs via dracut) and kubelet starts about 9s into it. On one node the validator container started at 08:05:27 and nvidia-kmod-load.service finished at 08:05:57, a 30s miss:

08:05:27 info  Attempting to validate a pre-installed driver on the host
         NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
08:05:27 info  No pre-installed driver detected on the host: exit status 9
08:05:27 info  Validating containerized driver installation
08:05:27 info  Attempting to validate a driver container installation
08:05:27 warn  failed to validate the driver, retrying after 5 seconds
... repeats indefinitely

The host driver is fully healthy the whole time the pod waits — nvidia-smi on the node reports the GPU, all /dev/nvidia* nodes are present, and nvidia-persistenced is running. /run/nvidia/driver is empty, as expected on a node with no driver container.

Reproduces on any GPU node where the driver is installed by the OS image rather than by the operator (nvidia.com/gpu.deploy.driver=false), and loading the modules takes longer than kubelet's start-to-first-pod time. Environment where it was observed:

• GPU Operator Version: v26.3.3 (the code path is unchanged on v26.7.1 and main)
• OS: Amazon Linux 2023.12.20260611 (EKS NVIDIA accelerated AMI)
• Kernel Version: 6.18.33-63.124.amzn2023.x86_64
• Container Runtime Version: containerd 2.2.4
• Kubernetes Distro and Version: EKS v1.36.2

The change

The wait loop moves out of validateDriverContainer into a new waitForDriver, which retries the container and re-probes the host on each iteration, so a driver that becomes usable after startup is picked up. validateDriverContainer is reduced to a single attempt. The driver container readiness check (.driver-ctr-ready) is also a single attempt now. Previously, with WITH_WAIT=true, it blocked until the file appeared. On a node where a driver DaemonSet exists in the cluster but doesn't run on that node (e.g. driver.enabled=true with the node labelled nvidia.com/gpu.deploy.driver=false), the file never appears, so the host would never have been re-probed. Retried host probes are silent, to avoid repeating the same nvidia-smi failure output every interval.

Behaviour is otherwise unchanged: the first host probe keeps its existing logging and fast path, isDriverManagedByOperator is still resolved once, and the non-waiting path returns the same wrapped error.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Unit test Test_waitForDriver covers four cases, including the regression one: the container probe failing while the host driver appears late. I verified it fails without the fix — the container probe is given a finite failure count, so removing the re-probe makes the case fail on its assertion rather than hang.

Test_waitForDriver_driverManagedByOperatorWithLateHostDriver runs the real validateDriverContainer with the driver marked as operator-managed and waiting enabled, so the readiness check is exercised for real. A timeout guard turns a regression into a failure instead of a hang. It fails against the previous blocking readiness check.

make unit-test passes. make lint reports zero issues in cmd/nvidia-validator/. E2E was not run, since the holodeck suite needs NVIDIA infrastructure; ./tests/e2e/ and ./tests/e2e/suites/ were compile-checked and go vet is clean.

---

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The validator now separates driver-container checks from retry handling. When waiting is enabled, it retries container validation and probes the host after each failure. If the host probe succeeds, validation returns host-driver information. Tests cover container readiness, host detection, and the error path when waiting is disabled.

---

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to c761a

The new host re-probe may not run on nodes where the operator manages the driver DaemonSet but the node uses an OS-image driver, so those nodes can still wait indefinitely. The impact is limited to that configuration, so this is mergeable with awareness or a small follow-up.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

cmd/nvidia-validator/main.go (1)

801-813: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep the driver-container readiness check non-blocking inside waitForDriver.

When withWaitFlag is true, validateDriverContainer calls assertDriverContainerReady, which enters runCommandWithWait and does not return until .driver-ctr-ready exists. waitForDriver can call probeHost only after probeContainer returns, so host re-probing is skipped on this path. isDriverManagedByOperator checks for any matching DaemonSet in the namespace, not whether it selects this node, so an OS-image-driver node can enter this path while the operator DaemonSet runs elsewhere.

Suggested fix

-func assertDriverContainerReady(silent bool) error {
+func assertDriverContainerReady(silent, wait bool) error {
 	command := shell
 	args := []string{"-c", fmt.Sprintf("stat %s", driverContainerStatusFilePath)}
-	if withWaitFlag {
+	if wait {
 		return runCommandWithWait(command, args, sleepIntervalSecondsFlag, silent)
 	}

Then call assertDriverContainerReady(silent, false) from validateDriverContainer.

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/gpu-operator/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: da31415b-1c95-401e-ab24-f5d0f921ad7a

📥 Commits

Reviewing files that changed from the base of the PR and between 57378e9 and c761af0.

📒 Files selected for processing (2)

• cmd/nvidia-validator/main.go
• cmd/nvidia-validator/main_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.