[fjvbn2003]

Description

When a node has a pre-installed driver (driver.enabled=false) and nvidia-smi on the host cannot talk to the driver, for example after the driver packages were upgraded without a reboot, the driver-validation init container logs:

Attempting to validate a pre-installed driver on the host
Failed to initialize NVML: Driver/library version mismatch
No pre-installed driver detected on the host: exit status 18
Validating containerized driver installation
Attempting to validate a driver container installation
failed to validate the driver, retrying after 5 seconds\n
failed to validate the driver, retrying after 5 seconds\n
...

and stays in Init forever. Both log lines point away from the problem. A driver is installed on the host, and the retry warning never says what failed (it also prints a literal \n). #1276 and #1687 show users stuck on the same retry line without a reason.

This PR:

• validateHostDriver: when nvidia-smi was found on the host but failed, return found '<path>' on the host, but running it failed: <err>.
• runValidation: log No usable pre-installed driver detected on the host: ... instead of No pre-installed driver detected ....
• validateDriverContainer: include the error in the retry warning and drop the trailing \n.

Control flow does not change. Validation still falls through to the driver container path, which is still needed when the host has a stray nvidia-smi but the driver comes from the driver container.

This overlaps with #3034 on the retry warning line. If #3034 merges first, I will rebase and move the error into the warning in waitForDriver.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint) (golangci-lint not available locally; gofmt -s and go vet with GOOS=linux are clean)
• Generated assets in-sync (make validate-generated-assets) (no generated files changed)
• Go mod artifacts in-sync (make validate-modules) (no module changes)
• Test cases are added for new code paths (log/error text only; validateHostDriver chroots into /host, so it is not unit-testable without a refactor)

Testing

• go test ./cmd/nvidia-validator/ passes.
• On a cluster running GPU Operator v26.3.3, I ran the validator on a GPU node in a temporary pod with the nvidia-operator-validator service account. /host was a fake root whose /usr/bin/nvidia-smi prints Failed to initialize NVML: Driver/library version mismatch and exits with 18. The run used WITH_WAIT=true, --component driver, and a 12s timeout, and I ran it once with the main binary and once with the patched binary, both built from the same commit (CGO_ENABLED=0 GOOS=linux).

Before (main):

level=info msg="No pre-installed driver detected on the host: exit status 18"
level=warning msg="failed to validate the driver, retrying after 5 seconds\n"

After:

level=info msg="No usable pre-installed driver detected on the host: found '/usr/bin/nvidia-smi' on the host, but running it failed: exit status 18"
level=warning msg="failed to validate the driver, retrying after 5 seconds: failed to locate driver libraries: error locating \"libnvidia-ml.so.1\""

This change was drafted with AI assistance and reviewed by me.

Fixes #3085

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Host driver validation errors now identify the resolved nvidia-smi path and wrap the command error. Driver-container retry warnings include the validation error. The fallback log refers to the absence of a usable pre-installed driver.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: 🔵 Low · up to f68b4

Operator-validation retry warnings omit why host nvidia-smi failed, although that error is logged earlier. Include both errors to make the retry diagnostic actionable.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments: 1

cmd/nvidia-validator/main.go: 1

803-803: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Carry the host validation error into the retry warning.

With WITH_WAIT=true, this warning formats the separate container-validation error. Driver.runValidation has the host nvidia-smi execution error, but the retry path does not receive it. This omits the underlying host error required by issue #3085. Pass the host error into the container retry path and include both errors in the warning.

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/gpu-operator/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 36d15131-7599-43d7-a71c-706a25d4eff3

📥 Commits

Reviewing files that changed from the base of the PR and between dbb11e8 and f68b443.

📒 Files selected for processing: 1

• cmd/nvidia-validator/main.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[fjvbn2003]

@coderabbitai Thanks. The host error is not lost: runValidation logs it once, right before it falls back to the containerized check, so with this change the log reads

No usable pre-installed driver detected on the host: found '/usr/bin/nvidia-smi' on the host, but running it failed: exit status 18
Validating containerized driver installation
failed to validate the driver, retrying after 5 seconds: failed to locate driver libraries: error locating "libnvidia-ml.so.1"

(measured with a fake nvidia-smi that exits 18, WITH_WAIT=true). Repeating the host error on every 5-second retry would add the same line forever, so I kept it in the one-time message. Happy to carry it into the retry warning as well if the maintainers prefer that.

This reply was drafted with AI assistance and reviewed by me.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@fjvbn2003 The supplied log shows that runValidation preserves the host error in the one-time fallback message. The retry warning then reports the containerized validation error. Repeating the host error on every retry is unnecessary. I withdraw the concern.

---

✏️ Learnings added

Learnt from: fjvbn2003
URL: https://github.com/NVIDIA/gpu-operator/pull/3086

Timestamp: 2026-10-16T15:57:40.487Z
Learning: In NVIDIA/gpu-operator, cmd/nvidia-validator/main.go uses Driver.runValidation to log the host-driver validation error once before falling back to containerized driver validation. Retry warnings report the containerized validation error. Do not require the host error to be repeated in every retry warning; the one-time fallback message retains that diagnostic.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.