[jramkissoon]

Description

Fixes #2979

When draDriver.computeDomains.enabled is true, the nvidia-dra-driver-kubelet-plugin
DaemonSet runs two containers, gpus and compute-domains, each binding a gRPC
health service that the startup and liveness probes dial. The containers share
the pod's network namespace, so if both are configured with the same port the
second bind() fails with EADDRINUSE. The plugin treats a failed health
listener as fatal (startHealthcheck returns the error, NewDriver fails, the
process exits), so the losing container crash-loops with
listen tcp :<port>: bind: address already in use. Which container loses is a
startup race, so the symptom is nondeterministic and hard to trace back to the
GPUCluster spec.

This change validates the ports in the DRA driver state before rendering and
returns an error instead of producing a broken DaemonSet:

• validateKubeletPluginPorts in internal/state/dra_driver.go rejects a
  GPUCluster whose kubelet-plugin containers would bind the same TCP port. It
  runs on the resolved ports, so setting only one field to the other
  container's default (51516 / 51515) is caught too. Disabled health services
  resolve to -1 and are skipped, so two disabled sides are not a collision.
• The compute-domains listeners are only considered when
  IsComputeDomainsEnabled(); with it off the second container is not
  rendered.
• The error names both colliding listeners and the port, and surfaces through
  the existing getManifestObjects path: the GPUCluster gets a
  ReconcileFailed condition and the DaemonSet is left untouched, the same way
  an invalid image reference is handled today.

Scope note: the third commit also covers the containers' hardcoded metrics
endpoints (HTTP_ENDPOINT on :8080 and :8081), since a healthcheck port set to
either fails the same way, and the gpus case applies even with computeDomains
disabled. That goes beyond the issue text and is kept as a separate commit so
it can be dropped if you prefer the narrower scope.

A CEL rule was considered and rejected: the defaults live in Go, not in the CRD
schema, so CEL cannot see the single-field collision.

AI assistance: the analysis, implementation and tests were produced with help
from an AI coding assistant (Claude). I have reviewed and can defend every line.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Unit tests. TestDRADriverHealthcheckPortCollision in
internal/state/dra_driver_test.go drives getManifestObjects through ten
cases: equal explicit ports, one port set to the other container's default,
distinct ports, shipped defaults, computeDomains disabled, one and both health
services disabled, and healthcheck ports on 8080 / 8081 with computeDomains on
and off. Against the pre-fix dra_driver.go exactly the error cases fail and
the happy paths pass. go test ./internal/state/ passes.

Reproduction and verification on a cluster. Done on a two-node kind cluster
(Kubernetes 1.34, no GPUs) using NVIDIA's nvml-mock through the run-ai
fake-gpu-operator's mock backend, so both kubelet-plugin containers actually
start and reach their health bind. The overlays, scripts and captured evidence
are on my fork and are not part of this PR:

https://github.com/jramkissoon/nvidia-gpu-operator/tree/gpu-operator-issue-%232979/deploy/issue-test-bed

• scripts/issue-replication.sh deploys a GPUCluster with
  computeDomains.kubeletPlugin.healthcheck.port: 51516 against the release
  operator (v26.7.1). The DaemonSet renders both containers on 51516; gpus
  wins the bind and compute-domains crash-loops with:

  Error: error creating driver: error setting up healtcheck primitives: failed to listen on :51516: listen tcp :51516: bind: address already in use
  

• scripts/resolution-proof.sh runs against an operator built from this
  branch:

  • good config, ports 52000 / 52001: the DaemonSet renders both ports, the
    startup and liveness probes target them, both containers are ready with
    zero restarts across two liveness periods, GPUCluster ready;
  • bad config, computeDomains port 51516: GPUCluster condition
    Error=True ReconcileFailed: ... spec.draDriver.gpus.kubeletPlugin.healthcheck.port and spec.draDriver.computeDomains.kubeletPlugin.healthcheck.port must differ; both resolve to 51516,
    DaemonSet generation and ports unchanged, both containers still ready;
  • restoring distinct ports returns the GPUCluster to ready.

Getting the compute-domains container to start on a mock node needed
ALT_PROC_DEVICES_PATH pointing at a devices file with the IMEX channel major
(the hook the upstream DRA driver's mock e2e uses) and the
IMEXDaemonsWithDNSNames / ComputeDomainCliques gates off, since mock NVML
reports driver 550. Those settings are in the test-bed overlay, not in this
change.

---

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The DRA driver defines metrics ports for the GPU and compute-domains containers. It checks resolved healthcheck ports against enabled listener ports and returns a wrapped configuration error when positive ports conflict. Manifest generation performs this check before rendering. Tests cover collisions, disabled services, and non-colliding configurations.

Priority: ⚪ Not assessed

Merge Risk: 🟡 Moderate · up to b4275

Configurations with custom listener environment values can still render a DaemonSet whose listeners conflict. Reject those overrides or validate their effective values before merging, and add the missing collision tests.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[kvalliyurnatt]

@jramkissoon are you still planning to work on this ?

[jramkissoon]

@kvalliyurnatt Yes, still on it.

[devin-ai-integration]

Devin Review found 2 potential issues.