Skip to content

bug: Kubernetes E2E rewrites local images to GHCR #3696

Description

@danehans

User Story

As a contributor, I want ephemeral Kubernetes E2E runs to use the images built and imported locally, so that tests run without requiring unpublished images in GHCR.

Problem Statement

On main, e2e/with-kube-gateway.sh decomposes an unqualified local image such as openshell/gateway:e2e-<cluster> into Helm values with an empty per-image registry. The chart then falls back to global.image.registry=ghcr.io/nvidia, so the cert-generation Job requests ghcr.io/nvidia/openshell/gateway:e2e-<cluster> instead of the imported local image.

This behavior was observed after the image override changes in #3341.

Impact / Why This Matters

Ephemeral k3d E2E tasks fail during Helm installation before any tests run. Contributors can work around the failure by overriding the registry or clearing the chart-wide registry, but the default local workflow should use the images it just built and imported.

Acceptance Criteria

  • Ephemeral Kubernetes E2E uses unqualified locally built and imported gateway, sandbox, and supervisor images without adding the released chart registry.
  • Qualified registry overrides and published-image CI workflows continue to resolve correctly.
  • Automated coverage verifies the rendered image reference for an unqualified local image.

Reproduction Steps

  1. Check out main at 4cd1351e1 or later.
  2. Ensure Docker is running.
  3. Run mise run e2e:kubernetes.
  4. Observe the locally built/imported image openshell/gateway:e2e-<cluster>.
  5. Observe Helm fail because the cert-generation Job requests the nonexistent GHCR tag.

Environment

  • OpenShell: 4cd1351e1
  • OS: macOS arm64, Darwin 25.6.0
  • Docker: 29.6.2
  • k3d: v5.9.0
  • k3s: v1.35.5-k3s1
  • Helm: v4.2.4

Logs

Failed to pull image "ghcr.io/nvidia/openshell/gateway:e2e-<cluster>":
... ghcr.io/nvidia/openshell/gateway:e2e-<cluster>: not found

Error: INSTALLATION FAILED: failed pre-install:
resource Job/openshell/openshell-certgen not ready

Activity

  1. kvnloo commented on Sep 27, 2026

    @kvnloo

    I think the important distinction here is:

    registry omitted because the chart should inherit global.registry

    vs.

    registry intentionally empty because this is an unqualified local image

    The current E2E path loses that distinction, so Helm's normal fallback to global.image.registry=ghcr.io/nvidia wins.

    For the local-image lane I would prefer explicitly clearing the global registry for that render rather than relying on an empty per-image value that the chart interprets as "inherit".

    Regression shape:

    • openshell/gateway:e2e-local -> renders exactly openshell/gateway:e2e-local;
    • registry.example.com/openshell/gateway:test -> remains fully qualified;
    • normal release values with no local override -> still inherit ghcr.io/nvidia;
    • gateway, sandbox, supervisor and cert-generation Job all resolve consistently.

    That tests the final rendered image references, which is the contract the k3d import actually depends on.

    If accepted, this looks like a small E2E/chart-boundary fix.

  2. danehans commented on Sep 29, 2026

    @danehans
    ContributorAuthor

    /assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions