Skip to content

feat(ha): harden gateway peer transport and expand failover conformance #3529

Open
Open
feat(ha): harden gateway peer transport and expand failover conformance#3529
Milestone
OpenShell 0.1.5

Description

@drew
drew
opened on Sep 21, 2026

User Story

As an OpenShell cluster operator, I want gateway peer routing to require an explicitly trusted encrypted transport and to be validated across supported sandbox operations, so that HA does not weaken confidentiality or fail only on less-common workflows.

Problem Statement

PR #1868 authenticates gateway peers with Kubernetes ServiceAccount identity, but the current transport and validation contract has gaps:

  • HTTPS peer clients fall back to the platform trust store when no peer CA is configured, widening the trust surface for internal peer traffic.
  • A multi-replica gateway that itself serves plaintext accepts an http:// peer endpoint with only a warning, even though peer relays carry supervisor traffic and projected peer credentials.
  • The HA e2e keeps a sandbox alive through scaling and pod rotation and covers create/watch, exec, upload, and download, but does not exercise the full supported set of operations that depend on owner routing, such as forwarding/service traffic and configuration changes.

Impact / Why This Matters

A deployment mistake can leave internal peer traffic unencrypted or trusted through roots unrelated to the cluster peer identity. Separately, operators can pass the current HA smoke test while encountering failures in untested session-bound or configuration workflows after scaling or rolling gateway pods. The workaround is manual cluster-specific testing and inspection, which is inconsistent and easy to omit.

Proposed Design

Define and enforce a fail-closed peer transport contract for multi-replica gateways:

  • Peer routing requires TLS with an explicit peer CA or an equivalently scoped cluster trust source; it does not silently fall back to general platform roots.
  • Multi-replica startup rejects plaintext peer endpoints.
  • Documentation identifies the operations expected to remain available through a non-owner gateway and during gateway scaling, failover, and rolling replacement.
  • HA conformance keeps a long-running sandbox active while exercising representative operations from that documented inventory through different gateway replicas.

Acceptance Criteria

  • Multi-replica gateway startup rejects an http:// peer endpoint.
  • HTTPS peer routing fails closed when no explicitly scoped peer trust source is configured.
  • Helm-generated and custom TLS configurations document the required peer CA, server identity, and client identity contract.
  • The supported HA operation inventory includes sandbox create/watch, exec, file transfer, TCP or service forwarding, and relevant policy/provider configuration workflows.
  • Automated HA tests exercise that inventory against a long-running sandbox across scale-up, scale-down, owner loss, and rolling pod replacement.
  • Negative tests cover plaintext endpoints, missing peer CA material, and peer identity mismatch.
  • Architecture and gateway-configuration documentation match the enforced transport and conformance behavior.

Alternatives Considered

Continuing to warn for plaintext preserves development configurations but permits an insecure production mistake. Platform trust roots are convenient for public endpoints, but internal gateway peers already have a chart-managed identity and should use narrowly scoped trust. Testing every RPC independently would be expensive and brittle, so the conformance suite should test the documented operation classes and shared routing paths instead.

Agent Investigation

This groups the remaining peer-security and coverage review discussions from PR #1868:

Related umbrella issue: #1021.

Activity

  1. drew
    added this to OpenShell Roadmapon Sep 22, 2026
  2. drew
    added this to the OpenShell 0.1.1 milestone on Sep 22, 2026
  3. github-project-automation
    moved this to Todo in OpenShell Roadmapon Sep 22, 2026

  4. lunarwhite commented on Sep 25, 2026

    @lunarwhite
    lunarwhite
    on Sep 25, 2026
    Contributor

    📋 triage-agent

    Triage Assessment

    Classification: validated-feature

    Summary

    All three gaps exist on main at f213e9a, which includes PR #1868. The behavior is documented as intended (docs/reference/gateway-config.mdx:528,532), so this is hardening and test expansion, not a regression. A fail-closed design is feasible, but rejecting http:// in multi-replica mode conflicts with supported topologies. Confidence is high for code facts and medium for runtime effects.

    Investigation

    • Platform-root fallback. Without OPENSHELL_PEER_TLS_CA_FILE, peers trust the platform roots (crates/openshell-server/src/supervisor_session.rs:86-97). The chart sets it only with the PKI job or cert-manager, so default TLS installs pin the chart CA. With both off, the handshake most likely fails, because no public root signs the cluster-local name the chart verifies. An SSL_CERT_FILE bundle, which the chart sets for an OIDC CA, replaces the platform roots.
    • Plaintext endpoints. Startup rejects http:// only when the local gateway serves TLS, and otherwise warns (crates/openshell-server/src/lib.rs:487-502). This applies to any PostgreSQL-backed gateway, even with one replica. Peers share the main listener, so server.disableTls=true makes peer traffic plaintext.
    • HA coverage. The HA e2e exercises create/watch, exec, and file transfer across scaling and pod deletion. CI runs it only in plaintext, only with the test:e2e-kubernetes label, and never in merge groups. The TLS peer path, TCP and service forwarding, interactive exec, and the provider and endpoint-status peer RPCs are untested.
    • Design. Rejecting http:// in multi-replica mode would break edge TLS termination with server.disableTls=true, the OpenShift PostgreSQL setup, and the HA CI job. Options include requiring pod TLS, a dedicated peer listener, or a warned opt-out like allow_insecure_transport. Each needs the dialing side to enforce the policy too. The criteria don't define "multi-replica" or "peer identity mismatch".

    Impact Signals

    • Affected users/scope: PostgreSQL-backed Kubernetes gateways that set server.disableTls=true, disable both the PKI job and cert-manager, or run outside the chart. Default TLS installs already pin the chart CA.
    • Regression: No. New in PR feat(kubernetes): support HA gateway rebalancing #1868 and documented as intended.
    • Workaround: Available, partially. Keep pod TLS with the PKI job or cert-manager, or inject OPENSHELL_PEER_TLS_CA_FILE with a post-renderer.
    • Evidence quality: High for code, chart, and CI facts. Medium for runtime behavior, which wasn't reproduced on a cluster.
  5. drew
    modified the milestones: OpenShell 0.1.1, OpenShell 0.1.5 on Sep 28, 2026

  6. EmilienM commented on Sep 28, 2026

    @EmilienM
    EmilienM
    on Sep 28, 2026
    Contributor

    I've started to take a look at it today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions