Skip to content

feat(podman): relay callbacks through the rootless network namespace #2540

Open
Open
feat(podman): relay callbacks through the rootless network namespace#2540
Labels
area:computearea:gatewayGateway server and control-plane workstate:review-readyReady for human reviewstate:staleInactive item at risk of automatic closure.

Description

@elezar
elezar
opened on Jul 29, 2026

Problem Statement

Rootless Podman sandboxes using slirp4netns cannot connect directly to a
gateway callback listener in the host network namespace. Podman launches
slirp4netns with --disable-host-loopback; QEMU guest validation showed that
listeners bound to loopback, the host default-route address, and 0.0.0.0 were
all unreachable from an OpenShell-style user-defined Podman network.

PR #2492 therefore restricts direct local rootless callbacks to an explicitly
reported pasta helper. Explicit slirp4netns, missing helper metadata, and
unknown helpers fail closed unless the operator configures a genuinely remote
grpc_endpoint.

OpenShell should support local callbacks with rootless slirp4netns without
requiring operators to deploy remote gateway ingress.

Proposed Design

Add a driver-owned, byte-transparent callback relay spanning Podman's rootless
network namespace and the gateway host namespace:

sandbox supervisor
  -> host.openshell.internal:<callback-port>
  -> TCP relay inside `podman unshare --rootless-netns`
  -> mode-0600 Unix socket under $XDG_RUNTIME_DIR/openshell
  -> host-side relay
  -> gateway callback-only listener on IPv4 loopback

The initial proof in PR #2473 used two socat processes:

  1. A host-namespace Unix listener forwarding to the gateway loopback listener.
  2. A TCP listener launched with podman unshare --rootless-netns, forwarding
    to that Unix socket.

Production support should implement and supervise the relay in OpenShell code
rather than require socat:

  • Add a Podman-driver relay manager, or a narrowly scoped relay subcommand
    launched by the driver under podman unshare --rootless-netns.
  • Use one relay per driver/network and callback port, not one per sandbox.
  • Start and verify the relay before persisted sandboxes resume or new sandboxes
    are created.
  • Retry transient forwarding failures, surface terminal setup failures as
    driver preconditions, and clean up child processes and socket files on
    shutdown.
  • Report/request only an IPv4 loopback callback listener from the gateway. Do
    not broaden the primary listener.
  • Preserve supervisor-to-gateway mTLS end to end; the relay must not terminate
    TLS or inspect application payloads.
  • Create the Unix socket in a user-owned runtime directory with mode 0600,
    reject unsafe/stale socket paths, and avoid logging credentials or payloads.
  • Keep callback listeners restricted to sandbox-callable gRPC methods.
  • Probe podman unshare --rootless-netns support and fail closed when the relay
    topology cannot be established.

Explicitly reported slirp4netns should select this relay. Missing helper
metadata may use it only when the relay capability and reachability can be
verified; absence of rootlessNetworkCmd must not be treated as proof of a
legacy direct-listener topology.

Alternatives Considered

  • Bind the gateway to the host default-route address: works with pasta, but
    failed with slirp4netns because host loopback access is disabled.
  • Bind the gateway to 0.0.0.0: the diagnostic wildcard listener was also
    unreachable and would unnecessarily broaden the gateway attack surface.
  • Set host_gateway_ip: changing the alias target cannot bypass
    slirp4netns host-loopback isolation.
  • Require pasta permanently: safe as the current interim behavior, but
    excludes distributions and installations that use slirp4netns.
  • Require a remote grpc_endpoint: already supported and appropriate for
    deployments with secure routable ingress, but adds unnecessary
    infrastructure for a single-host local gateway.

Agent Investigation

Acceptance Criteria

  • Rootless Podman with explicitly reported slirp4netns establishes local
    supervisor callbacks through the relay.
  • Relay startup is readiness-gated and failures prevent sandbox startup
    with an actionable error.
  • Supervisor-to-gateway mTLS and callback-only RPC authorization remain
    end to end.
  • Relay sockets and processes are securely created, supervised, restarted,
    and cleaned up.
  • Rootless pasta retains the direct default-route listener path.
  • Rootful Podman retains the exact bridge-gateway listener path.
  • Explicitly remote callback endpoints remain unchanged and request no
    local listener.
  • Missing helper metadata fails closed unless relay capability and
    reachability are positively verified.
  • QEMU E2E coverage includes Fedora and Ubuntu slirp4netns guests plus a
    pasta regression case.
  • Architecture, Podman networking, gateway configuration, and
    troubleshooting documentation describe the supported topologies.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Activity

  1. elezar
    added
    area:gatewayGateway server and control-plane work
    area:compute
    on Jul 29, 2026

  2. elezar commented on Jul 29, 2026

    @elezar
    elezar
    on Jul 29, 2026
    MemberAuthor

    🏗️ build-plan

    Implementation Plan

    Issue type: feat
    Complexity: High
    Confidence: Medium — the relay topology is validated on Ubuntu 24.04 with
    Podman 4.9.3, but listener-purpose deduplication and driver lifecycle semantics
    need maintainer confirmation.

    Summary

    Add a driver-owned, byte-transparent callback relay for rootless Podman
    slirp4netns, stacked on PR #2492. The initial implementation will prepare the
    shared relay idempotently from GetGatewayListenerRequirements, return a
    dedicated callback listener requirement, and fail closed if the relay cannot be
    made ready. It will not split CreateSandbox into Create/Start because relay
    readiness is driver-wide infrastructure rather than per-sandbox state.

    The validated transport is:

    sandbox supervisor
  -> host.containers.internal mapped to the inspected network gateway
     (for example, 10.89.0.1)
  -> 0.0.0.0:<callback-port> relay inside
     `podman unshare --rootless-netns`
  -> mode-0600 Unix socket under XDG_RUNTIME_DIR
  -> host-side byte relay
  -> gateway callback-only listener

    The wildcard bind exists only inside Podman's rootless network namespace. The
    gateway listener remains loopback or otherwise narrowly scoped and
    callback-only.

    Validated Findings

    • Ubuntu 24.04.4 with rootless Podman 4.9.3 uses
      slirp4netns --disable-host-loopback while rootlessNetworkCmd is
      unreported.
    • Podman's host-gateway maps the sandbox alias to 10.0.2.15 in this
      topology. A direct callback to the host fails, and a relay in the rootless
      network namespace does not make that blocked slirp host path usable.
    • Mapping the alias to the inspected user-defined Podman network gateway
      (10.89.0.1 in the test) and relaying from the rootless network namespace
      succeeds.
    • The relay must be ready before sandbox creation. Binding exactly to
      10.89.0.1 fails with EADDRNOTAVAIL because Podman does not materialize
      that bridge address until a container joins the network.
    • Binding 0.0.0.0:<callback-port> inside the rootless network namespace
      before sandbox creation succeeds; after the first container materializes the
      bridge, traffic to 10.89.0.1:<callback-port> reaches that listener.
    • IP_FREEBIND may permit an exact pre-bind to the future gateway address, but
      it is Linux-specific, requires relay support and additional validation, and
      is optional hardening rather than an initial implementation dependency.
    • The proof-of-concept preserved the intended host boundary: a mode-0600 Unix
      socket connected the namespace relay to a host-loopback tagged responder.
      The direct path failed and the relayed path returned the expected tag.

    Scope

    • proto/compute_driver.proto: clarify that listener-requirement discovery may
      idempotently materialize driver-owned routing prerequisites. Add no new
      sandbox lifecycle RPC in the initial implementation.
    • crates/openshell-driver-podman/src/relay.rs (new): own the secure runtime
      directory, Unix listener, host forwarding tasks, child readiness handshake,
      supervision, bounded connection handling, and cleanup.
    • crates/openshell-driver-podman/src/main.rs and CLI/config modules: add a
      private relay-child mode invoked with structured arguments under
      podman unshare --rootless-netns; do not invoke a shell or require socat.
    • crates/openshell-driver-podman/src/driver.rs and grpc.rs: select the relay
      for explicit slirp4netns; allow missing helper metadata only after positive
      relay capability/readiness checks; map the sandbox callback alias to the
      inspected Podman network gateway for the relay topology; keep pasta,
      rootful, Podman Machine, and remote endpoint paths unchanged; make
      requirement discovery async and idempotent as needed.
    • crates/openshell-server/src/lib.rs and listener tests from PR feat(compute): negotiate gateway callback listeners #2492: ensure
      the relay target is a callback-only listener and cannot be silently
      de-duplicated onto a full-primary listener with broader routing authority.
    • nix/test-guest/ QEMU coverage: add or extend Fedora and Ubuntu rootless
      slirp4netns callback tests, plus pasta and rootful regression coverage. Reuse
      PRs test(e2e): add reusable QEMU infrastructure for E2E tests #2471/test(e2e): add VM-backed E2E suite runner #2473 after resolving their stack dependency.
    • crates/openshell-driver-podman/NETWORKING.md,
      architecture/gateway.md, docs/reference/sandbox-compute-drivers.mdx, and
      .agents/skills/debug-openshell-cluster/SKILL.md: document topology,
      readiness, failure behavior, and troubleshooting.

    Implementation Steps

    1. Resolve the callback-only relay target. Extend feat(compute): negotiate gateway callback listeners #2492's listener
      materialization only as needed so the relay targets a listener whose purpose
      remains ComputeDriverCallback. A primary loopback or wildcard listener
      must not satisfy the requirement if that would expose the full multiplexed
      API through the relay. Prefer a dedicated loopback address/port when it can
      coexist safely; otherwise introduce the smallest typed callback endpoint
      needed by the requirement contract.
    2. Implement a private relay child. Add a hidden Podman-driver subcommand
      launched as podman unshare --rootless-netns <driver> <relay-child> ....
      It binds 0.0.0.0:<callback-port> inside that namespace before any sandbox
      materializes the Podman bridge address, reports readiness over an explicit
      pipe/socket handshake, and forwards opaque bidirectional streams to the
      private Unix socket with half-close and cancellation propagation. Do not
      bind the gateway process itself to a wildcard address.
    3. Implement the parent relay manager. Create a user-owned directory under
      XDG_RUNTIME_DIR, reject symlinks/unexpected owners/unsafe permissions,
      create a mode-0600 Unix socket, forward accepted streams to the
      callback-only gateway listener, bound connections and buffers, and remove
      only artifacts whose identity this process established.
    4. Prepare the relay during requirement discovery. Make
      GetGatewayListenerRequirements idempotently ensure one relay per
      driver/rootless-network-namespace/port, wait for the namespace listener,
      Unix socket, and parent relay tasks to become locally ready, and then return
      the dedicated callback listener requirement. The gateway materializes that
      host listener before sandbox resume or creation. Concurrent or repeated
      calls must reuse the same healthy relay and roll back partial startup.
    5. Use the Podman network gateway as the sandbox destination. Inspect the
      OpenShell user-defined Podman network, map host.containers.internal (or
      the callback alias selected by feat(compute): negotiate gateway callback listeners #2492) to its gateway address, and do not use
      Podman's host-gateway value for this relay topology.
    6. Supervise and fail closed. Keep the child and relay tasks alive for the
      driver lifetime. Treat unexpected death as a terminal driver health failure
      (or reject subsequent validate/create calls) rather than falling back to a
      host listener. Ensure gateway bind rejection/startup failure drops the
      managed driver and reaps relay children and sockets.
    7. Preserve existing topologies. Keep rootless pasta on the direct
      default-route listener, rootful Podman on the exact bridge gateway, Podman
      Machine on gvproxy loopback, and explicitly remote grpc_endpoint values
      without local requirements. Unknown named helpers remain unsupported.
    8. Add tests and failure injection. Cover topology and alias selection,
      idempotency, concurrency, readiness timeout, child exit, filesystem attacks,
      raw-byte fidelity, half-closes, backpressure, cleanup, port collisions,
      listener-purpose isolation, and unchanged topology paths.
    9. Add QEMU validation. Make Ubuntu 24.04 with Podman 4.9.3 and missing
      helper metadata a regression case: prove the direct callback fails and the
      relayed mTLS callback through the inspected network gateway succeeds. Also
      run Fedora with explicit slirp4netns, restart/resume behavior, and
      pasta/rootful regressions.
    10. Update architecture, operator docs, and diagnostics. Describe the
      relay's trust boundary, prerequisites, observable failures, and
      SELinux/AppArmor troubleshooting without adding a new operator-facing
      configuration field unless implementation proves one necessary.

    Test Plan

    • Unit tests: topology and callback-alias selection; repeated/concurrent
      prepare; secure runtime-directory and socket creation;
      stale/symlink/ownership rejection; child handshake/timeout/exit; cleanup;
      relay byte fidelity, half-close, cancellation, bounded concurrency, and port
      collisions.
    • Integration tests: gateway startup ordering and callback-listener purpose;
      prove sandbox resume/create does not proceed through an unready relay; prove
      primary-only methods remain rejected through the relay target.
    • E2E tests: Ubuntu 24.04/Podman 4.9.3 proves the direct path fails and the
      network-gateway relay establishes supervisor mTLS callbacks; Fedora explicit
      slirp4netns and restart/resume succeed; pasta and rootful callbacks remain
      direct.
    • Validation: mise run pre-commit, mise run test, relevant Podman/QEMU
      E2E lanes, and mise run ci before PR creation.

    Risks & Open Questions

    • Getter side effects: Document
      GetGatewayListenerRequirements as an idempotent prepare/ensure operation.
      If maintainers require strict read-only semantics, add a driver-wide
      activation RPC after listener binding; do not split CreateSandbox.
    • Listener-purpose isolation: How should a dedicated callback-only target
      coexist with a primary 127.0.0.1:<port> or 0.0.0.0:<port> listener?
      Existing address-coverage de-duplication must not widen the relay's API
      surface.
    • Namespace wildcard exposure: The wildcard listener is limited to the
      same-user Podman rootless network namespace, but other containers sharing
      that namespace may reach the callback port. Retain mTLS, expose only
      callback-authorized methods, detect port collisions, and never relay the
      gateway's primary API surface.
    • Post-readiness failure: Prefer failing the Podman driver process so the
      gateway observes driver loss. Confirm whether gateway-wide unhealthiness or
      driver-scoped unavailability is the desired blast radius.
    • Missing helper metadata: Podman 4 may use the relay only after
      podman unshare --rootless-netns and the complete relay readiness handshake
      succeed. Missing metadata is never evidence of a direct topology.
    • Multiple networks and instances: One wildcard listener can cover
      multiple Podman networks that share a rootless network namespace and port.
      Relay identity and ownership must therefore be keyed by the actual namespace
      and port, while socket names must also avoid cross-driver collisions and path
      traversal.
    • LSM compatibility: Do not identify children through
      /proc/<pid>/exe; SELinux can return ENOENT across labels. Use an explicit
      readiness handshake and test Fedora enforcing SELinux and Ubuntu AppArmor
      behavior for podman unshare, Unix sockets, and child execution.
    • QEMU stack dependency: PR feat(compute): negotiate gateway callback listeners #2492 does not currently contain PR test(e2e): add reusable QEMU infrastructure for E2E tests #2471's
      test-guest tooling. Decide whether test(e2e): add reusable QEMU infrastructure for E2E tests #2471 lands first, the stacked PR carries
      the minimal harness commits temporarily, or E2E lands as a follow-up commit
      after rebasing.
    • Optional exact binding: IP_FREEBIND could narrow the namespace listener
      to the future bridge gateway address, but it requires a Linux-specific relay
      implementation and independent kernel/namespace validation. It is deferred
      hardening; wildcard binding inside the isolated rootless namespace is the
      validated initial mechanism.

    Documentation Impact

    • Update Podman networking and gateway architecture docs with the stable
      topology, startup ordering, authorization boundary, and failure behavior.
    • Update published compute-driver documentation with slirp4netns support and
      runtime prerequisites.
    • Update debug-openshell-cluster diagnostics for relay readiness, child
      failure, runtime-directory permissions, port collisions, and
      SELinux/AppArmor checks.
    • No gateway TOML change is planned. If a new driver config field becomes
      necessary, update docs/reference/gateway-config.mdx in the same branch.

    Revision 2 — incorporates the Ubuntu 24.04/Podman 4.9.3 proof of concept:
    network-gateway aliasing, namespace-only wildcard pre-bind, concrete readiness
    ordering, and IP_FREEBIND as optional future hardening.

    Revision 1 — initial plan; selects idempotent requirement-time relay
    preparation and records callback-listener isolation as the main review
    decision.

  3. elezar
    added
    state:review-readyReady for human review
    on Jul 29, 2026

  4. github-actions commented on Aug 13, 2026

    @github-actions
    github-actionsbot
    on Aug 13, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  5. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Aug 13, 2026

  6. elezar commented on Aug 21, 2026

    @elezar
    elezar
    on Aug 21, 2026
    MemberAuthor

    A second callback-relay use case surfaced in the Fedora RPM release canary: the gateway itself runs inside a nested container and uses rootful Podman. Listener discovery returns the future Podman bridge gateway (currently 10.89.0.1), but that address is not present in the gateway network namespace until the first sandbox joins the network. The gateway therefore fails at startup when it attempts the exact secondary bind, as seen in release-canary run 32448768823, job 96673135310: https://github.com/NVIDIA/OpenShell/actions/runs/32448768823/job/96673135310

    We are handling this in two stages:

    1. Add an opt-in PR E2E lane that runs the PR CLI and gateway binaries in the nested Fedora/rootful-Podman topology and verifies a sandbox reaches the callback API.
    2. Once that lane reproduces the failure, use a narrowly scoped wildcard callback listener only for the detected nested-container/rootful-Podman case. The primary API listener remains loopback-bound. The exception must be documented as expanding callback-port reachability to every interface in the outer container network namespace; mTLS and callback-only RPC authorization remain required defenses.

    Longer term, the driver-owned relay proposed here may be the cleaner common abstraction if it is generalized beyond rootless slirp4netns. It could keep the gateway callback listener on loopback and place any required wildcard bind inside the driver-controlled Podman network namespace, covering both slirp4netns reachability and the nested-rootful bridge-materialization race. This is not necessarily the same implementation path: the current issue assumes a rootless network namespace and its acceptance criteria explicitly retain the exact rootful bridge listener. The nested-rootful topology should therefore be evaluated as an additional relay mode, not silently folded into the existing rootless assumptions.

  7. github-actions
    removed
    state:staleInactive item at risk of automatic closure.
    on Aug 23, 2026

  8. github-actions commented on Sep 6, 2026

    @github-actions
    github-actionsbot
    on Sep 6, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  9. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:computearea:gatewayGateway server and control-plane workstate:review-readyReady for human reviewstate:staleInactive item at risk of automatic closure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions