[Bug] Egress proxy fails to relay WebSocket frames after successful HTTP CONNECT + 101 Switching Protocols

Agent Diagnostic

Pointed coding agent at the OpenShell repository and inspected the relevant files: crates/openshell-sandbox/src/l7/rest.rs, relay.rs, provider.rs, and mod.rs
Identified that is_bodiless_response() treats 101 Switching Protocols as a generic 1xx informational response. After forwarding the 101 headers, the relay loop returns to HTTP parsing instead of switching to raw TCP passthrough.
Confirmed that access: full only expands to broad HTTP-level rules (rule_json("*", "**") in expand_access_presets()). It does not provide true raw TCP passthrough — L7 inspection continues inside the CONNECT tunnel.
Verified with a raw Python socket test: the HTTP/1.1 101 Switching Protocols response arrives correctly, but all subsequent WebSocket frames are dropped (timeout). The same test run directly from the host (bypassing the proxy) receives both the 101 and the WebSocket frames immediately.
Proxy logs confirm continued L7 inspection even after the upgrade: [openshell_sandbox::l7::relay] HTTP_REQUEST ...
Wrote and tested a patch: detect 101 in relay_response(), added RelayOutcome::Upgraded variant, and switch to tokio::io::copy_bidirectional after upgrade. All 322 existing tests pass + new test for 101 behavior.
Built the patched binary (cross-compiled for linux/arm64 via Docker), deployed it to the K3s container via hostPath, and verified that WebSocket frames now relay end-to-end through the proxy.
Consulted Grok (xAI) for architectural guidance on the multi-hop networking and security implications throughout the investigation.

Description

Summary

When an application inside an OpenShell sandbox establishes a WebSocket connection through the egress proxy (via HTTP CONNECT tunnel), the proxy correctly relays the HTTP/1.1 101 Switching Protocols response but drops all subsequent WebSocket frames. This makes persistent bidirectional WebSocket connections impossible from inside a sandbox.

This appears to be a deeper manifestation of (or closely related to) the existing issue #409 ("openshell-sandbox egress proxy kills WebSocket connections after ~2 minutes").

Root Cause (from code inspection)

In crates/openshell-sandbox/src/l7/rest.rs, the is_bodiless_response() function returns true for all 1xx status codes (including 101):

fn is_bodiless_response(request_method: &str, status_code: u16) -> bool {
    request_method.eq_ignore_ascii_case("HEAD")
        || (100..200).contains(&status_code)  // 101 falls here
        || status_code == 204
        || status_code == 304
}

After forwarding the 101 headers, relay_response() returns control to the HTTP parsing loop, which then tries to parse the next HTTP request from the client. However, the connection has been upgraded — the subsequent bytes are WebSocket frames (binary), not valid HTTP. This causes the relay to either fail silently or block waiting for more HTTP data.

Additionally, access: full only provides HTTP-level permissiveness. L7 inspection continues inside the CONNECT tunnel even with this policy, as confirmed by proxy logs showing HTTP_REQUEST after the CONNECT is allowed.

Impact

This blocks any sandboxed application that needs persistent bidirectional WebSocket communication, including:

OpenClaw node → remote OpenClaw gateway meshes (connect.challenge frame is dropped)
Discord/Slack bot integrations
Any long-lived protocol-upgraded connections

Workaround

A custom patch that detects 101 and switches to raw tokio::io::copy_bidirectional works reliably. We have a tested version and are happy to submit a PR.

Acknowledgments

This issue was diagnosed and fixed collaboratively. Claude (Anthropic) performed the deep code analysis and wrote the patch. Grok (xAI) provided architectural and security guidance. The human drove the investigation and maintained strict security requirements throughout.

Reproduction Steps

Create an OpenShell sandbox with a network policy that allows the destination (protocol: tcp, access: full).
Install socat inside the sandbox.
Start a socat PROXY listener inside the sandbox:
socat TCP-LISTEN:18789,reuseaddr,fork PROXY:10.200.0.1::,proxyport=3128 &
Run the following raw WebSocket test from inside the sandbox:

import socket, time

s = socket.socket()
s.settimeout(15)
s.connect(('127.0.0.1', 18789))

req = (
    b'GET / HTTP/1.1\r\n'
    b'Host: localhost\r\n'
    b'Upgrade: websocket\r\n'
    b'Connection: Upgrade\r\n'
    b'Sec-WebSocket-Key: RylUQAh3p5cysfOlexgubw==\r\n'
    b'Sec-WebSocket-Version: 13\r\n'
    b'\r\n'
)
s.send(req)

time.sleep(1)
resp1 = s.recv(4096)
print('RECV1:', repr(resp1[:300]))

time.sleep(3)
try:
    resp2 = s.recv(4096)
    print('RECV2:', repr(resp2[:200]))
except socket.timeout:
    print('RECV2: TIMEOUT - WebSocket frames not relayed')

s.close()

Expected: RECV2 contains WebSocket frame data from the server.
Actual: RECV2 times out — no WebSocket frames are relayed after the 101 response.
Control test: Running the same test directly from the host (bypassing the sandbox proxy) receives both the 101 and the subsequent frames immediately.

Environment

OS: macOS 15.6.1 (Apple Silicon M4 Max)
Docker: OrbStack (Docker Engine 28.5.2)
OpenShell: 0.0.16
K3s cluster image: ghcr.io/nvidia/openshell/cluster:0.0.16
Sandbox image: ghcr.io/nvidia/openshell-community/sandboxes/openclaw:latest
OpenClaw: 2026.3.11 (both gateway and node)

Logs

Proxy log showing continued L7 HTTP inspection inside the CONNECT tunnel (after CONNECT is allowed):

[sandbox] [INFO] [openshell_sandbox::proxy] CONNECT action=allow ancestors=/usr/local/bin/socat1 -> /opt/openshell/bin/openshell-sandbox binary=/usr/local/bin/socat1 dst_host=<gateway-ip> dst_port=18789 engine=opa policy=openclaw_gateway proxy_addr=10.200.0.1:3128 src_addr=10.200.0.2 src_port=33246

[sandbox] [INFO] [openshell_sandbox::l7::relay] HTTP_REQUEST credentials_injected=false host=<gateway-ip> method=GET path=/ port=18789 request_num=1

Agent-First Checklist

I pointed my agent at the repo and had it investigate this issue
I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
My agent could not resolve this — the diagnostic above explains why

fix(sandbox): relay WebSocket frames after HTTP 101 Switching Protocols

For anyone interested in the patch while I work through the vouch system, you can see the pr in my fork.

davidpeden3#1

Workaround: Split WebSocket and REST into Separate Network Policies

We hit this exact issue running OpenClaw's Slack Socket Mode inside an OpenShell sandbox (v0.0.15–0.0.16). Our reliable workaround is to split Slack into two separate network policies:

1. `slack` policy — REST endpoints

These can safely use protocol: rest and tls: terminate:

slack.com:443
api.slack.com:443
files.slack.com:443
hooks.slack.com:443
edgeapi.slack.com:443

2. `slack_websocket` policy — WebSocket endpoints

These MUST use tls: skip and must NOT have protocol: rest or tls: terminate:

wss-primary.slack.com:443
wss-backup.slack.com:443

Why this works

When protocol: rest or tls: terminate is set on a policy, the egress proxy intercepts the TLS session and performs L7 HTTP parsing. This breaks WebSocket frame forwarding after the 101 Switching Protocols upgrade — the proxy tries to parse binary WebSocket frames as HTTP and either drops or corrupts them.

Using tls: skip forces the proxy into raw TCP passthrough mode for those endpoints, bypassing L7 inspection entirely. The CONNECT tunnel is established and then the proxy just relays bytes, which is exactly what WebSocket needs after the 101 upgrade.

Important: avoid wildcards

On OpenShell 0.0.15+ you cannot use a *.slack.com wildcard to cover both policies. A wildcard policy will override the specific WebSocket host entries, so the L7 inspection settings from the wildcard apply to everything — including the WebSocket hosts. Each host must be listed individually in its respective policy.

Status

This workaround has been running reliably for us in production on OpenShell v0.0.15–0.0.16 with OpenClaw Slack Socket Mode (which uses @slack/socket-mode under the hood).

Affected version: OpenShell v0.0.16 | NemoClaw v0.1.0 | OpenClaw 2026.3.11
Platform: Ubuntu 24.04 LTS, K3s (via NemoClaw Docker), x86_64, RTX 5070 Ti

Adding our case to support prioritization of this fix.

Symptoms: Gateway UI loads at our Cloudflare tunnel endpoint, but chat is completely non-functional. UI shows "Disconnected from gateway" and "device identity required." Gateway logs show repeated Error reading events stream: ServiceError: error reading a body from connection — the event stream between gateway and sandbox dies within seconds of startup.

What we've ruled out:

Origin/CORS config (allowedOrigins includes our tunnel hostname + localhost)
Auth config (dangerouslyDisableDeviceAuth: true has no effect — gateway can't read sandbox config without the stream)
Pod health (both openshell-0 and sandbox pod show Ready, restarting either doesn't help)
Clean reinstall (nemoclaw onboard from scratch — same result)

Our understanding: The egress proxy's is_bodiless_response() treats HTTP 101 (Switching Protocols) as a generic 1xx informational response and fails to switch to raw TCP passthrough after the WebSocket upgrade handshake. All subsequent WebSocket frames are silently dropped.

Questions:

We see a tested patch exists (322 tests passing, cross-compiled for K3s/ARM64) pending the contributor vouching process. Is there a pre-built binary or branch we could test against? Happy to validate on our setup.
In the interim, would adding a tls: skip network policy for the WebSocket host(s) involved in the gateway↔sandbox stream be a viable workaround? The idea is to bypass L7 proxy inspection and force raw TCP passthrough for those connections.
Any ETA on the patch merge?

This is a 24/7 production agent PC — the broken event stream is our only remaining blocker to full functionality. Willing to help test any fix.

@CoreyAgent

@CoreyAgent — we hit this exact same issue and documented the root cause + workaround in detail earlier in this thread.

To answer your questions from our experience:

Pre-built patch binary: We haven't found one available yet either. We worked around it entirely at the application layer.
Policy workaround: Yes — splitting WebSocket endpoints into a separate network policy without protocol: rest or tls: terminate flags is the working workaround. This forces raw TCP tunnel passthrough instead of L7 inspection, which preserves WebSocket frames. In our case we split Slack's wss-primary.slack.com and wss-backup.slack.com into their own policy. Your gateway↔sandbox event stream likely needs the same treatment for whatever internal WebSocket endpoint it uses.
ETA: Unknown, but more confirmed reports on this issue should help prioritize it.

The key insight is that the proxy's is_bodiless_response() treats HTTP 101 as a generic 1xx, exactly as you described. Any policy flag that triggers L7 inspection will break the frame forwarding after the upgrade handshake.

@mjamiv

Thanks @mjamiv — we followed your advice and attempted the tls: skip split-policy workaround. Wanted to report back since the internal gateway↔sandbox case may behave differently than the Slack Socket Mode case you described.

What we tried:

Created a custom NemoClaw policy preset (openshell_gateway_ws.yaml) targeting the internal gateway ClusterIP address at openshell.openshell.svc.cluster.local:8080, configured with tls: skip and no protocol: rest or tls: terminate flags. Applied it as policy v8 via nemoclaw corey policy-add. Policy applied without errors.

Result:

Event stream still broken. Symptoms unchanged — gateway logs continue to show:

Sandbox watcher error: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection

Chat in the Control UI still shows "Disconnected from gateway" and "device identity required."

Our theory on why:

Your Slack workaround targets external WebSocket hosts that transit the egress proxy as normal outbound TCP. The gateway↔sandbox event stream may be different — the sandbox initiates the WebSocket upgrade back to the gateway over the K8s ClusterIP service, and we're not certain the tls: skip policy can match on a cluster-internal address (vs. a public FQDN). It's also possible our preset's binary field doesn't include the right executable name for the sandbox event stream process.

A few targeted questions if you have a moment:

Did your workaround ever apply to a K8s-internal ClusterIP address, or only to public FQDNs (Slack, Discord, etc.)?
Can you share the exact YAML structure of your working Slack policy (specifically the tls, access, and binary fields)? We want to confirm we're using the right field names.
Does the sandbox process need to be restarted after a new tls: skip policy is applied, or does it pick up live?

We're aware of @davidpeden3's PR #683 and watching it closely — but given that's pending NVIDIA review, we were hoping the tls: skip path could unblock us in the interim. Any insight appreciated.

fix(sandbox): relay WebSocket frames after HTTP 101 Switching Protocols

Due to challenges in reviewing and testing contributor PRs we have made an updated PR with some additional testing and implementation differences at #718. If anyone would like to deploy from this branch and report back that would be great.

@CoreyAgent

@CoreyAgent — happy to share the exact YAML. Here's the working Slack Socket Mode split-policy:

Main Slack policy (REST API + file uploads) — with L7 inspection:

slack:
  name: slack
  endpoints:
    - host: slack.com
      port: 443
      protocol: rest
      enforcement: enforce
      access: full
    - host: api.slack.com
      port: 443
      protocol: rest
      enforcement: enforce
      access: full
    - host: files.slack.com
      port: 443
      enforcement: enforce
      access: full
    - host: hooks.slack.com
      port: 443
      enforcement: enforce
      access: full
    - host: edgeapi.slack.com
      port: 443
      enforcement: enforce
      access: full
    - host: "*.slack-files.com"
      port: 443
      enforcement: enforce
      access: full
  binaries:
    - { path: /usr/bin/node }

Separate WebSocket policy — raw TCP tunnel, no L7:

slack_websocket:
  name: slack_websocket
  endpoints:
    - host: wss-primary.slack.com
      port: 443
      tls: skip
      enforcement: enforce
      access: full
    - host: wss-backup.slack.com
      port: 443
      tls: skip
      enforcement: enforce
      access: full
  binaries:
    - { path: /usr/bin/node }
    - { path: /usr/bin/curl }

To answer your specific questions:

K8s-internal ClusterIP vs public FQDN: Our workaround only applies to external public FQDNs transiting the egress CONNECT proxy. The proxy's is_bodiless_response() 101-handling issue is in the egress path. If your gateway-to-sandbox event stream is over a ClusterIP, it may not transit the same proxy path at all — which would explain why tls: skip doesn't help. The root cause may be different for internal connections.
Key fields: tls: skip is the critical one — it forces raw TCP passthrough and prevents the proxy from attempting L7 inspection after the 101 upgrade. Do NOT include protocol: rest or tls: terminate on the WebSocket policy. The access: full field is included but may not be strictly necessary.
Restart after policy change: The policy hot-reloads via openshell policy set <sandbox> --policy <file> --wait — no sandbox restart needed. However, the application inside the sandbox needs to re-establish the WebSocket connection to pick up the new routing. In our case, restarting the gateway process (which re-initiates Socket Mode) was sufficient.

Also noting @johntmyers' PR #718 — if that lands, it should fix the root cause for both external and internal WebSocket paths. We'll be testing it when it's available.

fix(sandbox): relay WebSocket frames after HTTP 101 Switching Protoco…

chore: upstream merge 2026-04-08 (Wave 3) (#5)

Navigation Menu

Repository navigation

[Bug] Egress proxy fails to relay WebSocket frames after successful HTTP CONNECT + 101 Switching Protocols #652

Agent Diagnostic

Description

Reproduction Steps

Environment

Logs

Agent-First Checklist

Workaround: Split WebSocket and REST into Separate Network Policies

1. `slack` policy — REST endpoints

2. `slack_websocket` policy — WebSocket endpoints

Why this works

Important: avoid wildcards

Status

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] Egress proxy fails to relay WebSocket frames after successful HTTP CONNECT + 101 Switching Protocols #652

Description

Agent Diagnostic

Description

Reproduction Steps

Environment

Logs

Agent-First Checklist

Activity

davidpeden3 commented on Mar 29, 2026

mjamiv commented on Mar 30, 2026

Workaround: Split WebSocket and REST into Separate Network Policies

1. slack policy — REST endpoints

2. slack_websocket policy — WebSocket endpoints

Why this works

Important: avoid wildcards

Status

CoreyAgent commented on Mar 30, 2026

mjamiv commented on Mar 30, 2026

CoreyAgent commented on Mar 31, 2026

johntmyers commented on Apr 1, 2026

mjamiv commented on Apr 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `slack` policy — REST endpoints

2. `slack_websocket` policy — WebSocket endpoints