Skip to content

feat(sandbox): log proxy CONNECT/FORWARD denials at warn level #704

Closed

Description

@johntmyers

Problem Statement

The egress proxy logs CONNECT and FORWARD deny decisions at info!() level (proxy.rs:399, proxy.rs:1814), but the sandbox defaults to WARN log level. This means all connection denials are completely silent at the default log level.

This directly caused a misdiagnosis in #681 — the reporter saw 403 Forbidden responses but had no log output indicating why the proxy denied the request. They attributed the problem to a procfs identity resolution bug when it was actually a policy misconfiguration. The issue was only resolved after manually setting RUST_LOG=openshell_sandbox=debug, which revealed that identity resolution was succeeding and the deny was purely policy-related.

Connection denials are operationally significant events that users need to see by default to diagnose policy issues.

Proposed Design

Change the log level for proxy deny decisions from info!() to warn!() at the following locations:

  1. CONNECT deny — crates/openshell-sandbox/src/proxy.rs:399
  2. FORWARD deny — crates/openshell-sandbox/src/proxy.rs:1814
  3. Inference interception deny — crates/openshell-sandbox/src/proxy.rs:331

The allow log lines can remain at info!() — allowed connections are expected behavior and don't need warn-level visibility.

This is a one-line change per location (info! → warn!).

Alternatives Considered

  • Change the default log level to info: Would expose denials but also increase log volume significantly with allow decisions, relay events, etc. Not desirable.
  • Add a separate deny-specific log at warn while keeping the structured log at info: Unnecessary duplication — the structured log line already contains all the needed fields.

Agent Investigation

Confirmed the log sites by reading proxy.rs. The deny log at line 399 uses info!() with structured fields including action = "deny", reason, binary, dst_host, etc. The forward-proxy deny at line 1814 follows the same pattern. Both are guarded by if matches!(decision.action, NetworkAction::Deny { .. }), so raising to warn!() only affects actual denials.

Context: #681, #684 (PR that can be closed once this lands — the original bug was policy misconfiguration, not procfs).

Activity

  1. self-assigned this
    on Mar 31, 2026
  2. mjamiv commented on Apr 6, 2026

    @mjamiv
    Contributor

    Strong +1 from a production deployment running 4 sandboxes with per-service network policies.

    When we were setting up Slack Socket Mode (WebSocket connections to wss-primary.slack.com), we spent significant time debugging silent 403s. The proxy was denying our CONNECT requests because the policy had protocol: rest and tls: terminate flags that triggered L7 inspection — which breaks WebSocket frame forwarding. But we had no visibility into why requests were being denied.

    Our debugging process was:

    1. Connections fail silently
    2. gateway.log shows nothing about the deny
    3. We had to test progressively (curl from sandbox → direct CONNECT → WebSocket with different policy flags) to isolate the cause
    4. Eventually discovered the policy needed a separate entry without L7 flags

    Having proxy CONNECT/FORWARD denials logged at warn level would have cut our debugging time from days to minutes. The deny reason (which policy rule matched, which binary was checked, which endpoint was attempted) is exactly the information operators need.

    We manage 4 sandboxes with ~10 network policy entries each. Any policy misconfiguration currently requires blind experimentation to diagnose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:sandboxSandbox runtime and isolation work

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions