Skip to content

bug(sandbox): network broker logs expected socket races as warnings #3526

Open
Open
bug(sandbox): network broker logs expected socket races as warnings#3526
Labels
area:sandboxSandbox runtime and isolation workos:linuxIssue affects Linux hostsstate:staleInactive item at risk of automatic closure.topic:observabilityLogging, metrics, and observability work

Description

@drew
drew
opened on Sep 21, 2026

This was generated by AI during triage.

Agent Diagnostic

  • Skills loaded: diagnose, OpenShell repository create-github-issue, and cluster inspection workflows
  • OpenShell version tested: 0.0.117-dev.211+g4cd5e5478
  • Latest release checked: v0.0.116; the issue reproduces on a newer development build containing the RFC 0012 sandbox runtime
  • Known fixes reviewed: release notes for v0.0.116, RFC 0012 implementation PR feat(isolation): implement the RFC 0012 sandbox architecture #2942, and current main network-broker handling
  • Possible duplicates reviewed: searched all OpenShell issues and merged PRs for the exact network notification denied text, Socket not connected, and related network/boundary warnings. No matching issue was found. bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited #3396 concerns a boundary reconnect that becomes fatal; the sandboxes here remained healthy.
  • Findings: six healthy Kubernetes workload pods emitted 4,183 sandbox network notification denied warnings in 12 hours. Nearly all were syscall 52 returning ENOTCONN; the remainder were syscall 62 returning ESRCH. All pods stayed Ready with zero restarts and there were no error-level runtime logs. Current main logs every dispatch_notification error at WARN, regardless of whether it represents an enforcement decision or an expected process/socket race.
  • Remaining reason for filing: these per-syscall warnings are high-volume and misleading, and obscure actionable policy or boundary failures.

Description

Actual behavior: The RFC 0012 sandbox network broker emits a WARN for every failed seccomp notification dispatch:

WARN openshell_sandbox::network_broker: sandbox network notification denied (tid=..., syscall=52): Socket not connected (os error 107)
WARN openshell_sandbox::network_broker: sandbox network notification denied (tid=..., syscall=62): No such process (os error 3)

During a 12-hour observation of six active agent workloads, this produced 4,183 network-broker warnings. The workloads remained Ready, had zero restarts, and recorded no error-level runtime events. A two-hour sample was dominated by 910 ENOTCONN warnings and 34 ESRCH warnings.

The current handler groups all dispatch_notification errors under the message sandbox network notification denied, even when the returned errno describes a socket/process race rather than an OpenShell policy denial. This makes benign application behavior look like a security or isolation failure and makes genuine broker failures difficult to find.

Expected behavior: Expected transient process/socket races should not create one warning per syscall. They should be handled at debug/trace level, aggregated, or rate-limited. Genuine OpenShell policy denials and broker-health failures must remain clearly observable and distinguishable from application-originated errnos.

Reproduction Steps

  1. Deploy the RFC 0012 sandbox runtime on Linux with seccomp network mediation enabled.
  2. Run a long-lived agent workload that performs ordinary concurrent HTTP/socket activity and periodically starts and exits subprocesses.
  3. Collect the workload pod's runtime logs for several polling cycles.
  4. Count sandbox network notification denied messages.
  5. Observe repeated syscall 52/ENOTCONN and syscall 62/ESRCH warnings while the sandbox remains Ready and functional.

Environment

  • OS: Ubuntu 24.04, Linux amd64
  • Kubernetes: v1.35.7-gke.1222000
  • Agent Sandbox controller: v0.5.0, v1beta1 API
  • Docker: not applicable; Kubernetes compute driver
  • OpenShell: 0.0.117-dev.211+g4cd5e5478
  • Deployment: Helm 0.0.0-dev, RFC 0012 separate workload and supervisor pods
  • Latest release checked: yes, v0.0.116; the tested development build is newer
  • Possible duplicates checked: yes; bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited #3396 is related to fatal boundary recovery but does not cover this non-fatal per-syscall warning storm

Logs

# Counts by workload over 12 hours:
workload-1 network_denied=190 errors=0 restarts=0
workload-2 network_denied=90  errors=0 restarts=0
workload-3 network_denied=1224 errors=0 restarts=0
workload-4 network_denied=897 errors=0 restarts=0
workload-5 network_denied=1046 errors=0 restarts=0
workload-6 network_denied=736 errors=0 restarts=0

# Dominant messages over a two-hour sample:
910 WARN ... syscall=52: Socket not connected (os error 107)
 34 WARN ... syscall=62: No such process (os error 3)

Acceptance Criteria

  • Expected ENOTCONN and ESRCH process/socket races do not emit an unbounded WARN per syscall.
  • Genuine policy denials remain observable as explicit policy decisions rather than being conflated with application errno handling.
  • Fatal listener or broker-health failures remain at error level and retain actionable context.
  • Telemetry provides aggregate counts or suitably rate-limited diagnostics for suppressed transient failures.
  • Tests cover notification targets disappearing and sockets closing between notification receipt and dispatch without producing warning storms.

Activity

  1. drew
    added
    topic:observabilityLogging, metrics, and observability work
    area:sandboxSandbox runtime and isolation work
    os:linuxIssue affects Linux hosts
    on Sep 21, 2026

  2. drew commented on Sep 22, 2026

    @drew
    drew
    on Sep 22, 2026
    CollaboratorAuthor

    🏗️ build-plan

    Implementation Plan

    Issue type: fix
    Complexity: Medium
    Confidence: High — clear path

    Summary

    Separate expected seccomp notification races from enforcement rejections and unexpected broker failures in the sandbox network broker. Expected ENOTCONN and ESRCH outcomes will be debug-level per event and summarized through bounded, periodic aggregate telemetry, while genuine denials and unexpected failures retain distinct warning-level diagnostics and listener-health failures remain errors.

    Scope

    • crates/openshell-sandbox/src/network_broker.rs: classify dispatch failures, aggregate transient race counts over a fixed interval, emit semantically distinct diagnostics, and add focused unit/logging tests.
    • architecture/sandbox.md: document the broker's notification-race diagnostic behavior and the separation from enforcement and health failures.

    Implementation Steps

    1. Add a small dispatch-failure classifier that recognizes ENOTCONN and ESRCH as expected target/socket races while separating enforcement errnos from unexpected operational failures.
    2. Add broker-owned, bounded counters keyed by syscall and errno, with a fixed summary interval and deterministic time injection for tests.
    3. Replace the catch-all warning with category-specific reporting: debug for individual transient races, periodic aggregate informational telemetry, explicit warning text for enforcement rejections, and explicit warning text for unexpected dispatch failures. Preserve the existing error path for listener failure.
    4. Document the observable logging contract in the sandbox architecture overview.
    5. Add regression tests for disappearing notification targets, closed/unconnected sockets, aggregation behavior, and warning-level separation.

    Test Plan

    • Unit tests: In network_broker.rs, verify errno classification, fixed-window counts/reset behavior, repeated ENOTCONN/ESRCH events producing no per-event warnings, and enforcement/unexpected errors remaining warning-level and distinguishable.
    • Integration tests: N/A — this changes broker-local diagnostic handling; the existing seccomp broker tests continue to cover notification response behavior.
    • E2E tests: N/A — no e2e/ behavior or deployment contract changes.

    Risks & Open Questions

    • Tracing tests must avoid global-subscriber races; use a scoped subscriber and test the reporting helper directly.
    • Aggregation must remain bounded and must not delay the errno response sent back to the notifying task.
    • No stakeholder decision is required: the issue explicitly permits aggregate or rate-limited diagnostics.

    Documentation Impact

    • Update architecture/sandbox.md with the stable diagnostic-level distinction.
    • No gateway TOML, driver option, Helm, or published user-workflow documentation changes.
    • LSM compatibility is unchanged: the fix does not add procfs access, process-identity resolution, binary execution, or cross-process visibility tests, so SELinux/AppArmor behavior is unaffected.

    Revision 1 — initial plan

  3. drew commented on Sep 22, 2026

    @drew
    drew
    on Sep 22, 2026
    CollaboratorAuthor

    🏗️ build-from-issue-agent

    Implementation Complete

    PR: #3569

    What was built

    The sandbox network broker now classifies expected ENOTCONN and ESRCH notification races separately from enforcement decisions and unexpected failures. Per-event races are debug-level, sustained races produce one-minute aggregate counts, and listener health failures remain errors.

    Tests

    • Unit: 3 tests added
    • Integration: full mise run test suite passed
    • E2E: N/A (no e2e/ changes)

    Docs updated

    • architecture/sandbox.md

    The issue will auto-close when the PR is merged.

  4. github-actions commented on Oct 7, 2026

    @github-actions
    github-actionsbot
    on Oct 7, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  5. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:sandboxSandbox runtime and isolation workos:linuxIssue affects Linux hostsstate:staleInactive item at risk of automatic closure.topic:observabilityLogging, metrics, and observability work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions