fix(sandbox-backend): recover TCP mediation after boundary disconnects

[shiju-nv]

Summary

A lost boundary connection can stop the TCP proxy even when another request reconnects successfully. Make pending TCP accepts use the existing authenticated recovery path so the proxy can accept new traffic after recovery.

Related Issue

Related to #3396.

Changes

• Share the existing process-wait recovery loop with TCP accepts, retaining their response stream and allowing healthy idle waits.
• Close the response side of the internal stream on gRPC failure so waiting readers observe the disconnect.
• Observe the connection generation before each retryable request so delayed errors cannot discard a recovered connection.
• Add TLS/gRPC regression tests and document the recovery behavior. No public API, configuration, protocol, or dependency changes.

An interrupted TCP open remains denied. Recovery accepts new traffic; it does not replay earlier decisions or established connections. The existing recovery-window policy is unchanged: a second outage after prolonged idle can still terminate the same pending accept.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[johntmyers]

gator-agent

PR Review Status

This focused fix is project-valid against issue #3396. The initial code review found no blocking issues, and no carried findings exist.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Focused sandbox-backend recovery fix for the validated failure path described in bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited #3396; the author has repository write authority.
• Docs: No direct user-facing UX change; the runtime recovery contract is documented in architecture/sandbox.md.
• Checks: Current-head Branch Checks, Helm Lint, DCO, and Trivy Changes are green; required E2E has not yet been dispatched.
• E2E: test:e2e is required for sandbox lifecycle and network mediation behavior and will be dispatched next.
• Head SHA: 6c5931f4595f2697d4b55c8ed56da16dcb068506
• Base SHA: 7e7a8d5610f336f5f7f9f60da0951adbf295475d
• Merge base SHA: e92c15cb27e96ef0e1e08f1500f7f7cbd9a1af6e
• Patch ID: 668b3100d522848a80f95d2f0c5031050c5acf70
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[github-actions]

Label test:e2e applied for 6c5931f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The follow-up review found no blocking issues. The new head only merges current main; the author patch is unchanged from the previously reviewed version, there are no carried findings, and required E2E work is queued for this head.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Gator metadata

• Validation: Focused sandbox-backend recovery fix for the failure path described in bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited #3396; the author has repository write authority.
• Docs: No direct user-facing UX change; the runtime recovery contract is documented in architecture/sandbox.md.
• Checks: Current-head Branch Checks and Branch E2E Checks are queued or running; Helm Lint, DCO, and Trivy Changes are green.
• E2E: test:e2e is applied, and the current-head Branch E2E workflow is queued.
• Head SHA: d59c59e76f809ca0907a56b5a9910ffb7feb6e82
• Base SHA: 9708ba999985ca18ffec5a3310f11e4b04b615f6
• Merge base SHA: 9708ba999985ca18ffec5a3310f11e4b04b615f6
• Patch ID: 2298221b7183b01bc1951f6ee30152afce5983ae
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: 6c5931f4595f2697d4b55c8ed56da16dcb068506
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[johntmyers]

gator-agent

PR Review Status

The follow-up review found no blocking issues. The new author delta updates the provider-readiness E2E fixture to build a custom trust bundle while preserving the distroless supervisor image; upstream-only changes were excluded, and there are no carried findings.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Gator metadata

• Validation: Focused sandbox-backend recovery fix for the failure path described in bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited #3396; the author has repository write authority.
• Docs: No direct user-facing UX change; the runtime recovery contract is documented in architecture/sandbox.md.
• Checks: Current-head Branch Checks and Branch E2E Checks are running; Helm Lint, DCO, and GPU E2E are green.
• E2E: test:e2e is applied, and the current-head Branch E2E workflow is running.
• Head SHA: 07d90c88f34d822309c6d1410fac08d1b20af358
• Base SHA: c31ff5743fd930e227a9fba98bf4fe8f5f9003c9
• Merge base SHA: c31ff5743fd930e227a9fba98bf4fe8f5f9003c9
• Patch ID: 46b776c63d7b84ac26b60e3e4c9b1ee299cfb77b
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: d59c59e76f809ca0907a56b5a9910ffb7feb6e82
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[johntmyers]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator was watching the pipeline after a follow-up review found no blocking issues.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata

• Head SHA: 5dc83025f44ff297d425e1ecd768dcde27fdca13
• Gator payload: 9