Repository navigation
Description
User Story
I use OpenShell to run host-administration agents in sandboxes on SLES 16. Each agent gets an OAuth provider (client credentials from Keycloak), which the gateway renews, and calls a local OpenAI-compatible model server through an uninspected network endpoint.
I directly encountered connections from the sandbox ending in the middle of a request every few minutes, exactly when the gateway renewed the provider's token.
This affects every request that is open while a renewal happens: model calls of 30 s or more, and streaming responses through openshell forward.
Problem Statement
When a provider's credential value changes, every open TCP connection of the sandbox is closed. This happens with a gateway-renewed OAuth token, and also with openshell provider update. It also hits connections that carry no credential at all, to endpoints without protocol (L4 passthrough). The sandbox's policy is unchanged; the supervisor reports policy_changed:false provider_env_changed:true.
Impact / Why This Matters
-
Requests fail. The workload sees EOF on any request that spans a refresh. An agent loses its model call or tool result in the middle of a task.
-
It happens often. With 5-minute tokens (Keycloak's default) and refresh_before_seconds: 120, this happens every 4 minutes. With the default refresh_before of 300 s, it happens on every refresh sweep, once a minute.
-
Clients misbehave. A frontend whose stream breaks off resends its request while the first one is still being processed. In my case this mixed up an agent's conversation history.
-
The only workaround is long-lived tokens, with refresh_before_seconds well below their lifetime. That gives up short token lifetimes, and each refresh still drops every connection.
Acceptance Criteria
- A refresh that changes only a provider credential's value leaves the sandbox's open connections up, with the same policy, providers and bindings. This holds both for a gateway-driven OAuth renewal and for openshell provider update --credential.
- Requests made after the refresh carry the new credential value, including requests on an L7-inspected connection that was opened before it.
- Connections to endpoints without protocol are not closed by a credential refresh.
- A real policy change still closes connections as today: a new policy, a binding added or removed, a provider attached or detached, or fail-closed.
Reproduction Steps
- On the gateway host, serve a large file:
mkdir /tmp/srv && dd if=/dev/zero of=/tmp/srv/big bs=1M count=200 cd /tmp/srv && python3 -m http.server 8000
- Create a provider with a static credential:
openshell provider create --name refresh-test --type generic --credential TOKEN=one
- Create a sandbox with that provider and a policy that allows the host endpoint without
protocol(policy.yaml):version: 1 network_policies: host: name: host endpoints: - { host: host.openshell.internal, port: 8000 } binaries: - { path: /usr/bin/curl }
openshell sandbox create --name refresh-test --policy policy.yaml --provider refresh-test
- In the sandbox, start a slow download (about 3 minutes):
openshell sandbox exec -n refresh-test -- curl --limit-rate 1M -o /dev/null http://host.openshell.internal:8000/big - While it runs, change the credential's value:
openshell provider update refresh-test --credential TOKEN=two
- The download ends at once with a transfer error, not after 200 MB. The supervisor logs
Settings poll: config change detected [... policy_changed:false provider_env_changed:true]at that moment.
Suggested UX (if applicable)
None. No configuration or CLI change: open connections should survive a credential refresh.
Environment
- OpenShell: v0.1.8 (gateway, CLI and supervisor)
- OS: SUSE Linux Enterprise Server 16.1 (immutable), x86_64
- Deployment: single host, gateway as a systemd service, sandboxes via an external systemd compute driver. Providers: OAuth client credentials from Keycloak (5-minute tokens,
refresh_before_seconds: 120). Endpoint affected:host.openshell.internal:12400withoutprotocol(an OpenAI-compatible model server on the host).
Logs
# supervisor, one line per renewal, 4 minutes apart
Nov 20 07:40:06 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
Nov 20 07:44:06 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
Nov 20 07:48:07 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
# workload, a model call open across one of them
model endpoint: Post "http://host.openshell.internal:12400/v1/chat/completions": EOF
Activity
sdrahn commentedon Nov 20, 2026
Cause (v0.1.8). A credential refresh goes down the policy-reload path, and a reload closes every relay:
crates/openshell-supervisor/src/config_runtime.rs:277-282: a new provider env revision setsprovider_env_changed.config_runtime.rs:300-305: with a gateway-delivered policy,provider_env_changedalone makespolicy_runtime_changedtrue.config_runtime.rs:431-453: that runsreload_gateway_configuration_runtime, which ends inOpaEngine::reload_configuration_from_proto_with_pid.crates/openshell-supervisor-network/src/opa.rs:808: that function callsadvance_generation()on every reload, also for an identical policy.crates/openshell-supervisor-network/src/proxy/relay.rs:342andl7/relay.rs:258/274/289: each relay selects ongeneration_guard.wait_until_stale()and closes when the generation moves. That includes the L4 passthrough relay, which injects nothing.The printed
old_revision/new_revisionare the config revision. The provider env revision, which did change, is not in the message.sdrahn commentedon Nov 20, 2026
Proposed fix: install a value refresh without reloading the policy.
provider_env_changedis set, the policy hash is unchanged, and the prepared environment has the same static credential bindings as the installed one.ProviderCredentialState::install_preparedalready comparesstatic_credential_identities.ctx.provider_credentials.install_prepared(...)andprovider_readiness.credentials_installed(...), as the non-reload branch does atconfig_runtime.rs:639-646, and skip the reload. The generation stays, and no relay is woken.install_preparedkeeps existing clones on the live state.SecretResolverit was opened with (l7/relay.rs:66,:970). It should take it from the liveProviderCredentialStatefor each request, asl7/relay.rs:4586does withstate.resolver(). Otherwise an open tunnel goes on injecting the old value. I haven't traced every use of the captured resolver.A smaller first step, if (3) needs more design: keep the reload, but have relays that inject nothing (L4 passthrough,
proxy/relay.rs:342) wait on a policy generation that a credential-only change does not advance. That alone fixes uninspected endpoints such as the model server above.mr-miles commentedon Nov 20, 2026
I also found this breaking workloads - it's fixed in #5361 . Anything causing a policy reload (e.g. approving a policy update suggestion) caused all the connections to be dropped which is very confusing for the agent.
sdrahn commentedon Nov 20, 2026
@mr-miles Thank you, I did not find #5361 during my search before filing the issue.
mr-miles commentedon Nov 20, 2026
Hope it is useful! I investigated because it made the sandboxes unusable, agents were spending tokens investigating the network and getting confused. - which sounds like your experience. I found that the supervisor closed unchanged connections at every opportunity so fixing it was involved. But it has improved things massively for me.
Was hoping to get more traction from the maintainers (I don't understand how anyone could use it IRL as-is!) but I can only guess that their prod workloads use static policies that dont change and so they havent encountered these problems.
sdrahn commentedon Nov 20, 2026
Yes, very useful. Thank you. I agree the current state is not really usable. I have actually put in some workarounds in my framework so I can actually work atm - essentially limiting the renewals and tuning things in a way they do not collide with other timeouts, including extending token lifetimes. I would love to remove the workarounds quickly.