Skip to content

fix(extensions): resend calls an extension service did not process - #5735

Open
dvavili wants to merge 1 commit into
ai-openshell:mainfrom
dvavili:fix/5713-extension-call-retry/dvavili
Open

dvavili wants to merge 1 commit into
ai-openshell:mainfrom
dvavili:fix/5713-extension-call-retry/dvavili

Conversation

@dvavili

@dvavili dvavili commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

A gateway interceptor or supervisor middleware call fails closed when its connection closes as an extension service (like Agent Policy Fabric) restarts, even when a replacement instance is already serving. This PR resends a unary extension call when the transport reports the service did not process it.

Related Issue

Closes #5713

Changes

  • openshell-extension-core: add call_with_retries, which resends a call when the transport shows the service did not process it: a connect failure, a request cancelled before it was sent, a graceful GOAWAY, or a refused stream. Attempts are 5 ms to 100 ms apart and stop at the caller's deadline, returning the last transport failure in place of a timeout.
  • openshell-gateway-interceptors: Evaluate, SnapshotProviderProfiles and Describe go through call_with_retries with the binding's timeout as the deadline.
  • openshell-supervisor-middleware: EvaluateHttpRequest, Describe and ValidateConfig do the same.
  • Evaluations for a fail-open binding are not resent, so a service that is down does not delay them.
  • A call the service may have processed is not resent, and streaming calls are unchanged.
  • Docs: gateway interceptors and supervisor middleware operations pages describe the behavior.

Release Notes

Gateway interceptor and supervisor middleware calls no longer fail when an extension service is replaced by a rolling restart. A call that the service did not process is sent again within the call's timeout.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

cargo fmt --all -- --check, cargo clippy --all-targets -- -D warnings and cargo test pass for the three changed crates. The tests include a call that the service received and whose connection is then reset, which is delivered once, and a fail-open binding that is not retried.

Also run on a kind cluster against a build without this change, before the review changes to the deadline and to fail-open bindings, with egress requests and sandbox creations in loops while an extension service that serves an interceptor and a middleware is restarted:

Scenario Traffic Without this change With this change
Rolling update, new instance ready before the old one stops (40 restarts) Egress requests 5 of 98,974 failed 0 of 99,179 failed
Rolling update, new instance ready before the old one stops (40 restarts) Sandbox creations 2 of 15,563 failed 0 of 15,654 failed
One of two instances killed without a grace period (20 kills) Egress requests 1 of 31,412 failed 0 of 31,599 failed
One of two instances killed without a grace period (20 kills) Sandbox creations 3 of 4,951 failed 0 of 4,929 failed
Single instance down about 9 s, less than the 10 s middleware timeout (10 restarts) Egress requests 164 of 15,113 failed 10 of 14,937 failed
Single instance down about 9 s, longer than the 500 ms interceptor timeout (10 restarts) Sandbox creations 149 of 2,487 failed 128 of 2,452 failed

The interceptor used the default 500 ms timeout. The middleware was registered with a 10 s timeout, not the 500 ms default; with the default, the row for a single instance down about 9 s would show no improvement for egress requests either. The table counts calls that failed in the transport. When the service is down for longer than the call's timeout, calls still fail closed, as the last row shows.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026

@pimlock pimlock left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The design is sound: resending only when the transport proves the call never reached a handler is the same rule grpc-go and grpc-java use for transparent retry. I traced the hyper 1.9 / h2 0.4.19 / tonic 0.14 error paths and they hold up: is_canceled comes only from a still-queued envelope, and a remote GOAWAY(NO_ERROR) errors only streams above last_stream_id. I also ran a local harness against RemoteMiddlewareService. An abrupt close after the handler ran, a GOAWAY covering an accepted stream, and an Unavailable answer were each delivered exactly once. Ported to the base commit, the hand-over test fails, and at head it passed 200 of 200 runs.

Three things I'd like addressed before merge, inline below:

  1. Tie the retry window to the caller's deadline and keep the transport reason.
  2. Add a deterministic test that a call the service received is never resent.
  3. Move the middleware restart note out of the legacy accordion.

Optional nits

  • call_was_not_processed has no caller outside transport.rs, so it could drop out of the pub use in lib.rs (or become pub(crate)).
  • The PR description mentions a 10 s middleware timeout. The default is 500 ms (openshell-core/src/middleware.rs), so the rows in your results table may reflect a non-default configuration. Worth saying so.

/// `call` builds and sends one attempt. Wrap the returned future in the call
/// deadline: re-sending stops when that deadline drops the future, or after a
/// fixed window when there is none.
pub async fn call_with_retries<T, F, Fut>(mut call: F) -> Result<T, Status>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This retries ConnectionRefused until the outer tokio::time::timeout fires, which changes two things:

  • Fail-open bindings now wait out the full timeout when the service is down. Before this change, a down fail_open interceptor or middleware was bypassed immediately. Now every CreateSandbox waits the interceptor timeout (500 ms by default), and every egress request waits the middleware timeout, before the bypass. apply_failure_policy in runtime.rs and the apply_on_error branch in supervisor-middleware/src/lib.rs only run after the timed-out future resolves.
  • The logged reason changes from the real error to a timeout. Operators see evaluation timed out / middleware_timeout instead of tcp connect error: Connection refused, so a down service looks like a slow one.

Separately, a timeout longer than 5 s stops retrying at 5 s (UNPROCESSED_RETRY_WINDOW). gateway-interceptors.mdx says calls are resent "within the configured timeout" and that an endpoint fails operations only when it refuses connections "for longer than timeout".

Suggestion: pass the deadline into call_with_retries, for example call_with_retries(deadline, || ...). Stop when now + pause would cross it, and return the last transport Status, so the caller sees the real reason instead of a timeout. Then the 5 s constant only matters when there is no deadline. Whether fail-open bindings should retry at all, or only briefly, is your call. Either way, it'd be good to state the choice in the docs.

///
/// Every other failure, including a connection that drops after the request
/// was sent, leaves the outcome unknown and returns `false`.
pub fn call_was_not_processed(status: &Status) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The safety half of this function, never resending a call the service may have run, isn't pinned by any test:

  • The connecting guard is untested. Replacing both connecting = false; lines (170, 181) with no-ops leaves openshell-extension-core 27/27 and the remote tests 3/3 passing. Every unit test builds a synthetic Status. The hand-over test drains handlers before each shutdown, so it never produces an unknown outcome.
  • The GOAWAY / REFUSED_STREAM branch is only covered by chance. With that branch disabled, the hand-over test still passed 2 of 10 runs.

The classifier is correct against the current hyper/h2/tonic. But it depends on how those crates wrap their errors, and a dependency bump that widens the chain would start delivering post_commit interceptor or side-effecting middleware calls twice without any test failing.

Could you add:

  • A real-socket test whose handler records receipt, after which the server task is aborted or the connection is reset mid-call. Assert the call returns an error and handled == 1.
  • Unit cases that return false: a hyper::Error wrapping io::Error(ConnectionReset) on an established connection, and a GOAWAY that is local or has a reason other than NO_ERROR.


Upgrade the gateway before deploying the service. Recreate sandboxes whose supervisors do not support HTTP hooks. To support both contracts during migration, use separate service registrations.

When a service restarts, OpenShell sends an `EvaluateHttpRequest`, `Describe`, or `ValidateConfig` call again, within its deadline, if the transport shows the service never started processing it: the connection attempt failed, the call was still queued when the connection closed, or the service refused it while shutting down gracefully. A call that reached a handler is never sent twice. Streaming RPCs are not sent again.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph is inside <Accordion title="Legacy HTTP Hooks">, but Describe and ValidateConfig apply to every middleware service. Someone on the recommended http-hooks contract won't open the legacy accordion, so they won't learn that:

  • Describe and ValidateConfig are resent on restart.
  • EvaluateHttpRequestHook and EvaluateHttpResponseHook are streaming, so they are not resent and still fail on restart.

Suggestion: move it under ### Failures (line 126), name the HttpHooks RPCs explicitly as not resent, and carry over the "start the new instance before the old one stops accepting connections" guidance from gateway-interceptors.mdx:151.

Not for this PR: the streaming HttpHooks RPCs still fail closed on restart, and they're the replacement for EvaluateHttpRequest once it's removed in 0.2.0. Worth a follow-up issue.

A gateway interceptor or supervisor middleware call failed closed when
its connection to the extension service closed as the service
restarted, even with a replacement instance already serving.

Unary extension calls are now sent again when the transport reports
that the service did not process them: a connect failure, a request
cancelled before it was sent, a graceful GOAWAY, or a refused stream.
Attempts back off from 5 ms to 100 ms until the call's timeout, and
the last transport failure is reported in place of a timeout. A call
the service may have processed is not sent again, and evaluations for
a fail-open binding are not sent again, so a service that is down does
not delay them.

Closes ai-openshell#5713

Signed-off-by: Divya Vavili <dvavili@nvidia.com>
@dvavili
dvavili force-pushed the fix/5713-extension-call-retry/dvavili branch from 5325c7a to 145382b Compare November 21, 2026 05:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: extension calls fail closed while a gateway interceptor or supervisor middleware service restarts

2 participants