Repository navigation
refactor(inference): remove managed inference routes - #3195
Merged
johntmyers merged 2 commits intoSep 9, 2026
Merged
Conversation
johntmyers
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 4, 2026 20:48
2 of 13 tasks
|
🌿 Preview your docs: https://nvidia-preview-pr-3195.docs.buildwithfern.com/openshell |
Collaborator
Author
Docker E2E Test AttestationLocal Docker E2E tests passed. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe Tests Executed
|
Collaborator
Author
Python E2E Test AttestationLocal Python E2E tests passed against a Docker-backed gateway. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe 81 tests under Tests Executed |
johntmyers
force-pushed
the
refactor/3172-remove-managed-inference/johntmyers
branch
from
September 4, 2026 21:12
6599bfe to
36b3abd
Compare
johntmyers
commented
Sep 5, 2026
johntmyers
left a comment
Collaborator
Author
There was a problem hiding this comment.
gator-agent
PR Review Status
The accepted issue and migration plan make this project-valid, and the broad removal is accompanied by updated SDKs, Fern docs, migrations, and replacement provider E2E coverage. One policy regression must be fixed before test dispatch: alternate-upstream legacy providers now grant access to the built-in public vendor endpoint even though their credential binding is withheld.
Action required: restore the endpoint-activation guard in provider policy composition and its OpenAI/Anthropic regression test.
Blocking findings:
GATOR-36b3abdf-01: alternate-upstream providers widen sandbox egress to the built-in public vendor endpoint
Carried findings:
- None
Non-blocking suggestions:
- None
Gator metadata
- Validation: linked accepted issue #3172 explicitly authorizes this cross-cutting managed-inference removal
- Docs: Fern docs and migration guidance updated
- Checks: current branch and Helm gates are green; required E2E has not yet been dispatched
- E2E:
test:e2erequired after review feedback is resolved - Head SHA:
36b3abdf4b313699c01f226b46836bfe2f6ec449 - Base SHA:
c96b9bff47f719e6f6bec21edd9ed1c33e3ba3ca - Merge base SHA:
8605943a2d5fec5707251fc67bba718cb9dcacb2 - Patch ID:
c581a82c7882eb073fa155abb32823afbce1deea - Gator payload:
8 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
|
Label |
johntmyers
commented
Sep 5, 2026
johntmyers
left a comment
Collaborator
Author
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @johntmyers. I checked your update restoring provider_profile_endpoints_are_active at the shared provider-policy composition point and the OpenAI/Anthropic alternate-upstream regression coverage. The prior vendor-egress finding is resolved, its review thread is closed, and the bounded follow-up review found no new blockers. Required E2E has been dispatched for this head.
Blocking findings:
- No blocking findings remain
Carried findings:
- None;
GATOR-36b3abdf-01is resolved by this head
Gator metadata
- Validation: linked accepted issue #3172 explicitly authorizes this cross-cutting managed-inference removal
- Docs: Fern docs and migration guidance updated in the reviewed PR
- Checks: branch and Helm gates are green; required current-head E2E attempt is queued
- E2E:
test:e2eapplied; E2E Label Help requested re-run all jobs; run33984167072was rerun and is queued - Head SHA:
3045866bde58a3402792ba3762ff5d4cd1dbe183 - Base SHA:
c96b9bff47f719e6f6bec21edd9ed1c33e3ba3ca - Merge base SHA:
8605943a2d5fec5707251fc67bba718cb9dcacb2 - Patch ID:
c139a1f471d12a710f86968cf8a5b05b9eb92ebb - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
36b3abdf4b313699c01f226b46836bfe2f6ec449 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:watch-pipeline
johntmyers
force-pushed
the
refactor/3172-remove-managed-inference/johntmyers
branch
2 times, most recently
from
September 8, 2026 23:02
c11e397 to
49b11d2
Compare
johntmyers
commented
Sep 8, 2026
johntmyers
left a comment
Collaborator
Author
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @johntmyers. I reviewed the rebased author patch against the previously reviewed effective change, including the conflict-resolution delta on current main. The prior alternate-upstream isolation fix remains intact, no blocking findings remain, and current-head Branch Checks and required E2E are running.
Blocking findings:
- No blocking findings remain
Carried findings:
- None;
GATOR-36b3abdf-01remains resolved at this head
Gator metadata
- Validation: linked accepted issue #3172 explicitly authorizes this cross-cutting managed-inference removal
- Docs: Fern docs and migration guidance remain present in the rebased effective patch
- Checks: current-head Branch Checks and required E2E are in progress; Helm Lint is green
- E2E:
test:e2eis applied and current-head run34288808324is in progress - Head SHA:
49b11d251600d2d429faeb52c5a9549acfe57be3 - Base SHA:
118b250f015af84ce4a225cd45d1ce7b64bb54cb - Merge base SHA:
118b250f015af84ce4a225cd45d1ce7b64bb54cb - Patch ID:
5628f403f321ea3788d864c50f6047f43a55c6a8 - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
3045866bde58a3402792ba3762ff5d4cd1dbe183 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:watch-pipeline
9 of 10 tasks
Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
johntmyers
force-pushed
the
refactor/3172-remove-managed-inference/johntmyers
branch
from
September 9, 2026 18:33
49b11d2 to
314ae2b
Compare
drew
approved these changes
Sep 9, 2026
pimlock
approved these changes
Sep 9, 2026
johntmyers
enabled auto-merge
September 9, 2026 18:40
johntmyers
deleted the
refactor/3172-remove-managed-inference/johntmyers
branch
September 9, 2026 19:01
6 tasks done
Collaborator
Author
Monitoring CompleteMonitoring is complete because this PR has merged. Final status: The last reviewed head had no blocking findings; the PR was later merged at head I removed the active Gator metadata
|
This was referenced Sep 17, 2026
Closed
st-gr
added a commit
to st-gr/openshell-driver-kyma
that referenced
this pull request
Sep 18, 2026
NVIDIA/OpenShell#3195 ("refactor(inference): remove managed inference routes", merged 2026-09-09) deletes `inference.local`, the managed route control plane, and the openshell-router crate. It is in no release yet -- v0.0.116 was tagged 2026-08-28 -- but it will be. This driver injects ANTHROPIC_BASE_URL=https://inference.local into every sandbox and relies on the supervisor's L7 proxy to substitute the real credential. When a release carrying #3195 lands, that host stops resolving. Nothing would have warned. check-proto-drift.sh compares the ComputeDriver contract and check-pin-status.sh compares image digests; this is an ARCHITECTURAL change behind an UNCHANGED wire contract, so it would have arrived looking like a routine image bump and surfaced as a confusing red smoke days later. scripts/check-inference-local.sh probes a sentinel file at the resolved upstream tag and emits INFERENCE_LOCAL_PRESENT / _REMOVED / _CHECK, wired into the sync's detect job. Unlike its sibling checks it FEEDS THE VERDICT: an architecture change nobody noticed is the failure mode, so it opens a PR even when protos and digests are unchanged. The hints are deliberately concrete -- the three things that break here, the two upstream issues, and the replacement workflow -- because the value is turning a confusing failure into a named, expected event with somewhere to start. Worth recording: the credential-swap property SURVIVES upstream. A provider profile injects a placeholder and the proxy substitutes the real key at the profile endpoint. What changes is the path, not the trust boundary. An earlier reading of the closing comment suggested the agent would hold real credentials; the architecture docs say otherwise. Two bugs found while testing both branches rather than just the happy one: - a bare `body=$(curl ...)` aborted under proto-lib.sh's `set -e` before $? could be read, so the REMOVED branch never ran -- the only branch that matters. - the status probe used `curl -f`, which exits non-zero on a 404 while still writing the code, so `|| echo 000` concatenated into "404000". Verified: PRESENT for v0.0.116, REMOVED with hints for main, exit 0 both ways; workflow parses as valid YAML. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remove OpenShell's managed inference routing control plane and
inference.localdata path. Inference workloads now use explicitly imported, use-case-specific provider profiles attached to sandboxes and call provider-native endpoints directly;inference_capableremains informational for future observability hooks.Related Issue
Closes #3172
Changes
proto/,crates/openshell-core/,crates/openshell-server/, and SDKs: remove inference-route APIs, commands, generated clients, auth scopes, and persisted route behavior.crates/openshell-supervisor-network/andcrates/openshell-sandbox/: removeinference.local, request-shape matching, route refresh/caching, and inference-route supervisor configuration.crates/openshell-router/: remove the built-in privacy router crate and its packaging/dependency surface.crates/openshell-server/migrations/: delete legacyinference_routeobjects during SQLite and PostgreSQL upgrades.crates/openshell-providers/: retain provider aliases and Vertex constants, keepinference_capableinformational, and preserve endpoint credential-binding guards.e2e/: replace managed-route tests with explicitly imported OpenAI-style and Anthropic-style profile fixtures against native endpoints, including a negativeinference.localassertion.docs/,architecture/,examples/local-inference/, and agent skills: document the breaking migration and provide human-readable Now/After workflows.Deviations from Plan
None — implemented as planned in the approved issue comment and follow-up decisions.
Testing
mise run pre-commitpassesmise run testpassesmise run cipassesmise run e2e:dockerpassesmise run e2e:pythonpasses (86 passed, 81 OIDC-only skipped)Tests added:
x-api-keyprofiles calling native endpoints, secret substitution/model body assertions, and negativeinference.localbehavior.Checklist
Documentation updated:
docs/sandboxes/inference-routing.mdx: provider-backed inference workflow, breaking migration, and Now/After UX.docs/get-started/tutorials/andexamples/local-inference/: native endpoint examples using explicitly imported profiles.architecture/, provider/security/observability/reference docs, and agent skills: remove managed-route assumptions and align troubleshooting/workflows.