GetInferenceRoute intermittently returns NOT_FOUND (~51% of calls), aborting dcode -n non-interactive turns #2915

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Aug 25, 2026
2. elezar commented on Aug 25, 2026

   

   elezar

   on Aug 25, 2026

   Member

   More actions

   📋 triage-agent

   Classification: needs-information

   The report shows a material failure pattern, but the supplied gateway entries do not establish that the successful and NOT_FOUND RPCs queried the same route in the same authenticated workspace.

   GetInferenceRoute returns NOT_FOUND when the requested route is not configured in the resolved workspace. Also, HTTP status 200 with gRPC status 5 is expected gRPC response framing; the gRPC status is the operation result.

   Please provide, for one adjacent success/failure pair:

   1. Redacted GetInferenceRouteRequest values for both calls (workspace and route_name) and the full gRPC status/message.
   2. The authenticated principal or sandbox identity for both requests, or gateway diagnostics that establish it.
   3. The output of openshell inference get immediately after a failure, using the same gateway credentials and workspace as the NemoClaw integration.
   4. The route/provider setup commands and whether any concurrent process can set, delete, or switch the inference route or workspace.
   5. A retest on current OpenShell v0.0.111, with the NemoClaw/dcode versions used.

   This will distinguish a missing route or workspace/identity mismatch from an intermittent route-store defect.
3. elezar

   added

   state:needs-infoAssessment needs specific evidence or reproduction detailsAssessment needs specific evidence or reproduction details

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Aug 25, 2026
4. reqsgjh commented on Aug 26, 2026

   

   reqsgjh

   on Aug 26, 2026

   Author

   More actions

   Thanks for the detailed ask. I re-pulled every claim in the original report directly from
   our still-intact gateway log (it hasn't rotated since before this incident) rather than
   re-asserting from memory, and added what's happened since. Splitting this into
   re-verified original evidence (fact, re-derived independently just now) and new
   findings since filing (also fact) from our working hypothesis connecting them
   (clearly marked, not proven).

   Re-verified original evidence (2026-08-24)

   Recomputed straight from openshell-gateway.log, which still covers this window
   unmodified:

   • 104 total GetInferenceRoute calls, 53 NOT_FOUND = 51.0%, spanning 02:37-23:53 UTC —
     matches the original report exactly.
   • All 7 example pairs cited in the original report are still present, byte-for-byte,
     including the cited 23:53:04 pair:

     2026-08-24T23:53:04.440400Z  GetInferenceRoute  request_id=999a794f-24f0-468d-8665-81e5c0564a15  http=200 (OK)
     2026-08-24T23:53:04.441564Z  GetInferenceRoute  request_id=d302fc56-930c-4eaa-a392-7d4ea33b1fca  http=200 grpc.status_code=5 ERROR
     

     and the other six at 22:42:24, 22:43:08, 22:45:00, 22:45:10, 23:22:28, 23:51:50 — all
     independently re-confirmed present and unchanged.
   • The pattern was already fully deterministic then: of 52 pairs, 51 show
     first-call-succeeds / second-call-NOT_FOUND (~0.9-1.7ms apart, distinct request_ids each
     time), 1 pair both-fail, 0 pairs both-succeed.
   • Survived two Starting OpenShell server restarts logged at 02:22:08 and 02:35:05, as the
     original report noted.

   New findings since filing

   1. The identical pattern has continued, unchanged, ever since — including through a
      third gateway restart (17:58:36) and a complete provider/model swap. Today's sample:
      208 total calls, 108 NOT_FOUND (51.9%), 103/104 pairs still first-succeeds/second-fails,
      ~1.0-1.3ms apart. Same shape, different day, different models, different provider.

   2. But zero of those recent NOT_FOUND events correlate with an actual turn abort. Our
      own orchestration log shows every Unexpected error/managed non-interactive error
      we've had was confined to a separate, unrelated 2026-08-25 14:39-18:00 UTC window (a
      self-inflicted retry storm against our upstream provider, zero GetInferenceRoute calls
      even occurred in that window) — and zero turn aborts in the 200+ cycles since
      2026-08-25 19:40 UTC, despite the paired-NOT_FOUND pattern firing repeatedly and
      identically throughout. So whatever made Aug 24 turns abort, it doesn't reproduce here
      even though the RPC-level symptom you'd use to detect it does.

   3. What was different on 2026-08-24: we had three sandboxes (gjh-coder,
      gjh-reviewer, gjh-submitter) on one gateway, independently onboarded with three
      different models — exactly the three named in the original report. NemoClaw's own
      onboarding CLI told us why that's meaningful, verbatim from our terminal scrollback:

      $ nemo-deepagents gjh-coder connect
        Warning: gateway inference route (nvidia-prod/nvidia/nemotron-3-super-120b-a12b) differs
        from the recorded route for sandbox 'gjh-coder' (nvidia-prod/nvidia/nemotron-3-ultra-550b-a55b).
        Aligning the gateway to nvidia-prod/nvidia/nemotron-3-ultra-550b-a55b...
      

        Warning: Onboarding 'gjh-reviewer' will re-point the one shared inference route on
        OpenShell gateway 'nemoclaw' to nvidia-prod/nvidia/nemotron-3-super-120b-a12b. Affected
        registered sandboxes: 'gjh-coder'... OpenShell currently exposes this route per
        gateway, not per sandbox.
      

      Since 2026-08-25 19:40 UTC we've had all three sandboxes share one model on one route —
      no more onboarding-triggered realignment. That's also exactly when turn aborts stopped.

   4. Direct evidence of write contention on this store during onboarding: our gateway's
      own trace log has 14 sqlx::query: slow statement: execution time exceeded alert threshold WARNs. Two, both timestamped during onboarding activity:

      store.get_by_name object_type=inference_route workspace=default object.name=inference.local
        elapsed=1.529886341s
      store.put_if object_type=sandbox object.id=... object.name=gjh-coder workspace=default
        elapsed=5.062471243s
      

      A 1.5s single-row indexed SELECT and a 5s+ resource_version-guarded UPDATE against the
      same SQLite-backed objects store, both during the exact activity type (multi-sandbox
      onboarding) that produced the original report's 3-model reproduction. I checked whether
      these 14 slow-statement events line up with the 108 NOT_FOUND timestamps directly: only
      1 of 108 falls within 30 seconds of one. So this contention doesn't explain the raw
      per-call NOT_FOUND rate — it's a separate, real phenomenon that plausibly explains
      route mismatches during onboarding, not the deterministic per-call pairing itself.

   Our working hypothesis (not proven — flagging clearly as interpretation)

   Two independent things appear to be going on, and we conflated them in the original
   report:

   • (a) A deterministic, apparently benign RPC pattern: every logical
     GetInferenceRoute resolution fires two calls, one always succeeding and one always
     NOT_FOUND, ~1ms apart. Reproducible on demand with plain openshell inference get
     (no special flags) — that command prints both an "Inference" and a "System inference"
     section from one invocation, and shows "System inference: Not configured" for us without
     ever surfacing an error. We think the second call in each pair is a distinct, and on our
     setup legitimately-unconfigured, secondary route scope — not a flaky re-read of the same
     route. This pattern has never once correlated with a turn abort in anything we've been
     able to observe.
   • (b) The actual cause of Aug 24's aborts: three sandboxes independently onboarding
     different models onto the single gateway-level route, with real (seconds-long, logged)
     SQLite write contention on that shared row during the churn. We can't prove (a) and (b)
     are unrelated — both surface at roughly the same "GetInferenceRoute gets called" moments
     — but (a) alone, without (b)'s multi-model contention, produces zero aborts in 200+
     cycles of continuous operation since.

   If that split is right, the fix isn't in the NOT_FOUND pairing itself — it's in making
   GetInferenceRoute resilient to reading a route that's mid-realignment during another
   sandbox's onboarding, or genuinely scoping routes per-sandbox as the sandbox-level config
   already implies.

   Your five points, directly

   1. Adjacent pair / redacted request values: see the re-verified 23:53:04 pair above.
      I can't give you the literal GetInferenceRouteRequest.workspace/route_name fields —
      our structured logging captures RPC method + status only, not request bodies, and
      openshell -vvv doesn't log payload-level detail either (checked directly, only TLS
      handshake detail at that verbosity).
   2. Authenticated principal: single shared identity for every call across all sandboxes
      — openshell whoami → Subject openshell-client, Provider mtls.
   3. openshell inference get right after a failure:

      Inference: Workspace: default | Provider: compatible-endpoint | Model: nemotron-3-ultra-free | Version: 13
      System inference: Not configured
      

      Version 13 = 13 SetInferenceRoute writes logged against this one row total.
   4. Setup commands / concurrency: nemo-deepagents <sandbox> connect / onboard /
      rebuild against our three sandboxes — nothing else touches inference set/update/delete. But by the gateway's own design, onboarding one sandbox does
      concurrently affect the shared route used by the others — see the warnings quoted
      above. Our own orchestrator runs sandboxes sequentially under a flock, so
      cross-sandbox concurrency on our side is only in play during interactive
      onboarding/connect sessions, not our scheduled loop.
   5. Retest on v0.0.111: not reachable from here — nemoclaw update --check reports
      0.0.109 as the latest maintained version, bundling OpenShell 0.0.101. No upgrade path
      found to 0.0.111 through the maintained installer.

   Happy to pull anything else out of the gateway log — it's still intact back to before this
   incident started.
5. github-actions commented on Sep 9, 2026

   

   github-actionsbot

   on Sep 9, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
6. github-actions

   added

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Sep 9, 2026
7. johntmyers commented on Sep 17, 2026

   

   johntmyers

   on Sep 17, 2026

   Collaborator

   More actions

   📋 triage-agent

   Closing as obsolete after #3195. That merged change removed the managed inference control plane, route persistence, GetInferenceRoute, and the inference.local data path. The RPC and route records implicated by this report no longer exist; current inference workloads attach provider profiles and call provider-native endpoints.
8. johntmyers

   closed this as not plannedon Sep 17, 2026