feat(api)!: standardize mutation idempotency, results, and structured errors

User Story

As an API or SDK client, I want mutations to be safely retryable and to return structured outcomes and errors, so that transient failures do not create duplicate resources or require parsing human-readable strings.

Problem Statement

OpenShell mutations do not expose a consistent client request identifier or deduplication contract. Many mutation responses consist of a single boolean such as deleted, attached, or removed, which cannot distinguish accepted asynchronous work, already-absent state, or warnings. Errors generally use gRPC codes and text without standard structured details.

Impact / Why This Matters

Clients cannot safely retry a timed-out create or other non-idempotent mutation because they do not know whether the server committed it. Automation parses error strings for validation details and retry decisions. Boolean acknowledgements leave ambiguous behavior around asynchronous cleanup and partial completion.

Proposed Design

Define one mutation reliability contract:

Non-idempotent mutations accept a bounded UUID request_id with documented deduplication lifetime and payload-mismatch behavior.
Delete RPCs return google.protobuf.Empty or the affected resource and support allow_missing where idempotent deletion is appropriate.
Other one-boolean acknowledgements become empty success responses, the affected resource, or a typed outcome where multiple results are meaningful.
Validation, precondition, and retry guidance use standard structured gRPC error details such as BadRequest, ErrorInfo, and RetryInfo.
SDKs expose typed errors and retry-safe helpers without requiring string parsing.

Acceptance Criteria

Every non-idempotent public mutation is audited and either supports request_id or documents why it is naturally idempotent.
Request-ID validation, deduplication lifetime, replay behavior, and mismatched-payload behavior are documented and tested.
Single-boolean acknowledgements are removed or justified with an extensible typed outcome.
Delete operations have consistent not-found and allow_missing behavior.
Validation failures expose field-level structured details.
Retryable failures expose machine-readable reason and retry guidance where applicable.
All SDKs map structured errors without losing raw gRPC status details.
Status-code and retryability tables are published and migration notes cover response-shape changes.

Alternatives Considered

Rely only on server-generated transport request IDs. Those correlate logs but cannot deduplicate client retries. Keep boolean responses and add more fields later. This preserves ambiguity and makes the initial response shape harder to reconcile across SDKs. Adopt a full long-running-operations service. That is substantially larger than the immediate idempotency and result-shape problem.

Agent Investigation

The protobuf tree contains many one-boolean mutation responses and no common client-supplied idempotency field for gateway mutations. HTTP/gRPC request correlation exists but has different semantics. Cleanup acknowledgement behavior in #2999 is a related example.

Related: #2565, #2999. Source audit: https://gist.github.com/mrunalp/e80942c1544a0225ee588796a41ab30b.

🏗️ build-plan

Mutation reliability implementation plan

Issue: #3051

Type: feature with breaking API changes. Complexity: high. Confidence: medium;
durable replay and credential-sensitive operations need implementation-level
verification before their contracts can be finalized.

The direct implementation request authorizes this work. Acceptance and agent
workflow labels remain unchanged.

Sequence

Standard structured errors and SDK decoding. Preserve raw transport status,
unknown details, and metadata. Return field violations from shared validation
and machine-readable concurrency guidance. Publish error/retry tables.
Consistent delete requests with allow_missing and explicit success shapes.
Preserve asynchronous sandbox deletion as an affected resource/outcome.
Replace single-boolean acknowledgements and update consumers and migration docs.
Durable request admission and replay for public unary mutations. Use bounded,
nonzero UUIDs scoped to authenticated identity, workspace and method. Compare
canonical payload fingerprints. Reauthorize replay; never store credentials in
ordinary response records. Test concurrent duplicate requests, restart, payload
mismatch, cancellation, expiry and authorization changes on SQLite/PostgreSQL.
Exec launch deduplication with explicit stream semantics, credential/session
expiry and revocation handling, SDK request-ID helpers, full mutation audit.

Invariants

Transport correlation IDs do not deduplicate client retries.
Transient status does not establish that a mutation is safe to repeat.
A version conflict requires fresh state and a new mutation attempt.
Never expire or take over an unresolved admission record after effects may have
occurred. Return an uncertain outcome without blind retry guidance.
Replay must respect current authorization and resource/session lifetime.
Streaming exec cannot promise output replay without a persisted execution log.

Verification

Run focused server and SDK tests for each slice, repository unit tests and
pre-commit checks, all affected SDK CI lanes, and documentation validation.
Run sandbox E2E for lifecycle or exec changes and full local CI before PR creation.

Review evidence

The independent principal-engineer review confirms settings and provider
attachment mutations can advance revisions even for apparently identical writes.
Profile import/update return diagnostics with boolean flags; these need deliberate
typed outcomes. Sandbox deletion can acknowledge asynchronous cleanup. Supervisor
credential exchange RPCs require a separate contract from public client mutations.

Current status

The structured-error foundation is implemented in
PR #3313, commit
966d7ababf7e0bd3e2cf88dd34c456c4265dbdaa. It adds standard error construction,
shared validation and concurrency details, typed decoding with raw-status
preservation in all four SDKs, and status/recovery documentation. The independent
code review found no blocking defects.

Local mise run pre-commit, mise run test, mise run ci, mise run go:ci,
mise run sdk:ts:ci, the complete 242-test Python suite, and documentation
validation passed. No live sandbox E2E was needed for this first slice because
runtime, lifecycle, policy enforcement, and deployment infrastructure did not
change. Lifecycle and exec follow-ups require their relevant E2E lanes.

Phase 2 is implemented in PR #3317,
commit f24eb09eb40482661fbf3f8bd6c74b7f44b16407, stacked on #3313. Nine public
single-boolean delete/removal/revocation responses now return DeletionOutcome;
requests explicitly opt into missing-target success with allow_missing.
Sandbox deletion preserves its owned worker and captured identity and separates
accepted cleanup from logical completion. Current authorization and backend
failures remain authoritative. All four SDKs, CLI/TUI consumers, and migration
docs are updated. Independent review found no remaining correctness blockers.

Phase 2 verification passed: mise run pre-commit, full mise run ci (including
the repository test suite), Go and TypeScript SDK CI, and strict documentation
validation (0 errors, 3 warnings). The server suite passed 1,521 unit tests with
8 existing ignored tests; Python passed 247 tests and TypeScript passed 108.
Live Docker conformance smoke and both workspace lifecycle tests passed via
OPENSHELL_E2E_DOCKER_TEST=workspace_lifecycle mise run e2e:docker, using an
isolated gateway and freshly built supervisor.

The phase-2 audit deliberately retains supplementary booleans in provider
attach/detach, profile import/update, and config update responses because those
responses also carry resources, diagnostics, or versions. Internal extension
contracts remain separate. The final mutation audit can revisit those richer
responses; they are not included in the nine single-boolean migrations.

Phase 3 is sequenced into independently reviewable slices after implementation
review found handler-specific authorization and interceptor side effects:

3a (implemented in PR #3321,
commit 5dc3c72940b13ca0da55a0b62ec792d71e50cf17, stacked on feat(api): return typed deletion outcomes with explicit missing-target semantics #3317):
durable admission and six workspace/template mutations.
Optional canonical nonzero UUIDs, 24-hour successful replay, permanent unresolved
admissions, owned bounded execution, and current authorization checks. Store
resource UUID/version references or deletion outcomes, never response snapshots.
Missing or changed original resources make replay unavailable, not executable.
3b (implemented in PR #3323,
commit 80415d1604988cf4fcfcfef8f13660fbe57232be, stacked on feat(api): add durable workspace mutation admission and replay #3321):
remaining ordinary unary mutations, including original-payload fingerprint
propagation through interceptors and replay-aware post-commit delivery.
Credential capabilities remain coordinated with phase 4: SSH session IDs are
credentials and rootfs staging tokens are single-use in-memory capabilities.

This sequencing does not claim that 3a completes phase 3. It avoids giving the
unreviewed remaining methods a generic replay contract. Successful records become
reusable after 24 hours; unresolved records are never stolen or expired. Clients
must reconcile uncertain effects before choosing a new ID. SQLite and live
PostgreSQL exercise the same store-backed state machine.

Phase 3a verification passed: pre-commit, full local CI (1,532 server unit tests,
9 ignored; 247 Python tests; Go SDK CI), TypeScript SDK CI (108 tests), and strict
docs validation (0 errors, 3 warnings). Twelve server tests were added, including
the opt-in live PostgreSQL scenario, which passed against a disposable PostgreSQL
18.6 container. Docker conformance smoke, both Rust workspace lifecycle tests,
and all six Python workspace API E2E tests passed. Independent review found no
remaining correctness blockers. The disposable gateway and PostgreSQL test
resources were removed after verification.

Credential capabilities, curated request-ID helpers,
the final mutation audit, and handler-specific structured-validation adoption
remain unfinished. None of these PRs closes this issue. The stack provides
deduplication for thirty explicitly opted-in ordinary unary methods, and no SDK
automatically retries mutations.
Exec launch admission is implemented in phase 4a below.
In particular, allow_missing alone does not make a later name-based retry safe
against same-name recreation.

Phase 3b implementation

The directly authorized slice is implemented in
PR #3323, commit
80415d1604988cf4fcfcfef8f13660fbe57232be, on
feat/3051-interceptor-mutation-replay/mrunalp, stacked on #3321. Verification
passed after direct human approval to resume from the retry-limit pause;
independent review found no remaining correctness or security blockers.
It adds the remaining 24 ordinary
user-callable unary mutations: sandbox lifecycle and attachments (6), services
(2), provider/refresh operations (6), profiles (3), and policy/config (7).

The reviewed receipt design uses explicit references and public scalar or
diagnostic fields, with a 64 KiB completed-receipt bound. Sandbox replay projects
the live original UUID so routine status reconciliation does not invalidate it.
Provider/profile/service resources retain exact-version guards. Refresh replay
requires the original provider/refresh UUID and authorization epoch. Policy/config
results preserve historical versions/counts and require the original sandbox
when scoped. Deletes preserve historical outcomes even after target/parent
removal. Handler-private facts capture identities from actual reads/writes,
never post-return name lookups.

Original client payloads are retained only in private in-memory middleware
extensions. Current interceptor modification and validation rerun on every
attempt; changing a request ID is rejected, changed effective payload/scope makes
replay unavailable, and replay suppresses post-commit observation. Observation
remains best-effort, not a durable outbox.

All 24 new adapters use domain-separated keyed fingerprints derived from existing
gateway JWT signing material or primary TLS private material. No new keys or
configuration fields are introduced. Missing material rejects before admission;
rotation fails closed in the same admission namespace. Replicas must share the
key material. The six phase-3a formats remain compatible. Public documentation
records this availability constraint and live-versus-historical response behavior.

Targeted verification passed: 25 active replay tests, the opt-in live PostgreSQL
restart/concurrency test, TypeScript SDK CI (108 tests), and documentation
validation (0 errors, 3 existing warnings). The disposable PostgreSQL test
container and its temporary data were removed.

The initial three pre-commit attempts failed. The first two encountered the local
sccache execution failure; the third bypassed that environment problem using
mise run --shell 'env RUSTC_WRAPPER= bash -c -o errexit -o pipefail' pre-commit
and reached Clippy. After explicit approval to resume, the three redundant
visibility declarations and wildcard import were fixed. Pre-commit and full
local CI now pass: 1,546 server unit tests (9 existing ignored), 247 Python tests,
108 TypeScript tests, all other repository tests, and Go SDK CI. Docker conformance
smoke, both Rust workspace lifecycle tests, and all 11 Python sandbox/workspace
API E2E tests passed, including the new replay/replacement regression. Disposable
gateways, sandboxes, and temporary networks were removed. The Docker supervisor build uses its
supported auditable mode to remove the broken compiler cache wrapper after nested
mise environment setup; repository tool configuration is unchanged.

After phase 3b, credential capabilities, exec/streaming semantics, curated SDK
request-ID helpers, and the final mutation audit remained phase 4. This slice
does not close #3051.

Phase 4 sequencing and first slice

The directly authorized exec-only first slice (4a) is implemented in
PR #3324, commit
49c3cacc048a575b7c7d970bb099bf22f8ad90c3, on
feat/3051-exec-launch-admission/mrunalp, stacked on #3323. Independent
implementation planning found that credential capabilities need different
authorization and locator handling, so they remain a subsequent 4b slice.

Phase 4a adds optional request IDs to ExecSandboxRequest for both exec RPCs.
Both share durable keyed admission before relay opening, but the initial
interactive Start has a separate RPC namespace. A deferred CAS finalizer belongs
to the producer, never the returned stream. Only a real remote exit persists a
nonsecret terminal marker and starts 24-hour retention. A synthetic timeout,
transport failure, cancellation without exit confirmation, or failed persistence
leaves a permanent unresolved claim. Duplicates return uncertainty or
REQUEST_STREAM_UNAVAILABLE; they do not relaunch, reattach, replay output, or
accept a second interactive input stream. The existing transport disconnect
semantics and empty-ID behavior remain intact.

Tests cover single admission/relay under concurrency, cancellation before and
after admission, pending/terminal restart, no pending expiry, nonzero exit,
synthetic timeout, missing exit, payload mismatch, current authorization,
interactive wire dispatch, and a live command-side-effect counter for both RPCs.
Update API errors and gateway architecture docs; no gateway configuration,
deployment, process identity, or LSM enforcement changes are intended.

Phase 4a verification passed: pre-commit (including the commit hook), full local
CI (1,553 server unit tests, 9 ignored; 247 Python tests; Go SDK CI), TypeScript
SDK CI (108 tests), and docs validation (0 errors, 3 existing warnings). Seven
new admission tests cover the exec contract, including fail-closed completion
when the finalizer loses its CAS. The existing shared admission regression also
passed on disposable PostgreSQL 18.6. Independent final review found no blockers.

Docker conformance smoke and both Rust workspace lifecycle tests passed. All 16
selected Python exec-admission/sandbox/workspace API E2E tests passed, including
five new exec cases, with 159 unrelated tests deselected. The first Python run
exposed a test-lifecycle dependency: interactive timeout cleanup waits for client
input to close after the synthetic exit event. The test now closes its input at
that event; the production timeout transport behavior is unchanged. All 16
passed on rerun. Disposable gateways, sandboxes, networks, and PostgreSQL test
data were removed. The PR records the complete local E2E attestation.

Phase 4b must preserve reusable SSH token semantics and prevent replay after
revocation/expiry/reaping without storing bearer-token UUIDs in receipts. Token-only
revocation needs lookup-independent admission with captured workspace authorization.
Rootfs staging needs a nonsecret registry/slot locator and must fail closed after
consumption, expiry, restart, or routing to another process-local registry.
Curated SDK helpers, the final mutation audit, and remaining structured-validation
adoption follow separately. Phase 4a does not complete phase 4 or close #3051.

Revision 10 — phase-4a exec launch admission verified and published in stacked PR #3324.

Revision 9 — phase-4 independent planning and exec-only 4a implementation started.

Revision 8 — phase-3b verified and published in stacked PR #3323; full E2E attestation recorded.

Revision 7 — human-approved verification resumed; pre-commit and full local CI passed.

Revision 6 — phase-3b implementation reviewed; verification paused at the three-attempt limit.

Revision 5 — phase-3b reviewed receipt/interceptor design and implementation in verification.

Revision 4 — phase-3a implementation, independent review, and verification.

Revision 3 — phase-3 implementation review and explicit incremental scope.

Revision 2 — phase-2 implementation, reviewed scope decisions, and verification.

Revision 1 — reviewed implementation sequence and first-slice progress.

🏗️ build-from-issue-agent

Phase 2 is implemented in PR #3317, stacked on #3313.

Nine public single-boolean deletion/removal/revocation responses now carry typed outcomes and explicit allow_missing semantics. Sandbox deletion distinguishes accepted cleanup from logical completion; SDKs, CLI/TUI, tests, and migration guidance are updated.

Verification passed: pre-commit, full local CI, Go and TypeScript SDK CI, documentation validation, Docker conformance smoke, and both workspace lifecycle tests. Independent review found no remaining correctness blockers.

The canonical plan comment now records the implementation, scope decisions, and verification. This PR does not close #3051: durable request admission/replay and exec launch deduplication remain phases 3–4.

🏗️ build-from-issue-agent

Phase 3a is implemented in PR #3321,
commit 5dc3c72940b13ca0da55a0b62ec792d71e50cf17, stacked on #3317.

The six workspace/template mutations now accept optional UUID request IDs for
durable at-most-once admission and 24-hour success replay. Current authorization,
original workspace/resource identities, permanent unresolved claims, per-caller
storage quotas, and reference-only replay prevent unsafe blind retries.

Verification passed: pre-commit, full local CI (1,532 server unit tests, 9 ignored;
247 Python tests; Go SDK CI), TypeScript SDK CI (108 tests), strict docs validation,
and the live PostgreSQL backend scenario. Docker conformance smoke, two Rust
workspace lifecycle tests, and six Python workspace API E2E tests passed. Twelve
server tests and one wire-level E2E regression were added. Independent review
found no remaining correctness blockers. Full E2E attestation is on the PR.

Updated architecture/gateway.md and docs/reference/api-errors.mdx, including
the exact supported RPCs, retention, quotas, uncertainty and migration contract.

This does not complete all of phase 3 or close #3051. Remaining ordinary unary
mutations need interceptor-aware admission and post-commit semantics; credential
capabilities, streaming exec, curated request-ID helpers and the final mutation
audit remain in the canonical plan. Lifecycle/workflow labels are unchanged.

🏗️ build-from-issue-agent

Phase 3b is implemented in PR #3323, commit 80415d1604988cf4fcfcfef8f13660fbe57232be, stacked on #3321.

Twenty-four additional ordinary unary mutations now support optional request IDs with typed, bounded replay receipts and keyed fingerprints for potentially sensitive payloads. Intercepted retries retain their original identity, rerun current modification/validation, and do not repeat post-commit observation. The stack now supports thirty ordinary user-callable unary methods; empty IDs retain legacy behavior.

Verification passed after the human-approved retry-limit continuation: pre-commit (including the commit hook), full local CI (1,546 server unit tests, 9 existing ignored; 247 Python tests; Go SDK CI), TypeScript SDK CI (108 tests), documentation validation, the opt-in live PostgreSQL scenario, Docker conformance smoke, both Rust workspace lifecycle tests, and all 11 Python sandbox/workspace API E2E tests. Fourteen server tests and one live lifecycle/replacement regression were added. Independent review found no correctness or security blockers. The PR includes full E2E attestation.

Updated architecture/gateway.md, docs/reference/api-errors.mdx, and the canonical plan comment. Disposable test resources were removed. Lifecycle and workflow labels are unchanged.

This PR does not close #3051. Credential capabilities, exec/streaming semantics, curated SDK request-ID helpers, the final mutation audit, and remaining structured-validation adoption are still pending.

fix(python): preserve wrapped RPC cleanup handling

fix(sdk): bind deletion waits to sandbox identity

feat(api): extend mutation replay through gateway interceptors (#3323)

fix(api): scope workspace request IDs by target

feat(api): add structured gateway errors and SDK decoding (NVIDIA#3313)

test(e2e): accept asynchronous stopped sandbox deletion

feat(api): add structured gateway errors and SDK decoding (NVIDIA#3313)

feat(api): add durable exec launch admission

feat(api): add durable workspace mutation admission and replay

feat(api): extend mutation replay through gateway interceptors (#3323)

fix(api): scope workspace request IDs by target

feat(api): add durable exec launch admission

feat(api): return typed deletion outcomes with explicit missing-targe…

feat(api): add durable exec launch admission

feat(api): add durable workspace mutation admission and replay (NVIDI…

feat(api): add durable exec launch admission

feat(api): add durable exec launch admission (NVIDIA#3324)

Navigation Menu

Repository navigation

feat(api)!: standardize mutation idempotency, results, and structured errors #3051

User Story

Problem Statement

Impact / Why This Matters

Proposed Design

Acceptance Criteria

Alternatives Considered

Agent Investigation

Mutation reliability implementation plan

Sequence

Invariants

Verification

Review evidence

Current status

Phase 3b implementation

Phase 4 sequencing and first slice

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat(api)!: standardize mutation idempotency, results, and structured errors #3051

Description

User Story

Problem Statement

Impact / Why This Matters

Proposed Design

Acceptance Criteria

Alternatives Considered

Agent Investigation

Activity

mrunalp commented on Sep 14, 2026

Mutation reliability implementation plan

Sequence

Invariants

Verification

Review evidence

Current status

Phase 3b implementation

Phase 4 sequencing and first slice

mrunalp commented on Sep 14, 2026

mrunalp commented on Sep 14, 2026

mrunalp commented on Sep 14, 2026

3 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions