stack-auth: a cancelled token refresh locks DeviceSessionStrategy out until restart #1170

Background

stack-auth is the Rust crate that gets the bearer tokens CipherStash services need: ZeroKMS, our key service, and CTS, our token service. Its AutoRefresh engine caches a token and refreshes it before or after it expires. Callers share one strategy across every request, and concurrent callers wait on a single refresh rather than starting their own.

Several strategies sit on top of AutoRefresh:

• AccessKeyStrategy: exchanges a long-lived access key.
• OidcFederationStrategy: exchanges a JWT from an identity provider.
• DeviceSessionStrategy: uses a rotating OAuth refresh token from a device-code login, the ~/.cipherstash/auth.json profile. AutoStrategy falls back to it when there is no access key.

In async Rust, cancelling a call means dropping its future. That happens routinely: tokio::time::timeout, tokio::select!, or a web framework such as axum dropping a request handler when the client disconnects.

Problem

One cancelled refresh locks DeviceSessionStrategy out until the process restarts.

1. AutoRefresh::get_token takes the refresh token out of shared state before it starts the refresh: try_credential calls take_refresh_token() (auto_refresh.rs:361, device_session_refresher.rs:57-58).
2. The refresh token is put back only on the error paths (restore, auto_refresh.rs:552-556 and :609-613).
3. If the future is dropped at any .await in between, the refresh token is gone from memory. That covers the HTTP call to the token endpoint, saving the new token, and taking the state lock. CancelGuard::drop (auto_refresh.rs:164-171) only clears the in-progress flag and wakes waiters.
4. After that, try_credential returns None on every call.
   • On the blocking path (token already expired), every later call returns Expired immediately.
   • On the non-blocking path (token still usable), callers get the old token until it expires, then Expired.
   • It never retries. DeviceSessionStrategy is built with an in-memory token and no store (device_session_strategy.rs:228,272), so nothing reloads auth.json.

Impact. A Rust service that uses device-session credentials, directly or through AutoStrategy, loses authentication for every request and tenant after one cancelled request lands on a refresh. A user closing a tab is enough. Every ZeroKMS call then fails until the service is restarted.

Worse case: the refresh token may already be spent. This one is plausible, not yet tested. If the refresh request reached the server before the drop, the refresh token has rotated upstream, but the new one was never saved: persist_refreshed didn't run, device_session_refresher.rs:65-114. auth.json still holds the old, spent token. A restart replays it, which by the code's own comment trips the identity provider's reuse detection and revokes the whole chain (device_session_refresher.rs:66-72). The user then has to log in again.

Why nothing catches it. The cancellation tests in auto_refresh.rs:1869-2033 (given_cancelled_refresh) abort a refresh but only assert that the next caller doesn't hang. The blocking-path test says so in its comment: "The credential is lost (refresh token was taken before the HTTP call), so the result is Expired" (:1898-1900).

Who is affected

Reproduced

In a throwaway test with a mock refresher that takes its credential the same way DeviceSessionRefresher does, the first call was cancelled with tokio::time::timeout:

• Rotating, blocking path: the next three calls each returned Err(Expired). The refresher was called once only.
• Rotating, non-blocking path: Ok(old token) until the clock passed expiry, then Err(Expired), with no retry.
• Non-rotating (access-key shape): the next call refreshed and returned the new token.

Proposal

The behaviour we want: cancelling a caller cancels only that caller's wait. The refresh it started either finishes, with the new token saved and installed, or fails and puts the credential back.

1. Main fix, native targets. Run the whole refresh (the exchange, saving the new token, installing it, waking waiters) in a spawned task that callers only await. A dropped caller then gives up its wait, and the rotation is still saved. That covers the already-rotated case too. This needs the engine's shared parts behind an Arc. On wasm, where the engine can't spawn, hosts drive it with block_on, which already runs to completion.

2. Smaller step that could ship first. Have CancelGuard hold the credential and put it back on drop. This fixes the in-memory lock-out, but not the case where the token already rotated upstream.

3. Close the related hang. CancelGuard wakes waiters without the state lock, so a caller entering wait_for_in_flight_refresh at the moment of a cancel can wait forever (auto_refresh.rs:148-157, which already calls this out). Notify under the state lock, or bound the wait.

4. Tests that check the outcome, not just that nothing hangs. After a cancelled refresh on each path, the next call gets a valid token. That includes a cancel after the token endpoint has answered: the rotated token must be saved and used.

5. Soak tests. Step 4 pins single cases; a soak runs the overlap a real service sees.

   • Setup:
     • One shared AutoRefresh on a mock refresher that rotates its refresh token.
     • A mock identity provider that revokes the whole chain when a refresh token is reused, as the real one does.
     • Random latency and errors.
     • Hundreds of tokio tasks calling get_token in a loop, each call wrapped in a random timeout or a random select! cancel.
     • Tokio's paused clock and short token lifetimes, so hours of expiry and refresh cycles run in seconds.
   • Invariants:
     • every call that is not cancelled gets a valid token;
     • the identity provider never sees a reused refresh token;
     • the saved token is always the newest;
     • no caller waits past a fixed bound, which also catches the hang in step 3.
   • When it runs:
     • Every PR: a short seeded run.
     • Release time: a long run before a new stack-auth is published. That is a job release-plz.yml's release-crates waits on, plus a run on the Version Packages PR (changeset-release/main), because @cipherstash/auth ships this crate.
     • Recommended: the long run on PRs that touch auto_refresh.rs or the refreshers too, so a regression shows up at the commit that caused it.

Relationship to other work

• Go SDK: one cancelled or timed-out call closes the shared encrypt.Client for every later caller #1169 is the Go SDK's version of the same theme: a single cancelled call breaks a client shared by every caller. The mechanisms are different, so the fixes are separate.