Skip to content

fix(server): Codex usage-limit stops keep their reset time - #18721

Open
voltcrash wants to merge 2 commits into
pingdotgg:mainfrom
voltcrash:fix/codex-usage-limit-reset
Open

voltcrash wants to merge 2 commits into
pingdotgg:mainfrom
voltcrash:fix/codex-usage-limit-reset

Conversation

@voltcrash

@voltcrash voltcrash commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Codex can refuse a turn while its last quota update still reads 99%, or before the session receives any quota notification. Those failures lost a reset time the server already knew, hiding the existing resume-at-reset and snooze actions. Fixes #14169.

The reset selection prefers current exhausted windows, accepts 99% only when none is fully exhausted, and discards expired observations. Both external CLI and managed Codex sessions can use the provider instance’s published limits before any main-allowance session data arrives. That fallback remains provisional: the first session notification replaces or clears it. A known session-derived reset is preserved, and later session data can fill an unknown reset. Model-specific quota buckets cannot claim the main allowance.

This takes over the original work by voltcrash in #18721, preserves the contributor commits, and merges current main. The original fallback-on-null behavior was replaced because stale published data could become a permanent reset. No client or contract changes.

Verification:

  • PR-added cases reduced from six to five in the existing adapter/helper test files; combined suites: 195 original → 194 passing.
  • With main’s production code restored, targeted coverage failed for the 99%, published fallback, and expired-window behaviors. The contributor implementation failed the stale-reset regression. The earlier takeover also failed when stale published data existed before the refusal. Both before/after-refusal replay variants now pass; the former emits unknown reset → correct session reset.
  • pnpm exec vp test run apps/server/src/provider/codexUsageLimits.test.ts apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts --maxWorkers=2: 194 passed.
  • Codex driver and deterministic orchestrator integration selection: 38 passed, 120 unrelated cases skipped. Repeated on the final implementation.
  • Targeted lint and formatting pass; two pre-existing unused-variable warnings remain. Server tsc-rs --noEmit passes with Effect suggestions only. git diff --check passes.

Proof is deterministic app-server replay through the real adapter, not a live quota refusal. No screenshots apply to this server-only change. Real accounts, macOS, Windows, iOS, Android, and Electron desktop were not verified on this headless Linux VM. Existing historical failures are not repaired.

Implementation: GPT-6.1 Sol; tests: GPT-6 Astra; independent reviews: Claude Opus 5.5 and GPT-6 Sol. Harness: Codex / Claude in T3 Code.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 12, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 98815ea

Macroscope's review found this PR approvable — This is a focused Codex server bug fix that restores reset timestamps for existing usage-limit recovery, including 99% and pre-notification cases. The implementation is localized, uses the existing published limits contract, and is covered by targeted tests without changing defaults, schemas, or quota enforcement.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
Commits

Reviewing files that changed from the base of the PR and between b6fa42a and a8abab3.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 6cb3476 and b6fa42a.

Files selected for processing: 3
  • apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
  • apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts
  • apps/server/src/provider/Drivers/CodexDriver.ts

Limit details: You’ve used all 10 included reviews currently available.

Walkthrough

Codex reset selection now evaluates usage windows against the current time. The adapter uses session reset data when available and published usage limits when no session snapshot exists. Session notifications can update a failed-turn error’s reset time.

Changes: Codex usage reset resolution

Layer / File(s) Summary
Reset calculation and provider tests
apps/server/src/provider/codexUsageLimits.ts, apps/server/src/provider/codexUsageLimits.test.ts
The calculation filters windows whose reset time has passed. It selects windows at 100% usage, or windows at 99% or higher when none are at 100%. Tests cover selection at a supplied time.
Failure reset resolution and session updates
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts
The adapter resolves reset times from session snapshots before published usage limits. It tracks the reset source and applies session updates to limited-turn errors when the reset value changes.
Provider wiring and adapter scenarios
apps/server/src/provider/Drivers/CodexDriver.ts, apps/server/src/provider/Drivers/CodexManagedProvider.ts, apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
Both provider drivers pass usage limits from the current snapshot. Replay tests cover near-full and published resets, stale published limits, and later error-item updates.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

Possibly related PRs

Suggested labels: bugfix

Suggested reviewers: t3dotgg

Merge Risk: Minimal · previous review b6fa4

The reviewed reset-time behavior has no identified issue requiring a fix before merge. Normal validation remains appropriate.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 98815

The change is narrowly scoped to reporting when Codex usage limits reset. No new permissions or cross-account access were identified. Combined interruption and recovery scenarios remain incompletely covered.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated new influence is on reset metadata for usage-limit failures in sessions belonging to the captured provider instance, including tracked failures awaiting late enrichment. The getter does not accept an instance selector or expose another instance’s snapshot.

Trust Boundaries and Controls

  • observed — Existing notification processing excludes explicitly non-Codex quota buckets, ignores non-finite usage percentages, clamps percentages, and normalizes valid epoch resets. Published limits originate from the existing instance status path. The PR reuses this observational data without adding an authority-bearing sink.
Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Linked Issues check Warning Issue #14169 requires the composer banner to show a known reset time and enable Resume at reset and Snooze until reset. The PR adds session-first and published-limit reset selection, the 99% fallback,… In the terminal usage-limit handler, keep a non-null event.failure.resetAt. Use the derived session or published reset only when the incoming failure reset is null. Add a regression test where the terminal failure has a known reset, the s…
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title "fix(server): Codex usage-limit stops keep their reset time" describes the main change: the server now keeps a usage-limit reset time for Codex refusals that previously lacked one. The chang…
Description check Passed The description follows the repository template in substance. It has a Problem section that explains the lost reset time and the hidden resume and snooze actions, and it links issue #14169. The Change…
Out of Scope Changes check Passed All changed areas support #14169 or the PR's stated reset-selection scope. These areas are: codexUsageLimits.ts (reset selection, expiry, published-limit helper), CodexAdapterV2.ts (session-first rese…
Full details: Title check

Explanation

The title "fix(server): Codex usage-limit stops keep their reset time" describes the main change: the server now keeps a usage-limit reset time for Codex refusals that previously lacked one. The changeset adds reset-time selection for 99% and exhausted windows, a published-limits fallback via getUsageLimits, and stale-reset handling, all of which serve that goal. The title is concise, uses a conventional commit prefix, and is specific enough for a teammate scanning history.

Full details: Description check

Explanation

The description follows the repository template in substance. It has a Problem section that explains the lost reset time and the hidden resume and snooze actions, and it links issue #14169. The Change section explains the reset selection rules and the provisional published-limits fallback. The Scope and approval section links the issue and the prior work (#18721), and it notes that the maintainer takeover preserved contributor history. The Verification section lists specific commands, test counts, and the limits of what was checked (no real quota refusal, no macOS/Windows verification). The template asks for approval evidence such as a maintainer approval comment. The description references the issue and the takeover but does not quote an explicit approval comment, so this is a minor gap. The template also asks for the model and harness used, which the description includes at the end.

Full details: Out of Scope Changes check

Explanation

All changed areas support #14169 or the PR's stated reset-selection scope. These areas are: codexUsageLimits.ts (reset selection, expiry, published-limit helper), CodexAdapterV2.ts (session-first reset lookup, provisional published resets, turn-item update gating, limitId filter), CodexDriver.ts and CodexManagedProvider.ts (wiring of getUsageLimits), and the matching tests. The diff adds no client, contract, or unrelated product-area changes. The limitId filter matches the stated rule that model-specific buckets cannot claim the main allowance.

Full details: Linked Issues check

Explanation

Issue #14169 requires the composer banner to show a known reset time and enable Resume at reset and Snooze until reset. The PR adds session-first and published-limit reset selection, the 99% fallback, expiry filtering, and the model-bucket filter (limitId must be codex). These changes address the reported cases in codexUsageLimits.ts and CodexAdapterV2.ts. The tests cover the near-full, published, expired, and stale-published cases.

The terminal handler still breaks the requirement. In CodexAdapterV2.ts, the terminal-failure path sets failure.resetAt to reset.resetAt without checking the incoming event.failure.resetAt. The earlier code used event.failure.resetAt ?? derived. If the terminal event carries a non-null reset and no usable session or published reset exists, the failure is stored with resetAt: null. The banner then reports the reset as unavailable. The same issue was found in the previous review. The current diff does not change this code path.

This review did not confirm whether event.failure.resetAt is non-null for any usage-limit failure before this handler. The base code treats it as possible, and that is the basis for this FAIL.

Resolution

In the terminal usage-limit handler, keep a non-null event.failure.resetAt. Use the derived session or published reset only when the incoming failure reset is null. Add a regression test where the terminal failure has a known reset, the session snapshot and published limits have no usable reset, and the emitted failure keeps the known reset.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

AdEx-Partners-DE added a commit to AdEx-Partners-DE/t3code that referenced this pull request Oct 12, 2026
…stream pingdotgg#18721)

Own diff of pingdotgg#18721 at 98815ea, applied on stark/main. The test table keeps the cases of pingdotgg#17501 next to the new ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@omar7l

omar7l commented Oct 15, 2026

Copy link
Copy Markdown

Hit this today on 0.0.46-nightly.20261015.3145 (macOS, ChatGPT Plus, Codex CLI 0.162.1), same as case 1.

The last codex update in my session log read primary: { usedPercent: 99, resetsAt: 1792065663 }, which is 14:01 CEST. The turn then failed with usage_limit_exceeded: "try again at 2:01 PM". The banner showed "Reset time unavailable; retry manually". Usage never reached 100 in that session; the peak was 99.

I wrote a smaller fix before finding this PR (fall back to the fullest window when none reads 100, skipped when a workspace credit/spend cap caused the stop) and ran it against the existing tests. The 99% case fails on main with expected null and passes with the change. Yours covers more, so I'm adding this as another repro rather than opening a second PR.

@nagywviktor-rgb

Copy link
Copy Markdown

Additional Windows reproduction on T3 Code 0.0.46-nightly.20261015.3176, Codex 0.163.0, ChatGPT Plus. This matches the missing in-session snapshot path described here and in #14169.

A read-only inspection of the local state found a Codex root failure with:

{
  "class": "usage_limit",
  "code": "usageLimitExceeded",
  "resetAt": null
}

The same failure's message ended with "try again at 5:24 AM", while the composer showed "Reset time unavailable; retry manually". After the allowance reset, Usage showed current countdowns and available capacity, while that chat retained its saved failure. The later usage panel does not establish the exact percentages at the original refusal.

Before finding this PR, we installed a small local workaround. Sharing it here rather than opening a competing PR:

  • After the adapter stores the failed root item in limitedTurnItems, fork a same-client account/rateLimits/read request, bounded to three seconds.
  • Prefer rateLimitsByLimitId.codex, otherwise accept the legacy snapshot only if its bucket is Codex or unspecified.
  • Feed that snapshot through the existing rate-limit update handler so the late-fill path emits turn_item.updated with the missing reset.
  • Preserve an existing reset time. Lookup errors/timeouts leave the original failure intact. No message-time parsing, quota changes, cross-instance lookup, or rewriting historical failures.

One implementation detail: the lookup must be forked. The raw protocol reader awaits notification callbacks, so awaiting another RPC inline inside the terminal notification callback can prevent that reader from consuming its reply. Starting the lookup after the failed item is registered also avoids a reply arriving before the late-fill map contains the item.

This complements the discussion here; it is not a replacement for this PR's 99% handling. Our workaround still relies on the existing helper's exhausted-window rule, so a refreshed snapshot that only reports 99% can still leave the reset unknown.

Verification: nine isolated regression cases passed using T3's bundled Effect runtime and mocked usage responses: a session reset, both windows exhausted (later reset wins), multi-bucket selection, an already-reset allowance, a missing timestamp, an unrelated bucket, request failure, request timeout/nonblocking dispatch, and preserving an existing reset. The patched server passed the JavaScript syntax check and the desktop app restarted normally. We have not run this against a new real quota refusal, or run the repository's native adapter suite for this bundled workaround.

Exact locally installed workaround against the shipped server bundle

This is the tested bundled-code diff, not a source-ready PR or a request to commit generated files. The source location is apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts.

--- a/apps/server/dist/binCli-BU_S3Cf-.mjs
+++ b/apps/server/dist/binCli-BU_S3Cf-.mjs
@@ -223198,7 +223198,7 @@
 					turnItem: artifacts.turnItem
 				});
 			}).pipe(orDie));
-			yield* client.handleServerNotification("account/rateLimits/updated", (payload) => gen$1(function* () {
+			const applyT3ResetSnapshot = (payload) => gen$1(function* () {
 				yield* update$4(rateLimitSnapshot, (previous) => mergeCodexRateLimits(previous, payload.rateLimits));
 				const resetAt = codexUsageLimitResetAt(yield* get$7(rateLimitSnapshot));
 				for (const item of (yield* get$7(limitedTurnItems)).values()) {
@@ -223226,7 +223226,8 @@
 						checkedAt
 					});
 				}
-			}).pipe(orDie));
+			});
+			yield* client.handleServerNotification("account/rateLimits/updated", (payload) => applyT3ResetSnapshot(payload).pipe(orDie));
 			yield* client.handleServerNotification("thread/tokenUsage/updated", (payload) => gen$1(function* () {
 				accumulateCodexTurnTokenUsage(usageStateForThread(payload.threadId), payload.turnId, payload.tokenUsage);
 				const context = yield* awaitActiveTurn(payload.turnId);
@@ -224130,6 +224131,19 @@
 						occurredAt: yield* now
 					});
 					yield* update$4(limitedTurnItems, (items) => new Map(items).set(context.providerThread.id, item));
+					// Local T3 reset-time fix: refresh after saving the failed item.
+					// Fork so the protocol reader remains free to receive the reply.
+					yield* client.request("account/rateLimits/read", null).pipe(
+						timeoutOption(millis(3000)),
+						flatMap$1((response) => {
+							if (!isSome(response)) return void_$1;
+							const snapshot = response.value.rateLimitsByLimitId?.codex ?? response.value.rateLimits;
+							if (!snapshot || snapshot.limitId && snapshot.limitId !== "codex") return void_$1;
+							return applyT3ResetSnapshot({ rateLimits: snapshot });
+						}),
+						catch_(() => void_$1),
+						forkIn(scope)
+					);
 				}
 			});
 			const emitOrDeferRootTerminal = fn("CodexAdapterV2.emitOrDeferRootTerminal")(function* (input) {

Prepared with Codex (GPT-6) in the Codex desktop app.

@jaikhuranna

Copy link
Copy Markdown
Contributor

Commenting on behalf of @jaikhuranna (posted by Claude Code). This adds another reproduction of case 1 and an end-to-end check of this PR in a real client.

Same refusal, different outcome, 3 seconds apart. On 2026-10-17, two Codex threads on the same account hit the limit 2.4 s apart. Codex sent both the same "try again at 7:12 AM" message:

last codex update before the refusal stored failure.resetAt client
thread A (21:52:06.961Z) primary: { usedPercent: 99, resetsAt: 1792287729 } null "The provider did not report a reset time. Retry manually"
thread B (21:52:09.350Z) primary: { usedPercent: 100, resetsAt: 1792287729 } 2026-10-18T01:42:09.000Z "Usage limit resets 07:12:09", with Cancel auto-resume and Snooze until reset

So the reset was already known for thread A. The >= 100 filter dropped it because the whole-percent update still read 99%.

End-to-end check. I wrapped the real codex app-server so only turn/start is faked. It answers with an account/rateLimits/updated at usedPercent: 99 (reset 4 h out), then turn/completed failed with codexErrorInfo: "usageLimitExceeded", the same sequence as thread A. It drops the real account's rate-limit notifications, so the 99% update is the only one T3 Code sees. I ran an isolated dev server (showcase data) with the Codex binary path pointed at the wrapper:

main @ 16918b5 this PR cherry-picked onto the same main
main: Usage limit reached, Reset time unavailable; retry manually This PR: Retry after 1:22 PM, banner with Resume at reset and Snooze until reset

The reset shown (1:22:57 PM) matches the faked 99% window's resetsAt to the second, so the result comes from the near-full-window rule, not the published-limits fallback.

Rebasing onto current main. The cherry-pick has a single conflict, in CodexAdapterV2Options: main removed the doc comment above resolveRuntime, and this PR adds getUsageLimits next to it. With getUsageLimits kept and the comment dropped:

  • vp test run apps/server/src/provider/codexUsageLimits.test.ts apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts: 194 passed
  • tsc-rs --noEmit in apps/server: no errors
Wrapper used for the end-to-end check
#!/usr/bin/env node
// Wraps the real `codex app-server`, but answers every turn/start itself with a
// 99% usage update followed by a usageLimitExceeded refusal.
import { spawn } from "node:child_process";
import { createInterface } from "node:readline";

const child = spawn(process.env.REAL_CODEX ?? "codex", process.argv.slice(2), {
  stdio: ["pipe", "pipe", "inherit"],
});
child.on("exit", (code) => process.exit(code ?? 0));
const send = (frame) => process.stdout.write(`${JSON.stringify(frame)}\n`);
const now = () => Math.floor(Date.now() / 1000);
const turn = (id, status, error = null) => ({
  id, items: [], itemsView: "notLoaded", status, error,
  startedAt: now(), completedAt: status === "inProgress" ? null : now(), durationMs: null,
});

let turns = 0;
createInterface({ input: process.stdin }).on("line", (line) => {
  let frame;
  try { frame = JSON.parse(line); } catch { child.stdin.write(`${line}\n`); return; }
  if (frame.method !== "turn/start") { child.stdin.write(`${line}\n`); return; }
  const threadId = frame.params.threadId;
  const turnId = `proxy-limit-turn-${++turns}`;
  const resetsAt = now() + 4 * 60 * 60;
  send({ id: frame.id, result: { turn: turn(turnId, "inProgress") } });
  send({ method: "turn/started", params: { threadId, turn: turn(turnId, "inProgress") } });
  setTimeout(() => {
    send({ method: "account/rateLimits/updated", params: { rateLimits: {
      limitId: "codex",
      primary: { usedPercent: 99, resetsAt, windowDurationMins: 300 },
      secondary: { usedPercent: 16, resetsAt: resetsAt + 6 * 86400, windowDurationMins: 10080 },
    } } });
    send({ method: "turn/completed", params: { threadId, turn: turn(turnId, "failed", {
      message: "You've hit your usage limit. ... try again later.",
      codexErrorInfo: "usageLimitExceeded",
    }) } });
  }, 500);
});

// Drop the real account's rate-limit updates so only the 99% one reaches T3 Code.
createInterface({ input: child.stdout }).on("line", (line) => {
  if (!line.includes('"account/rateLimits/updated"')) process.stdout.write(`${line}\n`);
});

@juliusmarminge

Copy link
Copy Markdown
Member

Taking over this one as maintainer for batch codex-06. I will preserve the contributor history, merge current main, keep focused regression coverage, and verify session reset precedence before publishing. No merge is planned.

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Oct 19, 2026
@voltcrash
voltcrash force-pushed the fix/codex-usage-limit-reset branch 2 times, most recently from 6cb3476 to b6fa42a Compare October 31, 2026 19:02
@juliusmarminge juliusmarminge added bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed labels Oct 31, 2026
voltcrash and others added 2 commits November 1, 2026 11:44
A Codex turn refused while its last rate-limit update read 99%, or before
the session saw any update, recorded no reset, so the composer hid Resume
at reset and Snooze until reset. Treat a 99% window as exhausted when none
reads 100%, ignore windows that have already renewed, and fall back to the
instance's published limits when the session snapshot has no reset.

Fixes pingdotgg#14169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Orchestrator V2 Usage Reset Tracker Broken

5 participants