[Bug]: Shared Codex app-server backlogs or dies silently; thread/start hangs for hours and new Codex threads never start #24300

Summary

T3 runs one shared Codex app-server per environment (provider-session:provider-instance:codex:shared). On long-running servers, that process's message stream falls hours behind or stops entirely. When that happens, thread/start requests wait hours or never get an answer. New Codex threads then sit at "Workspace ready" with the provider thread stuck in pending, and no turn ever starts. T3 doesn't detect any of this: there is no timeout, no restart, and no error in the UI.

A second case: if the app-server process is killed (for example by stopping its systemd scope), T3 keeps routing new threads to the dead session. The only trace is CodexAppServerTransportError: Codex App Server transport operation 'read-process-exit-status' failed from ProviderSessionManager.unloadIdleThread.

Environment

• T3 0.1.2-preview.20261022.3511, Linux server (headless, 4 vCPU / 16 GB)
• codex-cli 0.163.0
• Several Codex threads at once (up to 4 leads, each delegating step tasks), with the server up 1–2 days

Evidence (from ~/.t3/userdata/logs/provider/events.*.log)

thread/start latency on one server, request (outgoing id) to its response:

• 2026-10-22 08:15–08:19: 0 s
• 2026-10-22 09:50: 938 s
• 2026-10-22 10:30: 1510 s
• 2026-10-22 18:37 onward: 13,000–69,000 s (4–19 h)
• from 2026-10-23 18:06: no response at all (11 requests)

Example: request id 88 was sent at 2026-10-23T11:53:51.883Z, and the response arrived at 2026-10-23T20:43:19.075Z.

Notification lag: receive time minus the app-server's own emittedAtMs.

• About 20 min on 10-22.
• 2–5 h on 10-23 (for example, 7,000–9,000 s throughout 10:25–11:34).

The app-server emits on time, but the messages reach T3 hours later. That points at the shared stdio stream or T3's reader, not at model latency. The process itself is idle (0% CPU).

Across 40 servers running the same workload, counting thread/start requests since 2026-10-23:

• 61/61 unanswered on two servers.
• 21/26 unanswered on another.
• 46–55 answered late (over 5 min) on two more.
• Five servers whose app-server had been killed: every request unanswered.

Reconnecting... 5/5 errors show up in the same stream around the stalls.

Expected

• A thread/start (and turn start) that gets no response within a bounded time fails the run with a visible error, instead of hanging.
• T3 detects when the shared app-server is dead or backlogged (process exit, or message lag past a threshold) and restarts it. It doesn't keep routing new threads to it.
• Ideally, one busy or wedged thread can't stall thread/start for every other thread in the environment.

Workaround

Restart t3code.service, which starts a fresh app-server.