Local Windows desktop disconnects during 36–44 second server stalls with a 12.5 GB database #21539

What happened

T3 Code shows the client as disconnected even when using the Windows desktop app locally. The user says this started on the local Windows PC after a PC restart and a T3 Code update, rather than on the remote server. The user suspected that a large database might be involved. Read-only investigation found repeated server event-loop stalls that exceed the client's heartbeat timeout, together with very slow database and snapshot spans. The previous working version is unknown, so the update is a timing correlation rather than an established regression boundary.

Diagnosis

Confirmed observations:

• The desktop-owned server is running on 127.0.0.1:3773; the runtime metadata and the live /.well-known/t3/environment endpoint both report 0.1.1-nightly.20261018.3302.fork.2.
• statev2.sqlite is approximately 12.5 GB / 11.64 GiB, with 2,229 projected threads. Read-only PRAGMAs returned page size 4096, page count 3,050,730, freelist count 0, WAL mode, and auto_vacuum = 0.
• The retained stall log contains 44,399 ms and 36,178 ms event-loop stalls during this server session, plus several 15–20 second stalls.
• Retained SQL spans reach 72.0 seconds; one shell-snapshot request takes 38.7 seconds. Several unrelated database callers have overlapping long spans, suggesting a server-wide availability problem rather than one disconnected provider.

Interpretation and limits: The client uses a 15-second no-frame heartbeat timeout (packages/client-runtime/src/rpc/protocol.ts, checked at the installed release tag). Event-loop stalls of 36–44 seconds can prevent local keepalives and requests from being served. This is a plausible explanation for the reported disconnects, but a particular WebSocket timeout was not captured alongside the UI symptom.

Database size is context, not a proven root cause. SQL span duration can include connection scheduling, lock contention, and delays caused by other work; these numbers do not prove that each listed statement itself blocked the main thread for its full duration. No CPU/I/O profile was collected, corruption was not established, and the responsible statement or maintenance operation has not been isolated.

This is a personal-fork build based on upstream nightly 0.1.1-nightly.20261018.3302. Installed fork tag: v0.1.1-nightly.20261018.3302.fork.2, commit b4eebda3994fb72f382a5ed49e0ee3e791ff5cf7; upstream base d0918f1659f4c19172796c8d1910ca6e4bc18b51. Reproduction on an unmodified upstream build has not been verified. Upstream nightly 0.1.1-nightly.20261019.3348 was available during triage but was not installed or tested.

Steps to reproduce

Observed on an existing install; no standalone deterministic reproduction is available:

1. Update T3 Code and restart the Windows PC. The user reports the problem began after these actions; the exact ordering and previous working version were not established.
2. Run the Windows desktop build above with the existing approximately 12.5 GB database and substantial thread history, and use the client against its local server.
3. Observe intermittent disconnected state and inspect logs/event-loop-stalls.ndjson and logs/server.trace.ndjson for long stalls and database/snapshot delays.

Expected: the local server continues answering WebSocket keepalives and client requests while processing history and background work.

Actual: the client reports disconnected; the server's own monitor records stalls longer than the heartbeat timeout.

Version

0.1.1-nightly.20261018.3302.fork.2 (personal fork; upstream base nightly 0.1.1-nightly.20261018.3302).

Environment

Windows 11 Pro x64, OS version 10.0.26200; installed Electron desktop with its bundled local server. The diagnostic shell has Node v24.21.0; the bundled server's Node version was not independently checked. At inspection, Windows reported approximately 63.9 GiB total RAM and 29.5 GiB available, with approximately 74 GiB free on the database volume. No system-wide memory or disk-capacity exhaustion was observed in that snapshot.

Evidence

Selected server-owned stall-log records; times are UTC:

{"time":"2026-10-19T04:29:30.533Z","delayMaxMs":44399,"utilization":1,"cpuUserMs":3765,"cpuSystemMs":8844,"rssMb":390}
{"time":"2026-10-19T05:47:34.857Z","delayMaxMs":36178,"utilization":1,"cpuUserMs":1469,"cpuSystemMs":1984,"rssMb":323}
{"time":"2026-10-19T05:48:36.181Z","delayMaxMs":15232,"utilization":1,"cpuUserMs":1140,"cpuSystemMs":922,"rssMb":361}

Selected completed server trace spans from the same session, with start times in UTC:

Long-lived subscription spans were excluded from the slow-operation evidence. The selected trace records do not establish whether the caller used the reader worker or writer connection.

The server was alive during triage: three successive loopback environment-endpoint probes succeeded in 1,864 ms, 685 ms, and 1,297 ms. This was not a continuous outage, and those probes did not reproduce a timeout.

Related issues

• [Bug]: Background history compaction blocks the server in a main-thread WAL checkpoint #20203: an open report with similar reconnect symptoms, but it specifically attributes Linux stalls to a compaction DELETE/main-thread WAL checkpoint. That mechanism has not been confirmed on this Windows install; this may ultimately share its root cause.
• [Bug]: Server: the thread list read blocks the event loop and every other database call for seconds on large databases #14701 and [Bug]: Orchestrator V2 event writes stall the server on a high-latency disk (156 ms fsync), environment flips Offline #15707: closed reports of read-path and write-path stalls causing reconnects. This capture is from a newer build that already contains later database-worker changes.
• [Bug]: Orchestrator uses exponentially more storage #21120: database growth; this report focuses on local server availability, not file size alone.
• fix(server): keep history discovery off the writer and pace compaction #20877: open PR moving compactor discovery off the synchronous writer. It has not been tested against this install and is not presented as a verified fix.

Fix applied or workaround

None. No database writes, VACUUM, deletions, settings changes, source patches, or service restarts were performed. Logs and the database were inspected read-only; the private database and full traces are not attached.

Filed by

GPT-6.1 Sol at high reasoning effort, through the Codex harness in T3 Code, following the T3 triage playbook in an existing agent session.