[Bug]: Orchestrator V2 nightly silently reuses a preview-created database and misses newer V1 threads #15017

Activity

1. Paranjayy commented on Oct 3, 2026

   

   Paranjayy

   on Oct 3, 2026

   Last edited by Paranjayy

   Author

   More actions

   One additional observation from the same upgrade: saved, unsent composer drafts that were visible in the previous version no longer appeared in the Orchestrator V2 nightly.

   This is a user-observed symptom so far. We have not checked where those drafts were stored, whether they remain recoverable, or whether the cause is the stale database described above, the desktop profile change, or another migration path. The verified local thread repair did not verify draft recovery.

   It may be useful to check draft preservation alongside the V1 → Orchestrator V2 migration. I am adding it here for context; it would need separate storage evidence or a reproduction before treating it as the same confirmed defect.

   This comment was prepared by GPT-6.1-Sol through Codex in T3 Code, based on the user's report.
2. juliusmarminge commented on Oct 3, 2026

   

   juliusmarminge

   on Oct 3, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for such a careful report, @Paranjayy. The before-and-after counts and the offline reconciliation made this straightforward to confirm.

   Confirmed on 0.0.46-nightly.20261003.2610. If a statev2.sqlite file already exists, the nightly treats it as the finished V2 store and never reads the live V1 database again. It can't tell a file left by a preview build apart from one the nightly made itself. That means threads and thread-state changes you made in stable V1 after the preview get skipped without any warning, and restarting won't catch them up. Your state.sqlite isn't modified or deleted. The newer V1 rows are still there, they just never get read.

   • initializeV2Database returns as soon as the destination file exists (source).
   • The lazy importer reads projection_threads from inside that V2 file, not from state.sqlite (source).
   • Metadata repair only fills fields that are still empty on threads that were already imported. It never refreshes settlement or snooze state from a newer V1 database. That lines up with what you saw: threads and projects missing, later messages absent, and stale snooze and settlement metadata.

   How this relates to #14871: copying once is intentional after a real V2 start, and the thread migration guide says later V1 and V2 changes don't sync. This report covers an earlier gap. V2-branch preview builds such as v0.0.43-preview.20260923.2177 already wrote userdata/statev2.sqlite and had the same early return. The stable nightly from that week didn't use that path at all. So if a preview created that file on September 24, this nightly keeps it and treats it as the migration source.

   Until there's a fix, and keeping any V2-only work safe:

   • If the preview database has nothing you need to keep: stop T3 Code. Move statev2.sqlite and its -wal and -shm files somewhere else, but don't delete them yet and don't touch state.sqlite. Then start the nightly, which will take a fresh copy of your current V1 data. Once your threads look right, you can delete the moved files.
   • If the preview database has V2-only threads you need: leave it where it is. There's no supported catch-up command yet, so recovery takes an offline merge like the one you did.

   Possible fixes: detect a V2 snapshot whose copied V1 tables are older than the current state.sqlite and warn before continuing, or selectively import the missing V1 threads, messages, and metadata without overwriting V2-only data. Picking between those is a maintainer decision. Adding a note to #14871 alone wouldn't help installs that have already launched this nightly.
3. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 3, 2026
4. Paranjayy

   changed the title [-][Bug]: V2 nightly silently reuses a preview-created database and misses newer V1 threads[/-] [+][Bug]: Orchestrator V2 nightly silently reuses a preview-created database and misses newer V1 threads[/+] on Oct 3, 2026
5. kharitonov-egor commented on Oct 4, 2026

   

   kharitonov-egor

   on Oct 4, 2026

   More actions

   Hit a variant of this on Windows 11. I never ran a preview build. Here statev2.sqlite came from the first public V2 nightly (orchestration_v2_projection_metadata updated 2026-10-03T07:07:42Z).

   Then I lost my sidebar to #15001 and rolled back to 0.0.45-nightly.20261002.2595 to keep working. That puts me in the same trap this issue describes, just without a preview involved. Any V1 work I do now will be missing the next time I start V2, because V2 sees the existing statev2.sqlite and never reads state.sqlite again.

   The drift has already started. One day after the cutover, V1 state.sqlite has 428 threads and 12,493 messages, while statev2.sqlite has 422 threads and 12,267 messages.

   Anyone rolling back to V1 to dodge a V2 bug will hit this. To work around it, I moved statev2.sqlite* out of ~/.t3/userdata so the next V2 launch migrates from current V1 data. It would help if V2 compared V1's last-write time or event sequence with the snapshot it imported from, and re-imported or at least warned when V1 is newer.
6. anenger commented on Oct 16, 2026

   

   anenger

   on Oct 16, 2026

   More actions

   Still reproducible on stable 0.1.0 (macOS). The earlier reports here were on nightlies and a manual rollback; this one came through the normal stable release, so the trap is no longer nightly-only.

   Shape of this instance: a 0.0.46-nightly ran once on this machine, seeded statev2.sqlite, and the machine returned to stable 0.0.45 about 2.5 minutes later. Stable then wrote V1 for ~37 hours. When 0.1.0 arrived, it opened the Oct 14 snapshot and six threads were simply gone from the sidebar.

   Timeline (from logs/desktop.trace.ndjson.7, local EDT / UTC)

   Local	UTC	Event
   07:57:35	11:57:35Z	desktop.updates.applyUpdateAvailable → 0.0.46-nightly.20261014.3105
   07:57:44	11:57:44Z	applyUpdateDownloaded
   07:58:41	11:58:41Z	nightly boots: migrations 55–62 (OrchestrationV2 … ThreadFailureLookupIndex) and statev2.sqlite seeded
   08:00:08	12:00:08Z	applyUpdateAvailable → 0.0.45 (back to stable)
   08:00:16	12:00:16Z	applyUpdateDownloaded
   —	Oct 14 12:31Z → Oct 16 01:00Z	stable 0.0.45 writes V1 only: events 1000 → 6261, 6 new threads
   Oct 15 ~20:26	Oct 16 ~00:26Z	0.1.0 first appears in traces, reads the stale statev2.sqlite

   Divergence measured on disk

   	state.sqlite (V1)	statev2.sqlite
   projection_threads	9	3
   orchestration_events	6261	1151
   projection_state.last_applied_sequence	6261	999 (frozen at 2026-10-14T02:28:20.146Z)
   orchestration_v2_events	—	0
   auth_sessions	25	17
   auth_pairing_links	20	11

   orchestration_v2_projection_metadata reads last_sequence = 1151, updated_at = 2026-10-14T11:58:42.752Z — not one V2 write in the ~37 hours between the seed and the 0.1.0 upgrade.

   The seed itself was correct — the defect is purely the missing re-sync

   Worth isolating, because 999 looks like a truncation bug and isn't one. In V1, event 999 occurred at 2026-10-14T02:28:20.146Z and event 1000 at 2026-10-14T12:31:49.514Z. The seed ran at 11:58:41Z, between them, so V1 held exactly 999 events at that instant. Sequences 1–999 in statev2.sqlite match V1 one-for-one on both event_id and sequence (999/999).

   The copy was therefore complete and correct for the moment it was taken. Everything lost is a consequence of there being no staleness check afterward. Thread-level cut:

   • Imported (all events ≤ 999): 2f9e8b79… (seq 2–33), 5650d604… (36–999), 0d912b99… (751–966)
   • Never imported (all start ≥ 1000): 95b9898f… (1000), 089fff1c… (1069), 57dfed50… (1175), 6fdad9c0… (1244), 65753d74… (1265), 0bf63032… (1404)

   statev2.sqlite also carries 152 appended events (seq 1000–1151) whose event_ids appear nowhere in V1 (0/152) and which cover only those same 3 threads plus 2 projects — the migration's own re-emitted thread.created / message.updated / turn-item.updated records, ending with two thread.visited at 11:58:42Z.

   Code is unchanged at the v0.1.0 tag

   Both mechanisms @juliusmarminge confirmed are still present:

   • apps/server/src/persistence/initializeV2Database.ts#L26 — if (yield* fs.exists(destinationPath)) return;, guarded on file existence alone. The doc comment states the intent: "Seed V2 once. Its copied legacy tables remain the source for lazy transcript import."
   • LegacyV1ThreadImporter.ts#L575 — selects FROM projection_threads inside the V2 database, so it can only ever see the snapshot.

   Suggested detection

   Picking up @kharitonov-egor's suggestion, the comparison needs no new bookkeeping — both numbers already exist. At startup, compare V1 projection_state.last_applied_sequence (or max(sequence) from its orchestration_events) against orchestration_v2_projection_metadata.last_sequence. Here that is 6261 vs 1151: a 5110-event gap that could have been surfaced as a prompt or a warning instead of silence. A one-time guard would also help: refuse to seed, or mark the store provisional, when the build writing it is a pre-release and a stable build may follow.

   Recovery on this machine

   The maintainer workaround in this thread worked unchanged. With the app fully quit, state.sqlite verified integrity_check: ok at 9 threads / 6261 events; statev2.sqlite* was moved out of userdata/ (not deleted) and the app restarted to re-seed from current V1. state.sqlite was never written to. Re-pairing one device was expected, since 4 auth_sessions rows existed only in the old V2 file.

   Environment

   • macOS 27.0.0, darwin arm64, Node v26.8.2
   • Bug surfaced on: desktop app 0.1.0 against its local server (serverVersion 0.1.0, orchestrationProtocolVersion 2)
   • Drift accumulated on: stable 0.0.45
   • statev2.sqlite seeded by: 0.0.46-nightly.20261014.3105
   • updateChannel: "latest", updateChannelConfiguredByUser: true
   • Triage CLI that produced the context file: t3 0.1.0 via npx t3 triage

   ---

   Investigated with Claude Opus 5 (claude-opus-5) in Claude Code, run from t3 triage. Diagnosis read against a v0.1.0 tag checkout; counts are from this machine's own databases, with the home directory path redacted.
7. arsotirchellis commented on Oct 16, 2026

   

   arsotirchellis

   on Oct 16, 2026

   More actions

   Another stable 0.1.0 case (macOS, arm64), with a different starting point: I never rolled back by hand. The switch from Nightly back to the V1 build arrived as an ordinary in-app update.

   Timeline (from ~/Library/Caches/com.t3tools.t3code.ShipIt/ShipIt_stderr.log and file timestamps, UTC)

   UTC	Event
   Oct 12 18:01:13Z	A Nightly build seeds statev2.sqlite from state.sqlite (all orchestration_v2_legacy_imports.shell_imported_at = 2026-10-12T18:01:13.615Z; v1-local-storage-imported marker written at the same time)
   Oct 12 18:14:42Z	ShipIt: Moving bundle from T3 Code (Nightly).app to T3 Code (Alpha).app, so the machine is back on the V1 build about 13 minutes later
   Oct 12 → Oct 16 10:01Z	V1 keeps writing to state.sqlite for ~3.5 days
   Oct 16 10:01:25Z	ShipIt: Moving bundle from T3 Code (Alpha).app to T3 Code (Beta).app (0.1.0), which opens the Oct 12 snapshot

   Divergence measured on disk

   	state.sqlite (V1)	statev2.sqlite
   projection_threads	385	354
   projection_thread_messages	13,073	12,029

   • 31 threads created in V1 after the seed have no V2 thread.created event. Their created_at values run from 2026-10-12T18:16:53Z to 2026-10-16T08:42:09Z. The newest seeded thread was created at 2026-10-12T18:02:03Z, so the cut is exact.
   • 5 seeded threads gained 381 message rows in V1 after the seed (162 of them user/assistant). Their transcript_imported_at was already set on Oct 12, so ensureTranscript never looks at them again.
   • 0.1.0 added nothing to those 5 threads, so their gap is pure V1 drift.

   Same root cause, confirmed in the shipped 0.1.0 bundle

   initializeV2Database still returns when statev2.sqlite exists ("Seed V2 once"). reconcileShells and ensureTranscript only ever read the legacy tables inside statev2.sqlite.

   Why this one matters

   The issue was opened for preview builds, then manual rollbacks. Here there was no manual downgrade: the app's own updater (ShipIt) swapped the bundle from Nightly to Alpha 13 minutes after the seed, and I carried on working normally. Anyone who briefly ran a V2 Nightly and then moved back to the stable channel before 0.1.0 is likely in the same position without knowing it.

   Local recovery (offline, preserves V2-only data)

   Instead of moving statev2.sqlite aside, which would drop work done in 0.1.0 since the upgrade, I used the importer's own idempotency:

   1. Quit T3 Code and back up both databases.
   2. Copy the missing legacy rows (projection_threads for the 31 threads; INSERT OR IGNORE of projection_thread_messages, _pull_requests, _activities, _turns, _proposed_plans, _sessions) from state.sqlite into statev2.sqlite. The table layouts are identical in 0.1.0.
   3. Set transcript_imported_at = NULL on orchestration_v2_legacy_imports for the 5 seeded threads that gained messages. ensureTranscript skips messages whose import event already exists, so it only adds the missing ones.
   4. Start 0.1.0. reconcileShells imports the 31 threads at startup, and the transcripts fill in.

   Result after one restart of 0.1.0:

   Check	Before	After
   Legacy threads with no V2 thread.created	31	0
   orchestration_v2_projection_threads	355	386 (385 from V1 + 1 created in 0.1.0, untouched)
   orchestration_v2_legacy_imports with transcript_imported_at IS NULL	—	0, with 0 last_error
   V1 user/assistant messages without a migration:v1:message:<id> event	484	0 of 8,959
   V1 user/assistant messages without a V2 message row	484	0 of 8,959

   Of the 1,044 legacy message rows copied across, 484 are user/assistant (322 in the 31 new threads, 162 in the 5 continued ones), and all 484 are now in V2. The other 560 are tool and system rows, which the importer skips by design (role IN ('user', 'assistant')). The most-affected continued thread (300 missing rows) now ends at its last V1 message, a few seconds before the 0.1.0 update was applied. No restored thread came back archived or deleted.

   That is effectively the "selective catch-up" from the triage comment, and it takes about 20 lines of SQL. Running it automatically at startup whenever V1's max(orchestration_events.sequence) is ahead of the snapshot's would close this without touching V2-only data.
8. john3374 commented on Oct 16, 2026

   

   john3374

   on Oct 16, 2026

   More actions

   Same shape on stable 0.1.0, Windows 11, local desktop install. A V2 nightly ran once on 2026-10-03 and seeded statev2.sqlite, then the machine went back to 0.0.45 (to dodge #15001) and kept writing V1 until 2026-10-15. After the auto-update to 0.1.0, the app opened the Oct 3 snapshot.

   Before: state.sqlite 53 threads / 27 projects; statev2.sqlite 37 threads / 25 projects, all orchestration_v2_legacy_imports rows dated 2026-10-03, no errors, 0 orchestration_v2_events (nothing V2-native).

   Workaround from the triage comment above worked: app quit, statev2.sqlite plus -wal and -shm moved aside, relaunch. Result: 53 threads, 53 transcripts imported, 0 errors, 27 projects. Took a few seconds.

   Filed by Claude Fable 5.1 via t3 triage.
9. immakermatty commented on Oct 19, 2026

   

   immakermatty

   on Oct 19, 2026

   More actions

   Another way into this, with no channel switching and no manual rollback: an in-app Update server from 0.0.x to 0.1.x that the service launcher rolls back.

   1. The 0.1.x server runs as a trial under the launcher the unit names. That is still the old 0.0.x launcher, because the unit stays on the runtime that installed it ([Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart #17490). It gives the trial 120 s to report prepared. The 30 minutes from fix(server): service updates no longer fail on full disks or large databases #19311 only apply from 0.1.0 on.
   2. Before it listens, the trial seeds userdata/statev2.sqlite from state.sqlite (Sqlite.layerConfig → initializeV2Database). On a big V1 database the copy alone eats a good part of those 120 s. We measured 33 s for 5.2 GB on an Apple Silicon SSD, and a VM disk is slower.
   3. On rollback the launcher restores only the database the requesting server named. That is state.sqlite (restoreDatabaseBackup on pending.dbPath, the same in 0.0.45). statev2.sqlite and its -wal/-shm stay, outside the rollback boundary that docs/internals/server-updates.md describes.
   4. The 0.0.x server comes back and keeps writing state.sqlite. The next update finds statev2.sqlite and takes it as is, so everything written between the two attempts is missing.

   Two things that might help pick a fix:

   • A launcher-side fix can't cover this transition. The pre-0.1.0 launcher does the rollback, so the fix has to live in the 0.1.x server's seeding or startup.
   • In this path the leftover copy is disposable. A trial that never reported prepared never opened its command gate. As far as we can tell, its copy holds only the seed, the migrations and the import, and no user work. Seeding again when state.sqlite changed after the seed, and the copy has no V2-native writes yet, would cover it without risking V2-only data.

   This comes from reading the code at main 5b72385 and v0.1.0, not from an end-to-end reproduction. On our hosted machines we move statev2.sqlite* aside after a rolled-back update.