fix(server): compact superseded history in the background

[juliusmarminge]

The server never deleted superseded orchestration history, so databases grow without bound. One maintainer database reached 114 GB, most of it repeated whole-payload updates of the same thread, message, node or PR link.

The server now runs compactEventStore one minute after startup and every six hours after that. Each pass deletes events whose state a newer retained event already carries:

• Whole-thread events (rename, archive, settle, pin, model changes…): only the newest per thread is kept. Visit and mark-unread events supersede only each other, so a later visit never removes the rename that carries the thread's activity time.
• Messages, nodes and PR links: the oldest and newest event per entity are kept. Clients add an entity where they first see it, so keeping the first event preserves message order for a client that reconnects from an older cursor.
• thread.created, thread.deleted, runs and turn items are never compacted.

No replay boundary or snapshot fallback is needed: every deleted row has a newer row for the same key, so any replay from an older cursor still ends in the same state and order.

New databases are created with auto_vacuum = INCREMENTAL, and the server returns freed pages to disk online in small yielding batches. Existing databases get that with t3 doctor (#19436). Compaction pages are 50 rows, because a 500-row page with large payloads blocked the shared connection for ~90 ms.

A retried MCP metadata update whose events were compacted now answers from the current thread instead of failing.

Takes over #19249.

🤖 Generated with Claude Code

Fixes #18559

---

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	20.9 KiB	20.9 KiB	0 B (0.0%)	29.3 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.2 KiB	21.2 KiB	0 B (0.0%)	29.3 KiB	✅
Claude	Live turn messages	2	2	0 (0.0%)	8	✅

Baseline: fa1184d · PR result: e5cbab0 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 108.5 KiB
• Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR not approvable — The PR adds default-on background compaction and SQLite page reclamation, including deletion of durable superseded history and changed replay retention behavior. Its persistence and runtime lifecycle impact is broader than a small isolated fix, even with targeted tests covering replay and reclamation.

No code changes detected at e5cbab0. Prior analysis still applies.

^{You can add or adjust custom eligibility rules. Learn more.}

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

• Configuration used: Path: .coderabbit.config.ts
• Review profile: CHILL
• Plan: Team
• Run ID: de2a88b8-da3a-489a-b0cb-7d198becaf40

📥 Commits

Reviewing files that changed from the base of the PR and between cd8a1ef and e5cbab0.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Path: .coderabbit.config.ts
• Review profile: CHILL
• Plan: Team
• Run ID: 4a66dcd7-e1c4-4a7d-b91c-f79b653d3222

📥 Commits

Reviewing files that changed from the base of the PR and between cf8be3b and 8f1be1c.

📒 Files selected for processing: 3

• apps/server/src/orchestration-v2/FoundationPersistence.test.ts
• apps/server/src/persistence/DatabaseStorage.ts
• apps/server/src/server.ts

Included review availability: This review used your included allowance. 9 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

The server now schedules event-store compaction and incremental SQLite page reclamation. Compaction retention rules preserve selected thread and entity events. Metadata updates use the thread shell when dispatch returns no metadata-updated event.

Changes: Event Compaction and Storage Maintenance

Layer / File(s)	Summary
Compaction retention rules apps/server/src/orchestration-v2/ProjectionMaintenance.ts, apps/server/src/orchestration-v2/FoundationPersistence.test.ts, apps/server/src/orchestration-v2/runtimeLayer.test.ts, apps/server/src/ws.test.ts	Compaction handles read-state events separately, retains the oldest and newest keyed entity updates, and removes obsolete pull-request-link updates. Tests cover retained events, projection rebuilding, and replay after compaction.
Scheduled database maintenance apps/server/src/persistence/Sqlite.ts, apps/server/src/persistence/DatabaseStorage.ts, apps/server/src/persistence/DatabaseStorage.test.ts, apps/server/src/server.ts	New databases enable incremental auto-vacuum. DatabaseStorage schedules event compaction and page reclamation, and the server provides its layer. The storage test checks page reclamation and retained table data.

Thread Metadata Update Fallback

Layer / File(s)	Summary
Select the metadata update result apps/server/src/mcp/ThreadMetadataMcpService.ts	When dispatch returns no metadata-updated event, the update path retrieves the thread shell. It returns an orchestration error if the selected thread is null.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant DatabaseStorage
  participant ProjectionMaintenance
  participant EventStoreV2
  participant Sqlite
  DatabaseStorage->>ProjectionMaintenance: Run scheduled event-store compaction
  ProjectionMaintenance->>EventStoreV2: Scan and compact events
  DatabaseStorage->>Sqlite: Check auto_vacuum and freelist
  DatabaseStorage->>Sqlite: Reclaim pages incrementally

Loading

Suggested reviewers: t3dotgg

Merge Risk: ⚪ Minimal · up to 8f1be

No actionable merge-blocking issue was established in the reviewed changes. The scheduled maintenance and metadata fallback appear ready to merge after normal checks.

Pre-merge checks: 3 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Description check		The description clearly explains the database-growth problem and the compaction, retention, vacuum, and MCP changes. It does not include the required section structure, explicit maintainer approval or…	Add the required Problem, Change, Scope and approval, and Verification sections. Include the triaged issue or approval discussion with explicit maintainer approval, and document the focused tests or manual checks performed with their observ…

Passed checks: 3 passed

Check name	Status	Explanation
Title check		The title clearly and concisely describes the primary change: background compaction of superseded server history.
Linked Issues check		#18559 requires a production caller for ProjectionMaintenanceV2.compactEventStore. DatabaseStorage.layer resolves ProjectionMaintenanceV2, waits one minute after startup, and repeats compaction …
Out of Scope Changes check		The changes remain connected to #18559. The 50-row batches, page reclamation, incremental auto-vacuum setup, replay coverage, and MCP fallback support safe operation after event deletion and address t…

Full details: Description check

Explanation

The description clearly explains the database-growth problem and the compaction, retention, vacuum, and MCP changes. It does not include the required section structure, explicit maintainer approval or scope rationale, or focused verification results.

Resolution

Add the required Problem, Change, Scope and approval, and Verification sections. Include the triaged issue or approval discussion with explicit maintainer approval, and document the focused tests or manual checks performed with their observed results and any unverified items.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[AdEx-Partners-DE]

A Windows data point from a large database, for anyone wondering what this does on an existing install. The fork I run has carried this change since 16 Oct. The numbers below are read-only queries on its statev2.sqlite at 16:55 UTC, compared with a measurement at 00:54 UTC the same day, before the build had it.

The compactor has caught up. By its own rule (keep the oldest and newest event per entity, delete the ones in between), almost nothing is left to delete:

	events now	still deletable
message.updated	32,863 for 32,773 messages	0
node.updated	133,397 for 78,651 nodes	22
thread.pull-request-synced	6	(1,990 were written in 19 hours on 12/13 Oct, see #18559)

The file did not shrink, as #19436 says it won't. 3,274,887,168 bytes before, 3,299,213,312 now. auto_vacuum is 0, so the background reclaim does nothing here. The free list holds 49,789 pages of 4 KB, about 204 MB or 6% of the file. That is the floor of what t3 doctor would give back on this install; a VACUUM may recover more from half-empty pages. I have not run doctor.

What is left is mostly turn items. Event payload is 1,077 MB, and turn-item.updated is 671 MB of it (174,144 events for 110,243 items). All events except the newest per item come to 63,901 events and 139 MB. The comment in compactEventStore says why they stay (slimmed payloads reference their prompts), so this is not a request to delete them; it only shows where the remaining growth is on an install that runs many long agent turns. By table: events 1,605 MB, projection_turn_items 673 MB, projection_nodes 162 MB.

No stall visible, with one gap. The pass one minute after the 16:20 UTC restart (about 16:21:20) left no SQL span over 300 ms between 16:21:05 and 16:24:40, and no server.eventLoop.stall span. That pass probably had little left to do. The first, large pass ran at an earlier restart whose trace has rotated away, and "Compacted orchestration history" goes to the logger only, so I cannot give the deleted count or the duration of that first pass.

Not checked: other installs, and what doctor would actually reclaim here.

Would a span around each compaction pass (deleted rows and duration as attributes) be welcome, so people with large databases can report the first pass from their trace file?

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.