Skip to content

refactor(core): use Effect's official Bun and Node SQLite clients - #60798

Merged
kitlangton merged 3 commits into
devfrom
sqlite-official-clients
Oct 31, 2026
Merged

kitlangton merged 3 commits into
devfrom
sqlite-official-clients

Conversation

@kitlangton

@kitlangton kitlangton commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Bun and Node now use Effect's own SQLite clients instead of ours

Before: packages/core carried its own SQLite client (database/sqlite.ts) with thin Bun and Node drivers on top. On Node every SQLite error came back as UnknownError, so a duplicate-key insert looked the same as a corrupt database. On Bun, opening with readonly: true threw, because we always passed create: true.

After: #sqlite still resolves per runtime, but the Bun and Node entries are now short shims over @effect/sql-sqlite-bun and @effect/sql-sqlite-node 4.1.0. Errors are classified on both (a duplicate key is a ConstraintError), read-only opens work, and Node caches prepared statements. sqlite.ts is deleted. The workerd client stays ours for now; it no longer imports sqlite.ts and keeps its scoped-transaction fix.

  // database/sqlite.bun.ts (sqlite.node.ts is the same shape)
- 100 lines: our own connection, semaphore, transaction and error mapping on top of sqlite.ts
+ import { SqliteClient } from "@effect/sql-sqlite-bun"
+ export const sqliteLayer = (config) => Layer.orDie(SqliteClient.layer(config))

  // database/sqlite.workerd.ts
- import { Sqlite } from "./sqlite.js"   // shared Native service + connection helpers
+ inline Connection over options.storage; die with an Error when given a path instead of storage

What changes in behavior

  • sql.withTransaction now opens with BEGIN IMMEDIATE instead of BEGIN. Direct callers: none in Bun or Node production code. The only one in src is drizzle/effect-sqlite/session.ts:141, on the workerd native path, which still uses our client. The rest are tests (sqlite-interrupt, sqlite-workerd).
  • Drizzle db.transaction is unchanged: it still reserves a connection and issues its own begin deferred and savepoints. Covered by the drizzle and new Node tests.
  • The upstream clients set busy_timeout before the WAL pragma and retry WAL on a lock. The pragmas in database.ts (WAL, synchronous, busy_timeout, cache_size, wal_checkpoint, foreign_keys) still run after open.
  • @effect/sql-sqlite-bun moves from devDependencies to dependencies, and @effect/sql-sqlite-node is new (root catalog 4.1.0).

Bundles: sqlite-bundle.test.ts now checks that Node and default builds include @effect/sql-sqlite-node but never bun:sqlite or @effect/sql-sqlite-bun, and that workerd includes neither upstream client. The Node CLI bundle (build-node.ts --bundle-only) builds, starts, and has no bun:sqlite.

Tests: new sqlite-node.test.ts builds a Node bundle and runs it under node: WAL is on, a Drizzle transaction commits two rows, a duplicate insert fails with ConstraintError, Drizzle's typed insert().returning() and select() work (including a constraint failure mid-returning() followed by a working query), and the queued-transaction and queued-reserve interrupt sweep from sqlite-interrupt.test.ts leaks no connection. A Bun case checks the same error and reopens the file read-only. Both failed on dev with the pinned Bun 1.4.4 (UnknownError, and Bun's "read-only database cannot have create: true"); older Bun versions accept that read-only open, so the Bun case only catches it on 1.4.4 and later. sqlite-interrupt.test.ts passes unchanged against the upstream Bun client.

median of 5 interleaved runs, warm 100k-row DB dev p50 this PR p50 dev ops/s this PR ops/s
Node point read 13.6 µs 7.9 µs 23.6k 28.0k
Node Drizzle 5-row tx 89.4 µs 72.5 µs 8.4k 9.6k
Node withTransaction 5-row tx 100.0 µs 85.0 µs 7.2k 8.4k
Bun point read 10.3 µs 10.4 µs 26.0k 25.1k
Bun Drizzle 5-row tx 81.2 µs 81.7 µs 7.2k 7.7k
Bun withTransaction 5-row tx 100.4 µs 94.8 µs 5.9k 6.7k

Bun is flat within noise; Node is faster, mostly from the statement cache. Not measured: cold reads on a large database, and lock contention with another process holding the file.

👀 Read the BEGIN IMMEDIATE note above and sqlite.workerd.ts; everything else is deletion or tests.


Added by the opencode bot (AI-generated).

Related: #53879

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks sound. On dev, the new Node test fails with UnknownError. With this PR it gets ConstraintError, WAL is on and the Drizzle transaction commits. I checked this by running the PR's test against both checkouts. The Drizzle, workerd, interrupt, bundle, database, migration and session tests pass (825 tests), and typecheck:tsc in packages/core is clean. @effect/sql-sqlite-node 4.1.0 is built on node:sqlite with no native addon or extra dependencies, so the Node build doesn't gain a native module. BEGIN IMMEDIATE applies only to writable clients, and as you note, no Bun or Node production code calls withTransaction directly.

Public API is unchanged. Only internal #sqlite modules in packages/core are affected, plus the dependency changes.

Two small notes, neither blocking:

  • The Bun case in sqlite-node.test.ts also passes on dev under Bun 1.4.2, including the read-only reopen. On this Bun version it doesn't catch the regression described in the PR.
  • Nothing passes timeout to sqliteLayer (database.ts only passes filename), so the timeout → busyTimeout alias in sqlite.node.ts is unused. The mapping itself is correct.

Related: #53879

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier notes are resolved. The new commit removes the unused timeout alias from sqlite.node.ts, which now passes its config straight to SqliteClient.layer. The description now says the Bun read-only case only catches the regression on Bun 1.4.4 and later, which matches what I saw on 1.4.2. The Node, bundle and database tests pass on the new head, and typecheck:tsc in packages/core is clean. Public API is unchanged.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commit adds tests and fixes a comment. The added Drizzle typed insert().returning() and select() checks pass on Node, including a working query after a constraint failure. The new tests passed in 5 consecutive runs, and typecheck:tsc in packages/core is clean. Public API is unchanged.

One suggestion (inline): the copied interrupt sweep in the Node fixture could be dropped, to keep that test small.


const typed = sqliteTable("typed", { id: integer().primaryKey(), name: text().notNull() })

// Same sweep as sqlite-interrupt.test.ts. It runs here because

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This copied sweep is the most intricate part of a test that otherwise checks simple, deterministic results. It depends on MaxOpsBeforeYield and counted yieldNows, and runs 62 fresh client layers inside a subprocess. Your note in sqlite-interrupt.test.ts says the Node client shares makeSqliteAcquirers with the Bun client, which is already swept there, so it adds little coverage. I'd drop it (and leakedTransaction/leakedReserve from the expected output) to keep the Node test focused on WAL, errors and Drizzle queries.

@kitlangton
kitlangton merged commit b95c1f3 into dev Oct 31, 2026
24 checks passed
@kitlangton
kitlangton deleted the sqlite-official-clients branch October 31, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant