Repository navigation
Conversation
Read SQLite id columns as text so snowflakes above 2^53 stay exact when the driver returns integers as JavaScript numbers, including Durable Objects. Co-authored-by: Rahul Kashyap <itisrmk@users.noreply.github.com>
🦋 Changeset detectedLatest commit: f388104 The changes in this PR will be included in the next version bump. This PR includes changesets to release 33 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SqlMessageStoragereads snowflake columns (id,request_id,reply_id,last_reply_id, and joined reply ids) underSqlClient.SafeIntegers, then turns them back into ids withString/Snowflake. That is exact only when the driver returns the full 64-bit integer.@effect/sql-sqlite-docannot do that. Durable Objectsql.exec().raw()types every SQLiteINTEGERas a JavaScriptnumber(ArrayBuffer | string | number | null), so a value above 2^53 is rounded on read. The stored integer stays exact. Default snowflakes have been above 2^53 since early 2025, so on a Durable Object nearly every id read back is wrong: a reply is saved with the roundedrequest_id,UPDATE ... SET processed = TRUE WHERE request_id = ...matches nothing, and the message is delivered again on every poll.requestIdForPrimaryKeyand duplicate-envelopeoriginalIdreturn the rounded id too.SQLite selects in this storage now use
CAST(column AS TEXT). Text comes back as the exact decimal digits on every SQLite driver, including drivers that already return bigints. PostgreSQL, MySQL, and SQL Server queries are unchanged.The worker SQL API has no bigint read, so making
@effect/sql-sqlite-dohonorSafeIntegerscannot recover these values. Casting at the query keeps the ids exact. The separate request to stopSnowflake.layerGeneratorfrom overriding an outer generator is left out of this change.Design assessment
Worth accepting: this fixes request/reply identity loss without a public API change. SQLite-only casts at the projection boundary preserve exact decimal IDs while leaving stored INTEGER values, joins, predicates, and numeric ordering unchanged. Casting nullable joined IDs preserves NULL. Other dialect projections remain unchanged.
The first takeover pass only simplifies tests; implementation refactoring and independent review remain separate follow-ups. The repeated projection lists are a candidate for simplification. Generator injection redesign remains out of scope.
Validation
SqlMessageStorage.integration.test.ts, reusing message fixtures and a scoped real SQLite connection. A small connection adapter converts bigint results to numbers to model number-returning drivers without a cross-package Durable Object dependency.nix develop -c env EFFECT_INTEGRATION_TESTS=1 pnpm test --run packages/platform/node/test/cluster/SqlMessageStorage.integration.test.ts -t "sqlite|preserves ids above"— 23 passed (22 existing SQLite tests + 1 regression). PostgreSQL/MySQL were not run in this pass.nix develop -c pnpm lint-fix— passed.nix develop -c pnpm check— passed.Closes EFF-2423
Closes #9483