Repository navigation

[MSSQL]: Disable Automatic Schema change handling #727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

stevensJourney merged 29 commits into main from sourcetable-metadata-mssql-capture-recording

Aug 11, 2026

.changeset/mssql-capture-instance-pinning.md

-Original file line number
+Diff line change
@@ -0,0 +1,21 @@
+    ---
+    '@powersync/service-module-mssql': minor
+    ---
+    Pin MSSQL source-table bindings to a specific CDC capture instance.
+    A source table now persists the object ID of the CDC change table it replicates from, and restores that
+    binding across restarts, so a stream always replicates the capture schema it started with. Existing
+    bindings without this metadata are backfilled at job startup with the same capture instance the previous
+    streaming logic would have selected.
+    SQL Server allows two capture instances per table. When a newer one appears, the current replication
+    process keeps polling the instance it is bound to and warns. Deploy a new sync config to use the newer
+    instance; the active sync config can continue serving clients through the old instance while the new sync
+    config snapshots and catches up. Keep the old instance available until the new sync config becomes active.
+    If CDC is disabled for a bound table, or its capture instance is removed while the source table and replica
+    identity remain unchanged, replication stops with `PSYNC_S1601`. A newly configured table that cannot be
+    replicated yet — it does not exist, or CDC has not been enabled for it — stops replication with
+    `PSYNC_S1602` rather than being skipped. Dropping, recreating, or otherwise changing the source identity of
+    a previously bound table stops replication with `PSYNC_S1603`.

.changeset/mssql-fixed-table-set.md

-Original file line number
+Diff line change
@@ -0,0 +1,47 @@
+    ---
+    '@powersync/service-module-mssql': minor
+    ---
+    [Breaking Change]
+    Pin the replicated schema at the start of replication for SQL Server connections.
+    SQL Server does not deliver schema changes in the replication stream, so they can only be found by
+    polling, and a poll is never atomic with a commit. That leaves a gap where checkpoints are committed
+    against a schema that has already changed — producing a state the source database was never in, which
+    clients cannot detect. PowerSync therefore no longer adopts schema changes automatically.
+    The recommended schema-change workflow treats a sync config deployment as the consistency boundary. A
+    replication stream keeps the exact table set and CDC capture-instance identities it selected when it
+    started; it does not adopt schema or table changes in place. Tables must therefore be listed explicitly
+    rather than selected with wildcards.
+    What this means when making schema changes:
+    - **Add a table:** create the table, enable CDC, add its exact qualified name to the sync config, and
+      deploy a new sync config. The new sync config snapshots the table before becoming active.
+    - **Change captured columns:** where SQL Server allows a rolling change, apply the DDL and create a second
+      capture instance with the desired captured columns. Update explicit sync queries if needed, deploy a
+      new sync config, wait for its snapshots to finish and for it to become active, and only then remove the
+      old capture instance.
+    - **Make an identity-breaking change:** for a primary-key change, identity-column rename, or another
+      change that requires CDC to be disabled, disable CDC, apply the DDL, re-enable CDC, update the sync
+      config if needed, and deploy a new sync config. The current replication process cannot adopt the
+      replacement in place, so this results in downtime. If it observes CDC being disabled first, it stops
+      with `PSYNC_S1601` because its capture instance was removed. If it next reconciles after the replica
+      identity has changed, it stops with `PSYNC_S1603` instead. Disabling and re-enabling CDC without changing
+      the replica identity stops with `PSYNC_S1601`.
+    - **Drop a table:** remove it from the sync config, deploy a new sync config and wait for it to become
+      active, and only then drop the source table. Dropping the table first stops replication with
+      `PSYNC_S1603`; already-replicated data is retained until the new sync config becomes active.
+    - **Rename or drop/recreate a table:** update the sync config to express the intended table explicitly,
+      ensure the resulting table is CDC-enabled with the expected schema and replica identity, and deploy a
+      new sync config. The current replication process stops with `PSYNC_S1603` if its bound table is removed.
+    - **No deployment is needed** for changes that do not affect the captured row shape or replica identity,
+      such as many index, constraint, or default changes. A non-identity column change also needs no PowerSync
+      action if that changed column does not need to be replicated; otherwise use the rolling capture-instance
+      workflow above.
+    This sequencing keeps the old table or capture instance available while the new sync config snapshots and
+    becomes active whenever a rolling transition is possible. If a pinned capture instance or bound table is
+    removed first, the current replication process stops and a new sync config must be deployed.

.changeset/postgres-storage-replica-id-type-oid.md

-Original file line number
+Diff line change
@@ -0,0 +1,11 @@
+    ---
+    '@powersync/service-module-postgres-storage': patch
+    ---
+    Fix hydration of replica id column type ids in Postgres bucket storage.
+    `source_tables.replica_id_columns` is persisted with a `type_oid` key, but the decoding codec declared
+    `typeId`, so every `SourceTable` read back from a row lost its column type ids. This affected
+    `getSourceTableStatus()` and the tables returned for dropping, and now also the identity comparison
+    used by source table reconciliation. The codec matches the persisted key - no data migration is
+    needed.

.changeset/source-owned-table-reconciliation.md

-Original file line number
+Diff line change
@@ -0,0 +1,14 @@
+    ---
+    '@powersync/service-module-mongodb-storage': minor
+    '@powersync/service-module-postgres-storage': minor
+    '@powersync/service-core': minor
+    ---
+    Add source-owned `SourceTable` reconciliation.
+    `resolveTables()` now queries all overlapping persisted candidates and passes them to a
+    source-provided reconciler that returns compatible and incompatible tables, can return
+    modified compatible copies, and supplies values used for potential new records. Storage persists allowlisted
+    source metadata differences but never interprets them. MongoDB v1/v3 and PostgreSQL storage were
+    refactored to this candidate-first model (PostgreSQL gains a nullable `source_metadata` JSONB column
+    via migration).

AGENTS.md

-Original file line number
+Diff line change
@@ Expand Up @@
     Use this section when working on source replication modules, replication streams, source connector behavior, bucket storage writers, checkpoints, write checkpoints, initial snapshots, sync config deployment, or replication tests.
+    ### User-Facing Sync Config Terminology
+    - In changesets, documentation, warnings, and errors, describe the operator action as **deploy a new sync config** (or **redeploy the sync config** when no config edit is required).
+    - Do not tell users to create or deploy a sync config "as a new replication stream." Replication streams, processing configs, and jobs are implementation details, not additional operator actions.
+    - When the internal effect is useful context, explain it separately: a sync config deployment starts replacement processing, and the active sync config can keep serving clients while the replacement snapshots and catches up. Depending on storage support, that work may use a separate replication stream or a processing config in the existing stream.
+    - For rolling source schema changes, state that the old table or capture instance must remain available until the new sync config becomes active. Call out expected downtime separately when the change requires disabling CDC or otherwise removes the active binding first.
     ### Load Spec Context Selectively
     Do not load every replication spec page by default. Start with the entry point, then load only the pages that match the task.
@@ Expand Down @@

docs/replication/09-resolve-tables-flow.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -39,6 +39,7 @@ This is a SourceTableRef with additional metadata used for replication:
  
    1. objectId / relation id: The underlying id of the table/collection in the source database. This is used to track renames.

    2. replicaIdColumns: The columns and types representing the "replica identity" for the table.

    3. sourceMetadata (nullable): Opaque, source-specific identity metadata. `null` means no metadata has been recorded. Storage persists and hydrates non-null values verbatim and never interprets them.

    ### SourceTable

    @@ -86,14 +87,47 @@ Each matching pattern is resolved independently.
  
    Conceptually it does:

    1. Look up existing `SourceTable` records that match the physical identity.

    2. Resolve matching sources through the parsed sync config set's definition mapping.

    3. Determine which persisted bucket and parameter definition ids are already covered.

    4. Create missing `SourceTable` records when coverage is incomplete.

    5. Return the `SourceTable` records that should receive replicated data.

    1. Find persisted records that overlap by `(schema + name) OR object/relation id`.

    2. Let the source connector classify them as compatible or incompatible.

    3. Resolve matching sync config definitions and reuse compatible records where possible.

    4. Persist source metadata updates and create records for missing definitions.

    5. Return the records to replicate into and the incompatible records to drop.

    Important: one physical table can resolve to multiple `SourceTable` records when sync config definitions have been added over time.

    ### Source-owned candidate reconciliation

    `resolveTables` delegates compatibility checks to the connector through `reconcileSourceTables`. If none is provided, it uses the default identity comparison.

    - Storage hydrates overlapping records into `SourceTable` values, then passes isolated clones to the connector through a read-only `SourceTableCandidate` type.

    - The source connector classifies those candidates and provides `sourceMetadata` for updated or new records.

    - Storage persists those changes and manages record creation, snapshot state, memberships, and incompatible records.

    The reconciler must not perform slow source queries. Candidate state is cloned before the callback; `withSourceMetadata()` is the supported way to return a metadata change. After reconciliation, storage rematerializes results from its original `SourceTable` values, so snapshot state, memberships, and identity remain storage-owned even if callback code mutates a candidate or bypasses the type boundary with a cast. Changes to the table identity still require a new record and snapshot.

    ### MSSQL: the replicated table set is fixed at deploy

    MSSQL rejects table wildcards and requires every configured table to exist with CDC enabled at startup. This keeps the table set fixed for the life of the stream. Polling cannot detect a new table atomically with a commit, so adding a table, enabling CDC, or renaming a table into scope requires a new sync config deploy.

    Dropping, renaming, or recreating a replicated table, or otherwise changing its persisted source identity, fails the job with `PSYNC_S1603`. Continuing would risk committing past changes that have not been read from that table. Existing replicated data is retained; it is removed when the replacement sync config is resolved.

    This failure applies when none of the overlapping persisted records match the discovered source identity. If at least one record remains compatible, it anchors the current binding; other incompatible overlapping records are stale storage state and can be removed without adopting a replacement identity.

    ### MSSQL capture-instance pinning

    SQL Server can keep two capture instances for one table. Each binding is pinned to the capture table's object id:

    - New and legacy bindings use the newest available instance.

    - A pinned binding keeps using its instance when a newer one appears and logs a warning.

    - If the pinned instance is removed, replication fails with `PSYNC_S1601`. A replacement is not adopted automatically because it may capture a different schema.

    - Conflicting persisted pins are treated as invalid state.

    Every configured table is resolved before streaming starts, which also backfills pins for legacy records. A table with no persisted binding that does not exist or does not have CDC enabled fails with `PSYNC_S1602`. If a persisted binding exists, a missing or identity-mismatched table fails with `PSYNC_S1603`, while a table whose CDC capture instance is unavailable fails with `PSYNC_S1601`. This preserves the original binding and its replicated data across normal job restarts.

    To adopt a new capture instance, deploy a new sync config. The replacement processing binds to the new

    instance and snapshots as needed. Keep the old instance available until the new sync config has finished

    its snapshots and becomes active.

    The parsed sync config set matters here. Source objects, hydration state, and definition mappings are identity-bound; resolving sources from one parse and writing them through a batch created from another parse can point at the wrong persisted ids.

    ### 4. Detect differences/conflicts

modules/module-mongodb-storage/src/storage/implementation/models.ts

-Original file line number
+Diff line change
@@ Expand Up / @@ -110,6 +110,10 @@ export interface SourceTableDocument { @@
       replica_id_columns2: { name: string; type_oid?: number; type?: string }[] | undefined;
       snapshot_done: boolean | undefined;
       snapshot_status: SourceTableDocumentSnapshotStatus | undefined;
+      /**
+       * Source-specific metadata. Absent for legacy records.
+       */
+      source_metadata?: storage.JsonValue;
     }
     export interface SourceTableDocumentSnapshotStatus {
@@ Expand Down @@

Navigation Menu

Repository navigation

[MSSQL]: Disable Automatic Schema change handling #727

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!