Skip to content

[Bug] SingleFile (.scdb) mode ignores DatabaseOptions.EncryptionKey, writing data in plaintext - #342

Closed
saltus7 wants to merge 2 commits into
MPCoreDeveloper:masterfrom
saltus7:master
Closed

saltus7 wants to merge 2 commits into
MPCoreDeveloper:masterfrom
saltus7:master

Conversation

@saltus7

@saltus7 saltus7 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes a critical bug where SingleFile (.scdb) storage mode ignored DatabaseOptions.EncryptionKey, writing data in plaintext even when encryption was configured.

Root Cause

The SingleFileStorageProvider accepted the encryption configuration but never instantiated AesGcmEncryption or intercepted the I/O paths. Consequently, WriteBlockAsync, ReadBlockAsync, GetReadSpan, and GetReadStream operated directly on the raw FileStream, bypassing encryption entirely. (Directory mode was unaffected as it correctly wired ICryptoService via Database.Core.cs).

Changes

  • src/SharpCoreDB/DatabaseExtensions.cs: Resolves ICryptoService from DI when encryption is enabled and passes it to the SingleFile provider.
  • src/SharpCoreDB/Storage/SingleFileStorageProvider.cs: Wires AesGcmEncryption into the provider and intercepts all block read/write/stream paths to encrypt/decrypt data at rest.
  • tests/SharpCoreDB.Tests/Security/SingleFileEncryptionTests.cs: Adds a comprehensive 12-test regression suite covering plaintext-at-rest, correct-key roundtrip, wrong-key rejection, and file integrity.
  • src/SharpCoreDB/DatabaseOptions.cs, src/SharpCoreDB/SingleFileTable.cs, src/SharpCoreDB/Storage/Scdb/SingleFileDatabase.Batch.cs: (Included from local build/test adjustments).

Also included

  • global.json (root): Added test.runner config for MTP-native dotnet test on .NET 10 SDK. Without this, dotnet test fails with error Testing with VSTest target is no longer supported.

Fixes #341

Verification

  • Build succeeds (warnings are non-fatal)
  • Full test suite passes: 2,416 passed, 16 skipped, 0 failed (Total: 2,432)

scdb-dev added 2 commits August 28, 2026 22:00
- Added test.runner config to global.json for MTP-native dotnet test
- Build succeeds (291 warnings, non-fatal)
- All 2,404 tests pass, 16 skipped, 0 failed
@sonarqubecloud

Copy link
Copy Markdown

MPCoreDeveloper pushed a commit that referenced this pull request Aug 29, 2026
…n fix (#341)

Credit @saltus7 (PR #342) for independently root-causing issue #341 and submitting a complete proposed fix (AES-256-GCM at rest on every block I/O path + 12-test regression suite) before v1.9.7 shipped. Adds CONTRIBUTORS.md, CHANGELOG/NuGet-README credit lines and a README pointer, plus a Co-authored-by trailer so the contribution is recorded in git history.

Co-authored-by: saltus7 <saltus7@users.noreply.github.com>
@MPCoreDeveloper

Copy link
Copy Markdown
Owner

Thank you so much for this contribution — and sincere apologies for the crossed wires on our side.

The short version: your diagnosis was 100% correct, and your PR arrived before we shipped our own fix. We resolved issue #341 with our own implementation in v1.9.7 (commit 16ed7e22) and only found your PR after the fact. That's on us — you deserved to be looped in from the start.

What you got right:

  • Root cause — SingleFileStorageProvider accepted EnableEncryption/EncryptionKey but never instantiated a cipher; WriteBlockAsync, ReadBlockAsync, GetReadSpan and GetReadStream bypassed encryption entirely, and directory mode was correctly wired (your note was spot on).
  • The fix surface — wire AesGcmEncryption into the provider and intercept every block I/O path.
  • The tests — your 12-test suite (plaintext-at-rest, correct-key round-trip, wrong-key rejection, file integrity) covers exactly the right invariants; our shipped SingleFileEncryptionTests validate the same ground.

Credit — we've made it permanent:

Since v1.9.7 already ships the fix and your branch is based on an earlier master, merging the PR as-is would conflict heavily with the released code — so I'm closing it as superseded by v1.9.7, with genuine thanks and full credit preserved. If any part of your implementation (e.g., the global.json test-runner tweak for the .NET 10 SDK) would be valuable on its own, we'd be happy to review it as a fresh PR.

We'd love to see more contributions from you — thank you for the excellent work. 🙏

@saltus7

saltus7 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator Author

Thank you for the incredibly thoughtful response and for ensuring the credit was preserved in the git history and release notes. I really appreciate the professional handling of the crossed wires!

It's great to see the fix shipped in v1.9.7. I'm evaluating SharpCoreDB for a local-first edge-node architecture (which requires strict AES-256-GCM at-rest encryption and single-file deployment), so finding and resolving this was a critical step for my own adoption of the library.

I'll hold off on the global.json PR for now to keep the noise down, but I'll definitely be keeping an eye on the project and look forward to future contributions as I integrate v1.9.7 into my stack. Thanks again for the excellent maintainership!

@saltus7

saltus7 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator Author

Quick follow-up: I've now reviewed the v1.9.7 implementation in detail, and I want to give credit where it's due.

My PR handled the happy path — encrypt on write, decrypt on read — but it left several serious gaps that I didn't catch. Your fix addressed all of them:

  • Blocking delta updates and in-place UpdateBlockAsync when encryption is enabled (ciphertext regions aren't positionally independent — my version would have silently corrupted files)
  • Blocking raw GetWriteStream (my version left an open bypass that could write plaintext to an "encrypted" database)
  • Validating checksums against ciphertext rather than plaintext (my approach was backwards and could mask on-disk corruption)
  • Enforcing encryption-mode consistency on open (my version would silently misinterpret a file opened with the wrong mode)
  • Zeroizing the cipher on dispose

My submission was a prototype. Yours is production-grade. The attention to edge cases — especially the ones that would have caused silent data corruption rather than loud failures — is exactly what this kind of security-critical path needs.

Thanks for taking the time to do it right. Moving forward with my integration on v1.9.7 now.

MPCoreDeveloper added a commit that referenced this pull request Nov 12, 2026
* build: pin the .NET 11 GA SDK and define Runtime Async once for the whole line

.NET 11 GA shipped 2026-11-10, so master moves off the RC: global.json (both copies) pins 11.0.100 with
rollForward latestPatch and no allowPrerelease, the five net11 workflows install 11.0.x without
include-prerelease (ci-net10.yml untouched), and the server Dockerfile comment names the GA images.

While doing it, Features=runtime-async=on turned out to be three hand-copied project lines (core,
EventSourcing, its tests) while the other 72 projects compiled classic - although docs/net11/UPGRADE.md
already described it as a property of the whole line. It is now defined once, in a new
Directory.Build.targets, conditioned on net11.0, with <UseRuntimeAsync>false</UseRuntimeAsync> as the
per-project opt-out.

Why targets and not props: a TFM-conditioned switch in Directory.Build.props is evaluated before the project
body, where $(TargetFramework) is still empty, so it silently never applies - measured, not reasoned (163
state machines and 71,4 B/call survived with the block in props; the same block in targets gave 2 and 0).

Measured on the GA runtime, 1.000-call loop: an awaited, synchronously-completing async Task<int> allocated
71.424 bytes before and 0 after; SharpCoreDB.dll went from 163 compiler-generated IAsyncStateMachine types
to 2 (the async iterators, which runtime async does not cover at GA).

Verified: SharpCoreDB.CI.slnf builds with the GA SDK (0 errors); core suite 2124 / 0 / 0 / 0.

* feat(net11): opt-in union outcomes, and canaries that pin the GA-era async codegen

SharpCoreDB.Unions (net11.0-only, additive) exposes QueryOutcome (rows / not found / failure),
ExecuteOutcome (rows affected / failure) and TableLookup (found / missing) as C# 15 union declarations with
seven record class cases, so a switch over them is exhaustive at compile time. Nothing is retrofitted:
QueryResult, ExecuteResult and every ExecuteSQL signature stay byte-identical, and BuildResult deliberately
stays a record hierarchy (a union there would break a shipped type). The union keyword form was chosen over
the [Union] class shape in the design doc because the GA compiler generates that shape itself.

Two canaries pin facts that would otherwise drift silently:
- RuntimeAsyncCodegenTests fails when the runtime-async flag is dropped (-p:UseRuntimeAsync=false: 163 state
  machines and 71,4 B/call) and passes on the GA build (2 state machines, 0 B/call on an awaited
  synchronously-completing call).
- UnionOutcomeTests carries no discard arm in any switch, so exhaustiveness is enforced by the compiler.

Docs: UPGRADE.md gains the GA section and the measured Runtime Async limits (the awaited shape is the win; a
blocking GetResult still materialises the task; async iterators keep their compiler state machine),
UNION_TYPES_DESIGN.md records that the design shipped, and ROADMAP/INDEX are updated to match.

Verified: union canaries 3/3, async canaries 2/2, core suite 2124 / 0 / 0 / 0.

* perf: measure what the gate can actually certify, and cut one real allocation on batch INSERT

The write-path gate's UPDATE cell is bimodal (~215k vs ~420k ops/sec) and six dials were tested and refuted
for it: machine load (INSERT/READ stay inside 1,07x in the same reps), warm-up depth (8/16/2 passes), GC
frequency (2 MB gen0), GC mode (DOTNET_gcServer=1 keeps the split at higher absolutes), tiering
(DOTNET_TieredCompilation=0), hash-index maintenance (SHARPCOREDB_HASH_INDEXES=0) and compaction
(SHARPCOREDB_COMPACTION_THRESHOLD=1e9). What the data says instead: inside every run the fastest rep is
50-70% above the median and the fastest rep agrees across runs to ~1%, a ~50 ms GC pause lands in 2 of 54
measured phases, and most slow passes carry one single-stage stall whose stage varies per pass. The mode is
a per-rep disturbance, and the median is the wrong statistic for a 22-35 ms window.

So the gate now reports and compares capability:
- GateRow prints a `best` column (fastest of the reps); the verdict still uses the median.
- --write-baseline records medians in raw/default and bests in rawBest/defaultBest.
- When a baseline carries the bests, a capability table (best-to-best) decides the verdict and the spread is
  reported as information; median-only baselines keep the old behaviour exactly.
- Recording is no longer gated on the rep spread (it is not a verdict - and on this box that gate made a
  baseline impossible: every attempt exited 2 with 2,63x-3,29x spreads), and instead refuses to write when any
  metric's best is more than 25% below the one on disk, which is what "a baseline recorded during a
  regression silently accepts that regression" is about.
- SHARPCOREDB_PHASE_DIAG=1 adds wall vs process CPU vs GC counts/pause per phase.

With that instrument the INSERT path got its first measured cut: NormalizeInsertRow allocated a
bool[Columns.Count] per row and rebuilt it from the same columnIndexMap every time, but which columns a
statement names is a property of the statement. Hoisting it to one mask per batch took the dictionary-free
object[] overload from 1.838.776 to 1.787.608 bytes for 2.000 rows - 919 -> 894 B/row (-2,7%) - with identical
behaviour, pinned by the new InsertBatchAllocationTests canary at a 900 B/row ceiling.

Refuted in the same session and reverted, so nobody re-derives it: the identity-remap fast path (skip the
normalized object[] when the column order matches) saves 3 B/row, because an insert never names the hidden
AUTO column _rowid and the normalized row is required to carry it.

Docs: docs/manual/performance.md documents server GC as the measured host lever (~1,75x INSERT, +20% READ,
costs a heap per core) and records that GC frequency and tiering are not levers for this engine.

The baseline file is re-recorded on the GA runtime with both statistics; review its diff before landing.

Verified: gate PASSED on the capability comparison (raw INSERT 0,98x, default UPDATE 1,03x), suite 2124 /
0 / 0 / 0, canaries 57 / 0 / 0 / 0.

* docs: the .NET 11 GA release narrative, the performance plan, and the session record

CHANGELOG gains the GA upgrade, the opt-in union surface, the GA-runtime gate evidence and the batch-INSERT
allocation cut. PROMOTION_TO_MAIN's GA checklist is closed item by item - SDK pin, CI pins, unions shipped,
gate run (INCONCLUSIVE then a documented PASSED, plus the capability statistic that made it certifiable), the
baseline recorded by the owner, the image rebuilt on the GA base images - and item 5 is stated as owner-only
rather than started, because version metadata, packing and publishing are the rail.

docs/net11/GA_RELEASE_NOTES.md is the release note draft plus the checklist: what the repository verified
(slnf 0 errors, 2124/0/0/0, canaries 57/0/0/0, gate PASSED, docs 9/9, image reports 11.0.0) and the six owner
steps (version bump, tag, merge that packs and publishes, GitHub release, image push, optional quiet-box
re-record).

docs/performance/NET11_C15_PERFORMANCE_PLAN.md is the standing plan: what is already in place, where the time
actually goes per UPDATE stage, the refuted-dial list (including the identity-remap fast path), the UPDATE
mode resolved as a statistic, phases P0-P3 and the zero-regression rails.

WORKLOG carries session 87 and its eight addenda: the GA upgrade, the runtime-async measurement, the union
implementation, the UPDATE mode hunt (seven refutations), the capability gate, the INSERT cut, the refuted
identity path, and the "no regression against the previous version" check (gate against the 2026-09-27
baseline: PASSED, worst cell 1,08x, UPDATE faster on both arms).

* ci: keep the allocation canaries out of the instrumented coverage run

The first CI run on this branch was red in two places. One is mine and is fixed here: the coverage job runs the
suites a second time under `dotnet-coverage collect`, and instrumentation changes what the JIT is allowed to
do, so both allocation canaries stopped measuring the engine and started measuring the collector. Evidence from
job 131142938452: RuntimeAsyncCodegenTests reported 71.424 bytes over 1.000 awaited calls - exactly the classic
(non-runtime-async) figure, i.e. the elision is gone under instrumentation, so asserting 0 was never true there -
and InsertBatchAllocationTests read 902 B/row against its 900 ceiling.

Both are now `[Trait("Category","Allocation")]`, and the coverage job passes -Filter
"...&Category!=Allocation". They still run and still gate in the uninstrumented Build and Test jobs: verified
locally, `Category=Allocation` selects exactly those 2 tests (green) and the coverage filter runs 2122 (green).
AGENTS.md documents the convention so the next allocation canary is tagged the same way.

The second failure is not this branch's: CI's "Fail on vulnerable NuGet packages" step fails on NU1903 by Aspire.Hosting
13.6.4 (GHSA-f7vh-h5g3-7jrh / CVE-2026-120006, high, published 2026-11-10, patched in 13.6.5) in
src/SharpCoreDB.Aspire.Hosting. No package moved in this branch. Bumping a package version is the owner's under the
standing rail, so it is recorded as a release blocker in docs/net11/GA_RELEASE_NOTES.md with the advisory, the
patched version and the file to change - not patched here.

* docs(worklog): session 87 addendum 9 - release pre-flight and the two CI reds

Records the pre-flight evidence (slnf 0 errors, suite 2124/0, canaries 57/0, gate PASSED, docs 9/9, image
reports 11.0.0), the five commits on this branch, and the two CI failures separated by cause: the coverage job
was mine (allocation canaries measure the collector under instrumentation - 71.424 bytes per 1.000 awaited calls
is exactly the classic figure - fixed with the Allocation trait and the coverage filter), and the three Build and
Test jobs are one dependency advisory (NU1903 Aspire.Hosting 13.6.4, GHSA-f7vh-h5g3-7jrh, patched in 13.6.5)
which is a package-version change the rail keeps with the owner.

* docs(status): record the .NET 11 GA line on the project status page

The status page still described 2.1.2 as the whole story, with no mention of the GA upgrade. Adds the factual
line: GA SDK pin, Runtime Async once for every project (0 B/call on an awaited synchronously-completing call,
2 state machines instead of 163), the opt-in union surface, the capability gate, and the validated numbers -
slnf 0 errors, core suite 2,124/0, canaries 57/0, gate PASSED (worst cell 1,08x, UPDATE faster on both arms),
docs 9/9, image reporting 11.0.0. Points at net11/GA_RELEASE_NOTES.md for the checklist and the one open
blocker, and states that version metadata, the tag and the publishing merge stay owner actions.

* fix(linq2db): execute the adapter on SharpCoreDB, not SQLite (#497)

SharpCoreDBDataConnection passed the connection string to ProviderName.SQLite, which resolves to Microsoft.Data.Sqlite and the native e_sqlite3 library: a password was refused ('the native SQLite library doesn't support encryption') and, without one, CreateTableAsync/InsertAsync wrote an unencrypted 'SQLite format 3' file that SharpCoreDB cannot read. The adapter is the SQLite dialect, not the SQLite engine. CreateOptions now pairs SQLiteTools.GetDataProvider with a live SharpCoreDBConnection from SharpCoreDB.Data.Provider - linq2db generates the SQLite SQL, SharpCoreDB executes it, owns the file and applies AES-256-GCM - with bulk copy pinned to MultipleRows, GUIDs written as text instead of a blob literal, and GetUnderlyingConnection returning the SharpCoreDB connection instead of closing it to release a SQLite handle. The string constructor, CreateSharpCoreDBOptions and UseSharpCoreDB all route through that one definition.

Making that work exposed seven engine defects, each a SQLite-compatibility gap and each fixed here: (1) INSERT assumed exactly one space after INSERT INTO and read the table name up to the next space, so a pretty-printed statement failed with 'Table notes](...' and INSERT/INTO across a newline threw ArgumentOutOfRangeException - one scanner now reads the token for all four INSERT parsers; (2) a SELECT list was not a projection (SELECT c, b, a returned a, b, c, aliases dropped, cached point lookups skipped it), so every ordinal-mapping consumer read the wrong column; (3) WHERE [t].[col] and WHERE NOT [t].[col] were not truth tests (all rows in directory mode, none in single-file); (4) LIKE ... ESCAPE matched nothing, missing from all five copies of the matcher; (5) SELECT EXISTS (SELECT ...) never reached an executor; (6) byte[]/Guid written as 0x.../X'...' literals failed as base64; (7) the reader threw 'Invalid read position' when a column type was asked for before the first Read(). Also: UPDATE SET [col] no longer drops the assignment silently, and the functional SQL translator now selects the columns its IS SOME/IS NONE post-filter reads.

Tests: MultilineStatementTests (6), BlobLiteralTests (3), SelectSemanticsTests (14, both storage modes), StorageTests (password honoured, no SQLite header, reopens through the core provider); the linq2db fixture issues one statement per command and one culture-dependent functional test now formats invariantly. Verified: red before / green after on the parser fix, core 2149, linq2db 25, EF Core 116, Functional 38, Dapper 3, SharpCoreDB.CI.slnf build clean; the SingleFileDirectoryParityTests canary caught a BETWEEN group-numbering regression during the work and is green.

Refs #497. Not published: no package version, release metadata or global.json is touched by this commit.

* release(2.1.3): version bump, changelog and docs for the LINQ to DB patch

Version 2.1.2 -> 2.1.3 in SharpCoreDBVersion (Directory.Packages.props) and in the 28 packable projects under src, FileVersion 2.1.3.0 for the core assembly, and the server image ARG VERSION=2.1.3 (it still said 2.1.0). Each package's PackageReleaseNotes gains a 2.1.3 summary and keeps the 2.1.2/2.1.1/2.1.0 history after it.

Docs: CHANGELOG [Unreleased] closed into [2.1.3] - 2026-11-12 with the four Fixed entries and the Added entries; README current-release banner, release table, install list and a new What's new in v2.1.3 section (2.1.2 demoted to previous, 2.1.0 to earlier); src/SharpCoreDB/NuGet.README.md title, current-version line, footer and a What's New in 2.1.3 section; docs/PROJECT_STATUS.md version, status, current-release paragraph and the GA section (the Aspire 13.6.4 blocker is recorded as closed by the 13.6.5 bump); AGENTS.md line updated; 25 markdown files had their install snippets moved to 2.1.3. The README's packable count is corrected from 27 to 28 (SharpCoreDB.Aspire.Hosting).

Verified on this tree: dotnet msbuild -getProperty:Version reports 2.1.3 for the packable projects; SharpCoreDB.CI.slnf builds clean; core 2149/0, linq2DB 25/0, EF Core 116/0, Functional 38/0, Dapper 3/0. GitHub runners: .NET 11 GA is 11.0.100 (released 2026-11-10, linux-x64/osx-arm64/win-x64 SDK archives published) and ci.yml installs it with setup-dotnet 11.0.x, which satisfies the global.json pin (11.0.100, rollForward latestPatch) on ubuntu-latest, windows-latest and macos-latest.

* chore(deps): Aspire 13.6.5 - SDK, Hosting package and AppHost together

Bumps the three Aspire pins from 13.6.4 to 13.6.5 as one change, because they are one dependency: the AppHost SDK supplies the orchestration and dashboard targets that Aspire.Hosting.AppHost compiles against, and SharpCoreDB.Aspire.Hosting is documented as matching those.

Supersedes dependabot PRs #499 (Aspire.Hosting) and #500 (Aspire.Hosting.AppHost), which bump one package each and leave the SDK pin and the central entries at 13.6.4.

Verified: dotnet build of src/SharpCoreDB.AppHost (SDK 13.6.5 + Aspire.Hosting.AppHost 13.6.5), src/SharpCoreDB.Aspire.Hosting, and SharpCoreDB.CI.slnf --configfile NuGet.Config /p:UseLocalProjectReferences=true - all succeeded, no NU1xxx warnings.

* chore(security): force minimatch 11.0.1+ in the JS client lockfile

Dependabot alert #249 (GHSA-g3hv-75wg-m249, CVE-2026-118764): minimatch >= 7.0.0 and < 11.0.1 is vulnerable to a denial of service via globparts amplification. The client resolved two copies inside that range - 10.2.6 through eslint 10 / @typescript-eslint / glob 13, and 9.0.9 through ts-jest -> @jest/transform -> babel-plugin-istanbul -> test-exclude -> glob 10 - and the patched version is 11.0.1.

Adds "minimatch": "^11.0.1" to the existing overrides block (the same mechanism already used for protobufjs and esbuild) and regenerates the lockfile with npm install --package-lock-only, which resolves 11.0.8 and removes the nested 9.0.9 plus its nested brace-expansion/balanced-match. Overrides apply to this project install only, so the published @sharpcoredb/client dependency graph is unchanged. Note the resolved 11.0.8 declares node 20 || >=22 for dev installs; the package engines field (>=16) is untouched.

Verified: npm audit no longer lists minimatch; npm run typecheck, npm run build (CJS/ESM/DTS) and npm test (16/16) pass.

* docs(changelog): record the 2.1.3 security bumps (Aspire 13.6.5, minimatch)

Adds the ### Security section to [2.1.3]: the four Aspire pins moving together (CVE-2026-120006 high, CVE-2026-118740 critical, Dependabot #341/#342/#343/#644) and the minimatch override in the JS client (GHSA-g3hv-75wg-m249, #249). Records that deepmerge (#246) has no patched release and stays open.

* test(allocation): give the InsertBatch canary each platform's measured ceiling

The canary recorded 894 B/row on win-x64 (GA 11.0.100) and a single 900 B/row ceiling, so the first Linux and macOS runs reported the same code at 903 B/row and failed both jobs: 1.806.872 bytes on linux-x64 and 1.806.856 on osx-arm64 against 1.787.608 on win-x64. The 9 B/row delta is the runtime's accounting for identical code, not a code path.

The ceiling is now per platform from its own measurement - 900 on Windows (894 measured), 915 on Unix (903 measured). That is ~1% headroom and still below the pre-hoist 919 B/row (about 928 on Unix), so reintroducing the allocation this canary was written to catch still fails on every platform. Verified on win-x64: the test passes with the new logic; the Unix figures come from the CI TRX of run 43306930953.

---------

Co-authored-by: MPCoreDeveloper <MPCoreDeveloper@users.noreply.github.com>
Co-authored-by: MPCoreDeveloper <support@mpcoredeveloper.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] SingleFile (.scdb) mode ignores DatabaseOptions.EncryptionKey, writing data in plaintext

2 participants