Skip to content

perf(dash-spv): stream segments to disk instead of encoding them into one buffer - #1088

Merged
ZocoLini merged 1 commit into
devfrom
perf/stream-segment-persist
Sep 29, 2026
Merged

ZocoLini merged 1 commit into
devfrom
perf/stream-segment-persist

Conversation

@ZocoLini

Copy link
Copy Markdown
Collaborator

Segment::persist encoded the whole segment into a Vec before handing it to atomic_write. For a block segment near the tip (up to 87 MB on disk) that Vec grew to 128 MiB on every write of the segment, in the middle of the sync.
atomic_write_items encodes one item at a time through a 1 MiB BufWriter into the temporary file, then syncs and renames it as before. atomic_write keeps its behaviour and shares the temporary file and rename logic.

… one buffer

`Segment::persist` encoded the whole segment into a `Vec` before
handing it to `atomic_write`. For a block segment near the tip (up to
87 MB on disk) that `Vec` grew to 128 MiB on every write of the segment,
in the middle of the sync.
`atomic_write_items` encodes one item at a time through a 1 MiB
`BufWriter` into the temporary file, then syncs and renames it as
before. `atomic_write` keeps its behaviour and shares the temporary file
and rename logic.

Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two
resident segments written on eviction, wallet identical in every run
(14114383 sat, 7112 records, 13389 addresses):

  whole-segment buffer   peak RSS 943 / 948 MiB   7.7 / 7.9 min
  streamed               peak RSS 845 / 855 / 856 MiB   9.4 / 7.7 / 6.5 min

The persist buffer, 68–133 MiB in the earlier heap snapshots, no longer
shows up. The 9.4 min run lost time to peers in the header and filter
header phases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf
@coderabbitai

coderabbitai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 55 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: dashpay/rust-dashcore/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 964e3e33-9b6e-4733-940f-08a7b2af9050

📥 Commits

Reviewing files that changed from the base of the PR and between 5edeecd and b69cece.

📒 Files selected for processing (2)
  • dash-spv/src/storage/io.rs
  • dash-spv/src/storage/segments.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ZocoLini

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@ZocoLini
ZocoLini merged commit b074628 into dev Sep 29, 2026
35 of 36 checks passed
@ZocoLini
ZocoLini deleted the perf/stream-segment-persist branch September 29, 2026 21:13
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.53%. Comparing base (5edeecd) to head (b69cece).
⚠️ Report is 4 commits behind head on dev.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1088      +/-   ##
==========================================
+ Coverage   77.51%   77.53%   +0.02%     
==========================================
  Files         318      318              
  Lines       81270    81294      +24     
==========================================
+ Hits        62997    63033      +36     
+ Misses      18273    18261      -12     
Flag Coverage Δ
core 78.90% <ø> (ø)
ffi 50.77% <ø> (ø)
rpc 20.00% <ø> (ø)
spv 92.25% <100.00%> (+0.06%) ⬆️
wallet 80.22% <ø> (ø)
Files with missing lines Coverage Δ
dash-spv/src/storage/io.rs 97.68% <100.00%> (+0.46%) ⬆️
dash-spv/src/storage/segments.rs 96.93% <100.00%> (+0.20%) ⬆️

... and 4 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant