Repository navigation
perf(dash-spv): stream segments to disk instead of encoding them into one buffer - #1088
Conversation
… one buffer `Segment::persist` encoded the whole segment into a `Vec` before handing it to `atomic_write`. For a block segment near the tip (up to 87 MB on disk) that `Vec` grew to 128 MiB on every write of the segment, in the middle of the sync. `atomic_write_items` encodes one item at a time through a 1 MiB `BufWriter` into the temporary file, then syncs and renames it as before. `atomic_write` keeps its behaviour and shares the temporary file and rename logic. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments written on eviction, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): whole-segment buffer peak RSS 943 / 948 MiB 7.7 / 7.9 min streamed peak RSS 845 / 855 / 856 MiB 9.4 / 7.7 / 6.5 min The persist buffer, 68–133 MiB in the earlier heap snapshots, no longer shows up. The 9.4 min run lost time to peers in the header and filter header phases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf
|
Warning Review limit reachedNext included review available in 55 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: dashpay/rust-dashcore/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai approve |
✅ Action performedComments resolved and changes approved. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #1088 +/- ##
==========================================
+ Coverage 77.51% 77.53% +0.02%
==========================================
Files 318 318
Lines 81270 81294 +24
==========================================
+ Hits 62997 63033 +36
+ Misses 18273 18261 -12
|
Segment::persistencoded the whole segment into aVecbefore handing it toatomic_write. For a block segment near the tip (up to 87 MB on disk) thatVecgrew to 128 MiB on every write of the segment, in the middle of the sync.atomic_write_itemsencodes one item at a time through a 1 MiBBufWriterinto the temporary file, then syncs and renames it as before.atomic_writekeeps its behaviour and shares the temporary file and rename logic.