Skip to content

perf(dash-spv): segment cache two resident - #1086

Merged
ZocoLini merged 2 commits into
devfrom
perf/segment-cache-two-resident
Sep 29, 2026
Merged

ZocoLini merged 2 commits into
devfrom
perf/segment-cache-two-resident

Conversation

@ZocoLini

@ZocoLini ZocoLini commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

PR Hygiene · 92e2439

  • Bots — coderabbitai not yet — /skip-bots proceeds without the ones not yet reported
  • Self-review — post /self-reviewed once the bots are done
  • Within your 5 open PRs
  • Build running
  • Approvals
    • dash-spv (dash-spv/src/storage/blocks.rs, dash-spv/src/storage/filters.rs, dash-spv/src/storage/mod.rs and 3 more) — QuantumExplorer or xdustinface

When every box is checked the PR Hygiene check passes and this can merge.

Summary by CodeRabbit

  • Storage
    • The segment cache now keeps a maximum of two segments resident at a time. Previously, it could keep up to ten.
    • Committed segments are no longer automatically released from the cache during persistence.
  • Filter Handling
    • Filter ranges that begin above zero can be read and cleared, with the cleared state preserved after reopening storage.

ZocoLini and others added 2 commits September 29, 2026 15:55
Reverts ae2a5d6 (#946). Segments were released from memory only when
a sync manager remembered to call `set_committed_height` on the block
and filter storages, and only on the next timed persist. The resident
set should be bounded by the segment cache itself, not by callers
reporting progress to storage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf
With the committed-height release gone, the segment cache itself has to
bound the resident set. Ten 50 000-item segments per cache kept most of
a restore in memory; two keep the peak below what the release achieved,
without costing time.

Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied,
jemalloc heap profiling, wallet identical in every run (14114383 sat,
7112 records, 13389 addresses):

  segments      peak RSS   time      segment loads from disk
                                     (headers/filter headers/filters/blocks)
  10 + #946     982 MiB    8.2 min   -
  10            1490 MiB   8.8 min   84 / 2 / 28 / 0
  2             930 MiB    8.8 min   212 / 103 / 74 / 0
  1             1000 MiB   7.7 min   427 / 742 / 156 / 33

Each cache served ~2.34M requests from memory in every run. One segment
starts reloading block segments of up to 87 MB from disk, which raises
the peak again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf
@coderabbitai

coderabbitai Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: dashpay/rust-dashcore/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 8fe7b0c0-51aa-44ee-b3e1-d6091fac4180

📥 Commits

Reviewing files that changed from the base of the PR and between 5edeecd and 92e2439.

📒 Files selected for processing (6)
  • dash-spv/src/storage/blocks.rs
  • dash-spv/src/storage/filters.rs
  • dash-spv/src/storage/mod.rs
  • dash-spv/src/storage/segments.rs
  • dash-spv/src/sync/blocks/manager.rs
  • dash-spv/src/sync/filters/manager.rs
💤 Files with no reviewable changes (5)
  • dash-spv/src/storage/filters.rs
  • dash-spv/src/storage/blocks.rs
  • dash-spv/src/sync/blocks/manager.rs
  • dash-spv/src/sync/filters/manager.rs
  • dash-spv/src/storage/mod.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The change removes committed-height tracking from segment storage and removes the storage and sync calls that used it. The segment cache’s maximum resident segment count changes from 10 to 2.

Changes

Segment storage and sync

Layer / File(s) Summary
Segment cache policy and lifecycle
dash-spv/src/storage/segments.rs, dash-spv/src/storage/blocks.rs
SegmentCache now has a maximum of 2 resident segments. Committed-height tracking, clean-segment release based on that height, and related tests are removed.
Storage interfaces and forwarding
dash-spv/src/storage/blocks.rs, dash-spv/src/storage/filters.rs, dash-spv/src/storage/mod.rs
Block and filter storage interfaces and their persistent implementations no longer expose or forward committed-height updates. DiskStorageManager no longer forwards those updates. The block segment-release test is removed.
Sync manager call sites
dash-spv/src/sync/blocks/manager.rs, dash-spv/src/sync/filters/manager.rs
Block processing no longer reports the highest applied height. Filter scan initialization and batch commits no longer update the storage watermark.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 92e24

The two-segment cache policy has no established data-loss or read-correctness regression. The change is mergeable subject to normal checks.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 92e24

The smaller cache makes an existing write-failure path more damaging: unsaved data can be discarded after accessing three distinct segments while saving appears successful. This can impair later wallet replay. No new remote access or privilege expansion was established.

Retained concerns

  • Medium · reliability · inferred: The two-resident policy worsens silent loss of unpersisted chain data on write failure. Accessing a third distinct segment can evict dirty data; persistence then discards that data even when its write fails, while storage wrappers report success. Under the old ten-resident policy, the same three-segment workload retained failed resident writes for retry. This weakens failure containment and can impair wallet block replay, whose inspected missing-block path warns and continues.
Security review details

Security Blast Radius

  • inferred — The demonstrated failure scope is chain data in the configured local storage owner, including shared block and filter caches and their header caches. Wallets depending on missing historical data can be affected. Cross-service, tenant, credential, or environment compromise is not established by this evidence.

Security Findings and Attack Paths

  • inferred — The supported conditional failure chain is dirty data entering the eviction queue, its write failing, the queue being cleared despite that failure, and subsequent historical reads seeing missing or stale disk data. The smaller cap makes this chain reachable with fewer distinct segments. This is a worsened persistence-integrity risk, not a verified remote exploit: attacker control over the required I/O failure was not established.

Trust Boundaries and Controls

  • observed — Inspected filter writes retain header-based validation. Segment filenames derive from numeric segment identities rather than filter bytes, and block and filter wrappers serialize cache mutations with write locks. These controls do not retain evicted dirty data after a failed persistence attempt.

Resilience and Maintainability Implications

  • inferred — The five-second background save reduces the usual unsaved-data window but does not enforce persistence before a third segment is accessed. Failed resident writes remain dirty for retry, while failed evicted writes lose that protection. Storage wrappers return success after the cache logs write errors, weakening detection and recovery.

Hardening Proposals

  • proposed — Retain failed dirty evictions until a successful write and propagate persistence failures through storage interfaces so callers can retry or enter an explicit degraded state rather than observing apparent success.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the performance change in dash-spv and accurately summarizes the reduction of the segment cache to two resident segments.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Sep 29, 2026
@ZocoLini
ZocoLini merged commit ad16b13 into dev Sep 29, 2026
35 of 39 checks passed
@ZocoLini
ZocoLini deleted the perf/segment-cache-two-resident branch September 29, 2026 21:17
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.48%. Comparing base (5edeecd) to head (92e2439).
⚠️ Report is 4 commits behind head on dev.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1086      +/-   ##
==========================================
- Coverage   77.51%   77.48%   -0.04%     
==========================================
  Files         318      318              
  Lines       81270    81075     -195     
==========================================
- Hits        62997    62817     -180     
+ Misses      18273    18258      -15     
Flag Coverage Δ
core 78.90% <ø> (ø)
ffi 50.77% <ø> (ø)
rpc 20.00% <ø> (ø)
spv 92.18% <ø> (-0.01%) ⬇️
wallet 80.22% <ø> (ø)
Files with missing lines Coverage Δ
dash-spv/src/storage/blocks.rs 100.00% <ø> (+1.16%) ⬆️
dash-spv/src/storage/filters.rs 100.00% <ø> (+1.61%) ⬆️
dash-spv/src/storage/mod.rs 86.05% <ø> (+1.00%) ⬆️
dash-spv/src/storage/segments.rs 96.43% <ø> (-0.29%) ⬇️
dash-spv/src/sync/blocks/manager.rs 94.96% <ø> (-0.09%) ⬇️
dash-spv/src/sync/filters/manager.rs 97.82% <ø> (-0.01%) ⬇️

... and 4 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-bots Waiting for the review bots to report on this head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant