Skip to content

[kv] Enable prefix Bloom filters - #4929

Open
platinumhamburg wants to merge 9 commits into
apache:mainfrom
platinumhamburg:enable-prefix-bloom
Open

platinumhamburg wants to merge 9 commits into
apache:mainfrom
platinumhamburg:enable-prefix-bloom

Conversation

@platinumhamburg

@platinumhamburg platinumhamburg commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #1856

Enable cluster-wide prefix Bloom filtering so prefix lookups on compatible primary-key KV tables can skip SSTs that do not contain the requested bucket-key prefix.

Brief change log

  • Add kv.rocksdb.use-prefix-bloom-filter, defaulting to false. TabletServers apply the setting when opening KV instances; restart TabletServers to apply changes to existing instances.
  • Resolve an immutable, storage-independent prefix layout from the actual primary-key encoder. Unsupported encoders, non-leading bucket keys and historical KV state retain their existing behavior, with INFO logs describing the fallback reason and each opened tablet's effective state.
  • Keep native layout conversion, extractor installation and matching Bloom policy inside RocksDBResourceContainer.
  • Use io.github.fluss-contrib:fluss-rocksdbjni:11.8.1-fluss-4.

Tests

120 targeted server tests passed (RocksDBPrefixBloomTest, RocksDBKvTest, KvManagerTest). Checkstyle, Spotless and license checks passed. Local RPC and encoder regression suites passed; two pre-existing disabled RPC tests were skipped. Generated configuration documentation was verified against the option definition.

  • Parameterized KV manager tests cover default-off/on, supported and unsupported encoders, historical state, and switching the cluster flag across reopen with existing SST data. Assert both stored options and actual point/prefix reads.
  • Run the full TabletServiceITCase suite on a local three-TabletServer cluster with prefix Bloom enabled.
  • 42 full/partitioned parameter cases cover all 16 supported scalar type roots, including Decimal precision 18/19 and timestamp precision 3/4/9. Independently varying candidate fields are queried with Bloom disabled and enabled against overlapping SSTs; assert exact payloads and SST filtering for existing and missing prefixes. Values cover varint length boundaries, signed extremes, empty/long binary and UTF-8 data, floating-point special values, and sub-millisecond timestamp components.
  • Temporary mutation checks confirm that extracting only the first field fails all 42 cases, treating VARINT32 as one byte fails the six INT/DATE/TIME cases, and omitting timestamp nanos fails all eight non-compact timestamp cases.
  • Native full/partitioned lifecycle tests cover existing SSTs, compaction and reopen; assert both returned bytes and actual SST filtering.
  • Parameterized Bloom on/off × full/partitioned tests cover updates, deletes and mixed Memtable/SST reads before flush, after flush and after compaction.
  • Existing encoder, metadata, configuration, prefix boundary, iterator lifetime and dropped-table recovery regressions passed.

API and Format

Adds a cluster configuration option. Primary-key bytes and the Fluss KV format version are unchanged. Encoders expose generic layouts; the RocksDB resource container converts them internally without per-key Java callbacks. Existing SSTs remain readable, and new filters are built through flush and compaction.

Documentation

Document the cluster option, default, restart requirement, compatibility fallback, INFO observations, SST behavior and interaction with the existing Bloom-filter configuration in the server configuration reference.

@platinumhamburg
platinumhamburg marked this pull request as ready for review October 28, 2026 15:51
@platinumhamburg platinumhamburg changed the title [kv] Enable prefix Bloom filters for compatible primary-key tables [kv] Enable cluster-wide prefix Bloom filters Oct 29, 2026
@platinumhamburg platinumhamburg changed the title [kv] Enable cluster-wide prefix Bloom filters [kv] Enable prefix Bloom filters Oct 29, 2026
@luoyuxia
luoyuxia requested a balanced review from Copilot October 30, 2026 01:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[KV] Primary Key Tables support prefix bloom filters to enhance PrefixLookup performance

2 participants