Repository navigation
feat: View Metadata Builder (apache#908) - #2
Open
shreyas2201 wants to merge 6 commits into
Open
shreyas2201 wants to merge 6 commits into
shreyas2201 wants to merge 6 commits into
Conversation
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
Signed-off-by: Shreyas220 <shreyas.ny@gmail.com>
sriv-e6x
added a commit
that referenced
this pull request
Jul 3, 2026
… grouping, MDV guards Three fixes to the commit_v4 flush path: - #2: Add flush-bytes and flush-entries triggers alongside inline-count, so root size is bounded across all dimensions. Read target-fanout from table property instead of hardcoded 64. - #3: Group inline entries by partition_spec_id before flushing to child manifests. A single commit can carry files from different specs after partition evolution; writing them all under default_partition_spec() corrupted the child manifest's partition column. - #5: Set MDV staleness guard (entry_count + checksum) when computing positional bitmaps during replace_data_files, so stale MDVs are detected at scan time. Includes regression test for partition-spec grouping.
srinath-prabhu
pushed a commit
that referenced
this pull request
Aug 9, 2026
…at base
compact_cold_tier's commit() unconditionally rewrote the root as a flat
base (chain_depth=0, prev_root_path=None), re-serializing the entire live
entry set every time. On sri-olly's `logs` table (12,896 hot-root entries,
chain up to 64), that added 5-8s of root-write cost on top of the ~15-20s
of reads inside the CAS window — every attempt exceeded laminar's ~15-30s
hot-append cadence, so 6b lost every OCC race and tessellate ticks
DeadlineExceeded consecutively for hours.
On an incremental table (root-manifest.incremental=true — sri-olly's
tiered tables set this by default via tiered_table_creation), the swap
now emits a small delta root:
prev_root_path = Some(current)
chain_depth += 1
entries = &[] // metadata-only write (~KB parquet)
removed_paths = Vec::new() // materialized by rewritten leaves;
// ancestor tombstones live on prior
// roots and are unioned by
// reconstruct_root
The bucket_index_path (the actual cold-pointer swap) still lands in the
new metadata — reconstruct_root treats HEAD meta as authoritative, so the
new cold state is visible to readers as soon as the delta commits.
At chain_depth == MAX_CHAIN (64, same cap as commit_v4 at
snapshot.rs:1349), do_delta falls through to the flat-base branch —
identical collapse semantics commit_v4 already applies to hot appends.
On non-incremental tables the code path is unchanged.
Fix #1 of 2 for the tessellate DeadlineExceeded regression. Fix #2 (split
prep from CAS so retries don't re-do the 15-20s serial leaf-load and
chain-walk) is next — this commit is a prerequisite because fix #2 relies
on the delta payload shape being cheap to re-emit per retry.
Tests:
- test_v4_compact_cold_tier_swaps_small_for_merged: existing e2e extended
to assert chain_depth=1 + prev_root_path=Some + empty entries after the
cold-compact commit (previously implicitly asserted flat-base shape).
- test_v4_compact_cold_tier_collapses_at_chain_cap: new e2e that pushes
chain to MAX_CHAIN via fast_appends and verifies compact_cold_tier
falls back to flat-base (chain_depth=0, prev_root_path=None) as the
amortization contract requires.
All 17 v4_commit tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.