Repository navigation
Use primitive IDs directly in Corgi reduce and join - #884
Merged
frankmcsherry merged 3 commits intoSep 15, 2026
Merged
frankmcsherry merged 3 commits into
frankmcsherry merged 3 commits into
Conversation
frankmcsherry
marked this pull request as ready for review
September 15, 2026 15:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corgi already uses primitive payloads as key/value IDs, but reduce still stores and looks up those payloads, and join still selects coordinates for primitive values. Use the primitive itself wherever it completely represents the data, and reconstruct primitive columns directly from matched IDs.
Three independently reviewable commits:
IdPool. Primitive pools retain singleton-product nesting depth; compound pools retain representative maps. Count/Distinct stop resolving input rows they ignore, and the input map retains capacity between retirements.Only
interactive/src/corgi/{reduce,join}.rschange. No proxy or timestamp API changes and no new join payload pools.Full diff: +271/−128 across two files.
Validation
CI passes Clippy and the test matrix on Ubuntu and macOS with stable Rust and Rust 1.86.
The combined head passes
cargo test --locked --offline -p differential-dataflow -p interactive: 200 regular tests and 54 doc tests (8 regular and 6 doc tests ignored). Coverage includes nested singleton products, signed extremes, zero/negative support, compound fallback, either primitive join side, both probe orientations, cross-chunk and complete-key cancellation, product-time reordering, and existing hash-collision checks. Focused join outputs also match the Vec backend.Reduce measurements: first two commits versus #883
These measurements stop at
96b4a78; they do not include the third commit. Baseline is merged #883 (51c4dd0). Server MiMalloc allocator:500k nodes/1M input records, 50 rounds of 1,000 replacements. With the system allocator, SCC at 500k nodes/1M edges improves from 4.717 to 4.569 s startup and 5.101 to 4.843 s for ten updates. The first two commits also retain gains above a locally rebased #870: 4.4% startup and 5.6% updates for SCC. That overlap measurement excludes the new join commit. Most benefit comes from the pools; the Min scan adds about 5% startup improvement to the isolated large integer-Min case with the system allocator.
Join measurements: third commit versus the first two
Measured separately against
96b4a78, rather than adding percentages to the preceding results. Longer self-join runs with actual output values:100k-record cases: 50k nodes, 100 × 1,000 replacements. 200k-record case: 100k nodes, 50 × 2,000 replacements. Dense case: 1k nodes, 50 × 200 replacements. Startup is mostly flat with the system allocator and improves modestly for primitives with MiMalloc. In the SCC system-allocator harness at 1M edges, the incremental startup change is 4.5378 → 4.4411 s (2.1%); updates are essentially flat (4.8185 → 4.7892 s). Small reachability updates are about 2% slower.
All comparisons use Apple M4, one worker, medians of three alternating repetitions, release/LTO, Rust 1.95, Timely
4eb3ee09681c209ebcaeced73548a9fbf8e1c71e, and MiMalloc 0.1.52 where indicated. Installation, input generation and snapshot hashing are outside the feed/tick timer. Checksums agree; no pageout/swap counter changes were observed.Tradeoffs
Memory results are mixed. In the reduce measurements, compound-Min updates are about 2.8% slower with MiMalloc, and system-allocator ADT peak RSS rises from 792 to 991 MiB despite the speedup; the latter does not carry over to MiMalloc. In the join measurements, primitive 200k-record peak RSS rises about 5% with the system allocator but stays flat with MiMalloc; primitive 100k-record RSS rises about 8% with MiMalloc. These are whole-process peaks including snapshots; their causes remain unassigned. No allocator-specific workaround is included.
#882 remains separate.