Skip to content

Use primitive IDs directly in Corgi reduce and join - #884

Merged
frankmcsherry merged 3 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-primitive-pools
Sep 15, 2026
Merged

frankmcsherry merged 3 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-primitive-pools

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Corgi already uses primitive payloads as key/value IDs, but reduce still stores and looks up those payloads, and join still selects coordinates for primitive values. Use the primitive itself wherever it completely represents the data, and reconstruct primitive columns directly from matched IDs.

Three independently reviewable commits:

  1. Share reduce's three resolvers in one IdPool. Primitive pools retain singleton-product nesting depth; compound pools retain representative maps. Count/Distinct stop resolving input rows they ignore, and the input map retains capacity between retirements.
  2. Compute primitive Min by scanning signed IDs per bracket, avoiding candidate-column construction and structural sorting.
  3. With primitive join keys, use primitive value IDs independently on either side. Read the original value slice, advance and consolidate entries, and reconstruct primitive key/value columns on output. Compound values retain coordinates; structured keys retain the existing coordinate path, including collision validation.

Only interactive/src/corgi/{reduce,join}.rs change. No proxy or timestamp API changes and no new join payload pools.

Scope Production additions / deletions Net production Added tests
Reduce, first two commits +101 / −110 −9 38
Join, third commit +58 / −18 +40 74
Combined diff +159 / −128 +31 112

Full diff: +271/−128 across two files.

Validation

CI passes Clippy and the test matrix on Ubuntu and macOS with stable Rust and Rust 1.86.

The combined head passes cargo test --locked --offline -p differential-dataflow -p interactive: 200 regular tests and 54 doc tests (8 regular and 6 doc tests ignored). Coverage includes nested singleton products, signed extremes, zero/negative support, compound fallback, either primitive join side, both probe orientations, cross-chunk and complete-key cancellation, product-time reordering, and existing hash-collision checks. Focused join outputs also match the Vec backend.

Reduce measurements: first two commits versus #883

These measurements stop at 96b4a78; they do not include the third commit. Baseline is merged #883 (51c4dd0). Server MiMalloc allocator:

Workload Startup before → after Updates before → after Peak RSS before → after
Reachability 0.415 → 0.397 s 4.117 → 4.013 s 470 → 469 MiB
Stable matching, compound values 0.944 → 0.856 s 1.310 → 1.299 s 624 → 596 MiB
ADT/count 0.155 → 0.136 s 2.213 → 1.152 s 384 → 377 MiB
Signed integer Min 0.281 → 0.210 s 0.117 → 0.100 s 511 → 476 MiB
Compound Min 0.644 → 0.638 s 0.154 → 0.158 s 1079 → 1115 MiB
Collect 0.299 → 0.270 s 0.256 → 0.247 s 768 → 739 MiB

500k nodes/1M input records, 50 rounds of 1,000 replacements. With the system allocator, SCC at 500k nodes/1M edges improves from 4.717 to 4.569 s startup and 5.101 to 4.843 s for ten updates. The first two commits also retain gains above a locally rebased #870: 4.4% startup and 5.6% updates for SCC. That overlap measurement excludes the new join commit. Most benefit comes from the pools; the Min scan adds about 5% startup improvement to the isolated large integer-Min case with the system allocator.

Join measurements: third commit versus the first two

Measured separately against 96b4a78, rather than adding percentages to the preceding results. Longer self-join runs with actual output values:

Case System-allocator update reduction MiMalloc update reduction
Primitive values, 100k records 5.6% (0.6302 → 0.5949 s) 5.6% (0.5480 → 0.5171 s)
Primitive values, 200k records 5.7% 4.5%
Primitive values, dense 20k records 4.3% 5.3%
One compound value side, 100k records 2.8% 3.0%
Both compound value sides / structured keys Approximately flat Approximately flat

100k-record cases: 50k nodes, 100 × 1,000 replacements. 200k-record case: 100k nodes, 50 × 2,000 replacements. Dense case: 1k nodes, 50 × 200 replacements. Startup is mostly flat with the system allocator and improves modestly for primitives with MiMalloc. In the SCC system-allocator harness at 1M edges, the incremental startup change is 4.5378 → 4.4411 s (2.1%); updates are essentially flat (4.8185 → 4.7892 s). Small reachability updates are about 2% slower.

All comparisons use Apple M4, one worker, medians of three alternating repetitions, release/LTO, Rust 1.95, Timely 4eb3ee09681c209ebcaeced73548a9fbf8e1c71e, and MiMalloc 0.1.52 where indicated. Installation, input generation and snapshot hashing are outside the feed/tick timer. Checksums agree; no pageout/swap counter changes were observed.

Tradeoffs

Memory results are mixed. In the reduce measurements, compound-Min updates are about 2.8% slower with MiMalloc, and system-allocator ADT peak RSS rises from 792 to 991 MiB despite the speedup; the latter does not carry over to MiMalloc. In the join measurements, primitive 200k-record peak RSS rises about 5% with the system allocator but stays flat with MiMalloc; primitive 100k-record RSS rises about 8% with MiMalloc. These are whole-process peaks including snapshots; their causes remain unassigned. No allocator-specific workaround is included.

#882 remains separate.

@frankmcsherry frankmcsherry changed the title Use primitive IDs directly in Corgi reduce Use primitive IDs directly in Corgi reduce and join Sep 15, 2026
@frankmcsherry
frankmcsherry marked this pull request as ready for review September 15, 2026 15:53
@frankmcsherry
frankmcsherry merged commit 1252197 into TimelyDataflow:master-next Sep 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant