Skip to content

build: point iceberg-rust fork at head carrying column stats on FileScanTask - #7

Draft
Perseus wants to merge 2 commits into
iceberg-dyn-filter/d4-task-open-pruningfrom
iceberg-dyn-filter/f2-fork-column-stats
Draft

Perseus wants to merge 2 commits into
iceberg-dyn-filter/d4-task-open-pruningfrom
iceberg-dyn-filter/f2-fork-column-stats

Conversation

@Perseus

@Perseus Perseus commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Builds on #6; review that first.

What and why

Upcoming changes need per-file column min/max on FileScanTask so whole files can be skipped. Upstream apache/iceberg-rust#3424 adds that (TableScanBuilder::with_column_stats(names) and FileScanTask::column_stats()), but it is not merged. This PR moves the root [patch] to a new fork head that carries it. There are no Rust changes in this repo, and nothing calls the new API yet. By default, tasks carry no stats unless with_column_stats is called.

Fork change ledger

All fork changes are tracked in DFD_FORK.md on branch dfd/runtime-predicate-5b82ab9 of Perseus/iceberg-rust. It is fork-only bookkeeping and is never to be upstreamed. For each change it records the fork and upstream SHAs, upstream status, backport notes, why DFD needs it, who uses it, and when to drop it. These changes were added on top as a fast-forward (cbfab28d..bd8a39b5). The branch was later re-made once, only to change the commit identity (see the History section in DFD_FORK.md), so SHAs from before that point are no longer on the branch.

Base is apache rev 5b82ab99, the rev datafusion-iceberg pins, so FileScanTask stays one shared type. The commits are:

The fork branch was later re-made once with a different commit identity, so its commit SHAs changed; the contents are unchanged, and the new head 1ddce488 (pinned here) only adds the old-to-new SHA table to DFD_FORK.md.

Backport conflicts in scan/mod.rs and how they were resolved

  • require_field_id is used without ?, because resolve_field_id returns Option on this base. Names resolve exactly as select does here. One consequence: an ambiguous case-insensitive name picks a field instead of failing, since the ambiguity check from #3070 is not on this base. The two #3070 ambiguity tests are not carried.
  • column_stats_names sits next to this base's snapshot_id (upstream now has snapshot_selection, #3260).
  • with_column_stats is added unchanged.
  • Test imports adjusted.
  • Datum::promote_to (#3483) is inlined as the same int→long and float→double widening.
  • Tests use Type::Primitive(..), the NestedField signature before #3527, and SchemaRef is imported in test_utils.

Consequences

  • Downstream users of this crate must carry the [patch."https://github.com/apache/iceberg-rust"] section.
  • The fork branch must be rebuilt when datafusion-iceberg's pin moves. The rule is in DFD_FORK.md.
  • Each carried change is dropped once the pin includes its upstream merge commit.

Verification

  • Fork: cargo check -p iceberg --tests is clean. cargo test -p iceberg --lib -- scan:: arrow::reader gives 237 passed, the 7 column_stats tests pass, and the with_column_stats doctest passes.
  • This repo: Cargo.lock changes only the three source lines. cargo tree -d shows a single iceberg crate, from the fork at 1ddce488. cargo fmt --all is clean, as is clippy on datafusion-distributed-iceberg --all-targets. cargo test -p datafusion-distributed-iceberg passes, cargo test --features integration --test dynamic_filtering gives 18 passed, and cargo check --workspace --all-targets --locked passes.

@Perseus
Perseus added this pull request to stack #3 October 30, 2026 21:36
Perseus added a commit to Perseus/iceberg-rust that referenced this pull request Oct 30, 2026
… ledger

with_column_stats' doc promised rejection of ambiguous case-insensitive
names, which needs apache#3070 (not in this base). It now
says names resolve as select does. DFD_FORK.md links the DFD PR as the
user of apache#3424 and explains the branch links in earlier Needed-by
trailers.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3
Needed-by: Perseus/datafusion-distributed#7
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Perseus added a commit to Perseus/iceberg-rust that referenced this pull request Oct 31, 2026
… ledger

with_column_stats' doc promised rejection of ambiguous case-insensitive
names, which needs apache#3070 (not in this base). It now
says names resolve as select does. DFD_FORK.md links the DFD PR as the
user of apache#3424 and explains the branch links in earlier Needed-by
trailers.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3
Needed-by: Perseus/datafusion-distributed#7
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Perseus added a commit to Perseus/iceberg-rust that referenced this pull request Oct 31, 2026
The fork maintainer's commits on this branch were re-made with the
Perseus noreply identity. Trees and messages are unchanged. Update the
ledger's fork SHAs and add a History section with the old -> new table.

Upstream-PR: none
Upstream-commit: none
Backport-notes: none
Needed-by: Perseus/datafusion-distributed#5, Perseus/datafusion-distributed#7
Remove-when: never (fork bookkeeping)
…canTask

Bumps the [patch] rev for iceberg, iceberg-property-macro and
iceberg-storage-opendal to Perseus/iceberg-rust@dacdc376, which adds a
backport of apache/iceberg-rust#3424 and the DFD_FORK.md change ledger.
No Rust changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant