feat(scan): optionally retain column statistics on FileScanTask by anuragmantri · Pull Request #3424 · apache/iceberg-rust

anuragmantri · 2026-10-12T22:55:36Z

Which issue does this PR close?

What changes are included in this PR?

FileScanTask drops a data file's column statistics today. Engines that order or merge files by key range, such as apache/datafusion-iceberg#55 and apache/datafusion-comet#5343, need per-file bounds on the sort columns.

TableScanBuilder::with_column_stats(columns) keeps the file's column sizes, value, null and NaN counts, and lower and upper bounds for the named columns, in a new FileScanTaskColumnStats on FileScanTask::column_stats(). It mirrors Java's Scan.includeColumnStats. The count and bound maps match FileMetrics in #3347.

It is opt-in, following #607. By default column_stats() is None, nothing is copied from the manifest entry, and the field is not serialized, so existing serialized tasks are unchanged. Bounds are promoted to the column's type in the scan schema, because manifests written before a type promotion keep the narrower type. Bounds serialize as their type plus the spec's binary encoding, because the JSON encoding of single values does not round-trip every float (#3409).

Are these changes tested?

Yes: retention and filtering by requested column, case-insensitive names, rejected names, bound promotion across manifests, and serde round trips including infinite and inexact float bounds.

AI Disclosure

I used Claude Code (Claude Opus 5.5) to write this change and its tests, and I reviewed it manually. Reviewers may want to focus on three choices. Nested primitive names are accepted, unlike select. An empty column list gives each task empty statistics, as select_empty does. The serialized form of FileScanTaskColumnStats becomes public API.

Add TableScanBuilder::with_column_stats, mirroring Java's Scan.includeColumnStats. The named columns resolve to field ids at build time with the scan's case sensitivity, and each planned FileScanTask carries a FileScanTaskColumnStats holding the data file's value, null and NaN counts and lower and upper bounds for only those field ids. Retention is opt-in: by default column_stats is None, nothing is copied from the manifest entry, and the field is omitted when a task is serialized, so serialized tasks are unchanged.

JeonDaehong

Thanks, this will be useful for #3343.

One thing I hit while round-tripping the new stats through serde: a float/double bound of ±Infinity (or NaN) serializes to "literal":null with serde_json, and deserializing it fails with None value at line 1 .... Every other primitive type round-trips (bool through decimal, including fixed, uuid and timestamp_ns).

±Infinity is a valid bound: the spec only excludes NaN, and Java's FloatFieldMetrics keeps infinite values as bounds. So a task for a file containing an infinite float would serialize but not deserialize. The root cause is Datum's serde rather than this PR, but this PR is what puts file bounds into serialized tasks. Could we add an infinite-bound case to the serde test and either fix it in Datum or document it?

Bounds went through Datum's JSON encoding, which writes non-finite floats as null and rejects many finite float values when reading them back, so a task with such a bound serialized but failed to deserialize. Serialize each bound as its type and the spec's single-value binary encoding, as the DataFile JSON serde already does, so every bound round-trips exactly.

comphead · 2026-10-13T20:30:32Z

    partition_spec: Option<PartitionSpecRef>,
    unified_partition_type: Option<Arc<StructType>>,
    sort_orders: Arc<HashMap<i64, SortOrderRef>>,
+    column_stats_field_ids: Option<Arc<Vec<i32>>>,


should ids be u32?

I kept i32 to match how field ids are typed everywhere else: the spec defines them as int, and NestedField::id, the DataFile stats maps this reads from, FileScanTask::project_field_ids and FileMetrics in #3347 all use i32. Switching here would mean converting at each of those boundaries.

comphead · 2026-10-13T20:32:50Z


+    /// One column's statistics: field id, value count, null count, NaN count, lower bound
+    /// and upper bound.
+    type ColumnStat = (i32, u64, u64, Option<u64>, Datum, Datum);


I'd prob prefer a typed structure instead of tuple type.... unless tuple type is cheaper than struct

nvm, its for test

Right, it's only a test helper, so each column's expected stats fit on one line. Happy to switch to a small struct if that reads better.

comphead · 2026-10-13T20:34:33Z

    }
 }

+/// Column statistics of a [`FileScanTask`]'s data file, from its manifest entry.


I feel the comments can be concised here

Trimmed in 5345a45 to what a caller needs: what a missing entry means, that bounds stay inclusive when truncated and are typed with the manifest's schema, and that the statistics cover the whole file.

comphead

Overall the opt-in design looks sound and matches Java's includeColumnStats for the maps it copies. I found no correctness issues. A few small points below.

Question: Java's BaseFile(copy, requestedColumnIds) also keeps column_sizes for the requested ids. Is leaving it out here intentional, for example because FileMetrics in #3347 does not carry it? If so, a sentence in the FileScanTaskColumnStats docs would help.

comphead · 2026-10-13T20:38:10Z

+
+impl FileScanTaskColumnStats {
+    pub(crate) fn from_data_file(data_file: &DataFile, field_ids: &[i32]) -> Self {
+        fn retain<V: Clone>(map: &HashMap<i32, V>, field_ids: &[i32]) -> HashMap<i32, V> {


This runs once per manifest entry and builds five maps. collect() over filter_map has a lower size hint of 0, so each map regrows. Consider HashMap::with_capacity(field_ids.len().min(map.len())) and returning early when map is empty (metrics mode none is common), so the empty case does no work.

Done in 5345a45. An empty map now returns before any lookups, and the result is sized with field_ids.len().min(map.len()).

comphead · 2026-10-13T20:38:10Z

        .collect()
 }

+fn collect_column_stats_field_ids(


The resolve plus Column {column_name} not found in table. Schema: {schema} construction duplicates collect_scan_field_ids. A small field_id_by_name(schema, name, case_sensitive) -> Result<i32> used by both would keep the two error messages from drifting. Related: column_names is an Option<Vec<String>> and this is a Vec<String> where empty means none. An Option would follow the existing field and drop the is_empty special case.

Done in 5345a45. Both paths now share a require_field_id helper and its not-found error. I named it differently so it isn't confused with Schema::field_id_by_name. The requested names are now an Option<Vec<String>> like column_names, so an empty list means no columns and each task carries empty statistics, the same way select_empty works.

comphead · 2026-10-13T20:38:10Z

    partition_spec: Option<PartitionSpecRef>,
    unified_partition_type: Option<Arc<StructType>>,
    sort_orders: Arc<HashMap<i64, SortOrderRef>>,
+    column_stats_field_ids: Option<Arc<Vec<i32>>>,


Option<Arc<Vec<i32>>> is a double indirection. Option<Arc<[i32]>> reads the ids with one pointer hop and is built with Arc::from(vec). Same for the field on ManifestEntryContext and PlanContext.

Switched to Option<Arc<[i32]>> in all three contexts in 5345a45.

comphead · 2026-10-13T20:38:10Z

+    }
+
+    #[tokio::test]
+    async fn test_plan_files_retains_requested_column_stats() {


The description calls out nested names as a deliberate difference from select, but no case here uses one. A nested leaf (for example a struct child, resolved by field_id_by_name) would pin that choice.

Added test_collect_column_stats_field_ids_resolves_nested_field in 5345a45, which resolves s.f to the struct child's field id.

anuragmantri

@JeonDaehong thanks for catching this. I reproduced it, and it's a little wider than infinite bounds: Datum's JSON path also rejects many finite float values, such as Datum::float(100.1), because the float conversion compares against an absolute f32::EPSILON. That's the same root cause as #3409.

In e0f3919, bounds now serialize as their type plus the spec's single-value binary encoding, as the DataFile JSON serde already does, so every bound round-trips exactly. The serde test now covers a float of 100.1 and infinite float and double bounds. Partition values and predicate literals still go through the JSON path, so I've left those to #3409.

anuragmantri · 2026-10-13T20:34:34Z

    partition_spec: Option<PartitionSpecRef>,
    unified_partition_type: Option<Arc<StructType>>,
    sort_orders: Arc<HashMap<i64, SortOrderRef>>,
+    column_stats_field_ids: Option<Arc<Vec<i32>>>,


I kept i32 to match how field ids are typed everywhere else: the spec defines them as int, and NestedField::id, the DataFile stats maps this reads from, FileScanTask::project_field_ids and FileMetrics in #3347 all use i32. Switching here would mean converting at each of those boundaries.

anuragmantri · 2026-10-13T20:34:35Z


+    /// One column's statistics: field id, value count, null count, NaN count, lower bound
+    /// and upper bound.
+    type ColumnStat = (i32, u64, u64, Option<u64>, Datum, Datum);


Right, it's only a test helper, so each column's expected stats fit on one line. Happy to switch to a small struct if that reads better.

Retain column_sizes for the requested ids too, matching Java's BaseFile(copy, requestedColumnIds). Share the column-name lookup and its error with column selection, keep the requested names as an Option like column_names so an empty list yields empty statistics, and pass the ids as Arc<[i32]>. Skip work for an empty map when filtering, trim the struct docs, and cover nested field names.

anuragmantri

@comphead thanks for the review. Leaving out column_sizes wasn't intentional. I had limited the maps to what FileMetrics reads, but there's no reason to differ from Java, so 5345a45 also keeps column_sizes for the requested ids. The other points are addressed inline.

anuragmantri · 2026-10-13T20:49:52Z

    }
 }

+/// Column statistics of a [`FileScanTask`]'s data file, from its manifest entry.


Trimmed in 5345a45 to what a caller needs: what a missing entry means, that bounds stay inclusive when truncated and are typed with the manifest's schema, and that the statistics cover the whole file.

anuragmantri · 2026-10-13T20:49:53Z

+
+impl FileScanTaskColumnStats {
+    pub(crate) fn from_data_file(data_file: &DataFile, field_ids: &[i32]) -> Self {
+        fn retain<V: Clone>(map: &HashMap<i32, V>, field_ids: &[i32]) -> HashMap<i32, V> {


Done in 5345a45. An empty map now returns before any lookups, and the result is sized with field_ids.len().min(map.len()).

anuragmantri · 2026-10-13T20:49:54Z

        .collect()
 }

+fn collect_column_stats_field_ids(


Done in 5345a45. Both paths now share a require_field_id helper and its not-found error. I named it differently so it isn't confused with Schema::field_id_by_name. The requested names are now an Option<Vec<String>> like column_names, so an empty list means no columns and each task carries empty statistics, the same way select_empty works.

anuragmantri · 2026-10-13T20:49:56Z

    partition_spec: Option<PartitionSpecRef>,
    unified_partition_type: Option<Arc<StructType>>,
    sort_orders: Arc<HashMap<i64, SortOrderRef>>,
+    column_stats_field_ids: Option<Arc<Vec<i32>>>,


Switched to Option<Arc<[i32]>> in all three contexts in 5345a45.

anuragmantri · 2026-10-13T20:49:57Z

+    }
+
+    #[tokio::test]
+    async fn test_plan_files_retains_requested_column_stats() {


Added test_collect_column_stats_field_ids_resolves_nested_field in 5345a45, which resolves s.f to the struct child's field id.

JeonDaehong · 2026-10-14T10:45:40Z

Thanks @anuragmantri! Storing bounds as the spec's binary encoding is a nice fix, since it now matches the DataFile serde. Good catch on finite floats like 100.1 too. Leaving partition values and literals to #3409 sounds right to me.

JeonDaehong

LGTM. Read through the full diff: opt-in default keeps existing serialization unchanged, stats are filtered to the requested ids, and the binary bound encoding round-trips. Thanks @anuragmantri!

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/scan/mod.rs

JeonDaehong

A second pass after my approval, none of these are blocking. One thing I checked that works better than Java: Java's includeColumnStats(columns) resolves names with schema.findField, so it is always case-sensitive and throws a NullPointerException on an unknown name. Here names follow the scan's case sensitivity and an unknown name fails build() with a clear error.

JeonDaehong · 2026-10-16T06:41:03Z

+    ///
+    /// [`build`](Self::build) fails if a name matches no field, or more than one field when the
+    /// scan is case-insensitive.
+    pub fn with_column_stats(


Java has a no-argument includeColumnStats() that keeps the stats of every column (BaseFile copies all maps when requestedColumnIds is null). Here the only way to get all of them is to list every leaf name. Is that intended? If engines need it, a with_all_column_stats() could come in a follow-up.

Yes, that's intended for now. Naming the columns keeps the default cheap, which was the concern about task size in #607, and the engines I have in mind only need the sort columns. If one needs every column, a with_all_column_stats() is a small follow-up.

JeonDaehong · 2026-10-16T06:41:03Z

+    /// [`FileScanTask`], exposed as [`FileScanTask::column_stats`].
+    ///
+    /// Names resolve against the scan's schema with the scan's case sensitivity and may name
+    /// nested fields. By default, tasks carry no statistics.


Naming a struct (or list/map) column resolves to the parent's field id, which never has stats, so with_column_stats(["s"]) gives empty maps without an error. I checked: s: struct<f> resolves to [1] and the stats come back empty while f (id 2) has them. Java behaves the same, so I'd either reject non-primitive fields at build() or say here that only leaf names return statistics.

Good catch. In fe1bf90, build() rejects a struct, list or map name with Column s is not a primitive field and has no column statistics, and the doc now says names must be primitive fields. test_collect_column_stats_field_ids_rejects_struct_field covers it.

JeonDaehong · 2026-10-16T06:41:03Z

        .collect()
 }

+fn collect_column_stats_field_ids(


nit, optional: select accepts metadata columns like _file, but here they fail with Column _file not found in table, which reads like a typo in the name. A message saying metadata columns have no statistics would be clearer.

Agreed. In fe1bf90, a metadata column now fails with Metadata column _file has no column statistics, and the planning test asserts that message.

JeonDaehong · 2026-10-16T06:41:03Z

+///
+/// A requested column missing from a map has unknown statistics of that kind, for example when
+/// the table's metrics mode records no bounds. Bounds are inclusive even when truncated, and are
+/// typed with the schema the manifest was written with, so after type promotion an `int` bound


This promises that after type promotion an int bound can describe a long column, which is a useful thing for callers to know, but no test pins it. A manifest written with an int column, read after promoting it to long, asserting the bound is still Datum::int, would keep it true.

Added test_plan_files_keeps_bounds_typed_with_manifest_schema in fe1bf90. It writes the manifest with y as an int while the table schema says long, and checks the bounds come back as Datum::int. This holds because the manifest reader parses bounds with the schema stored in the manifest.

This promises that after type promotion an int bound can describe a long column, which is a useful thing for callers to know, but no test pins it.

Should planning convert the retained bounds to the scan schema's type instead, so that one scan returns one type per field id?

As I read it, a bound's type depends on which manifest holds the entry, not only on when the data file was written. A file listed in a manifest written before y was promoted gets Datum::int. A file listed in a manifest written after the promotion gets Datum::long. Java's merge append writes old entries into such manifests: ManifestMergeManager copies them with writer.existing(entry) into a writer whose spec is rebound to the current schema, and the bound bytes stay 4 bytes. The type changes on read because Datum::try_from_bytes widens 4-byte values when the manifest schema says long. So one scan can hand out both types for the same column. Datum's PartialOrd has no cross-type arms, so comparing them returns None. Ordering files by their bounds is the comparison a sorted scan makes.

I checked this on fe1bf90 with a scratch test that writes two manifests, one with y: int and one with y: long, and scans with with_column_stats(["y"]). The lower bounds come back as Datum { type: Int, literal: Int(5) } and Datum { type: Long, literal: Long(6) }, and partial_cmp between them returns None.

The spec's promotion table describes decoding a bound against the current type. Datum::to already converts int to long and float to double, and decimals of different precision compare correctly today. date to timestamp (v3) would need its own conversion, or the bound could be dropped, since a missing bound already means unknown. If keeping the manifest's type is intended, what do you think about adding the two-manifest case to test_plan_files_keeps_bounds_typed_with_manifest_schema and saying in the docs that callers must convert bounds before comparing them across tasks?

Agreed, and thanks for the trace through ManifestMergeManager. Planning now converts bounds to the column's type in the scan schema. The two-manifest case is a test.

A decimal is converted only to a wider precision at the same scale. Any other bound that differs in type is dropped, so it reads as unknown.

date to timestamp never gets this far. Datum::try_from_bytes fails on the 4-byte bound, so the manifest read errors before planning. That seems like a separate issue?

Struct, list and map fields have no statistics of their own, and metadata columns are not stored in data files, so requesting stats for them silently returned empty maps or failed with a misleading "not found" error. Fail build() with an error that says why. Also cover bounds that keep the manifest's type after a type promotion.

JeonDaehong · 2026-10-17T02:09:17Z

Thanks @anuragmantri, all four look good to me. I checked fe1bf90 and the scan tests pass. A with_all_column_stats() follow-up sounds fine.

JeonDaehong

Checked fe1bf90, all four look good. Thanks for adding the tests.

alexanderbianchi · 2026-10-23T22:05:06Z

datafusion-contrib/datafusion-distributed#789 Thanks for this work, I have a PR that'll use it. Will also apply to datafusion-iceberg.

alexanderbianchi · 2026-10-27T20:03:28Z

@comphead is this PR in a state you are confident with approving after the iteration with @JeonDaehong? It's relevant for some datafusion work.

mbutrovich

Thanks @anuragmantri, left a couple of anchored comments and one response to #3424 (comment).

mbutrovich · 2026-10-27T22:50:28Z

+    /// Retains the data file's column statistics for the named columns on each planned
+    /// [`FileScanTask`], exposed as [`FileScanTask::column_stats`].
+    ///
+    /// Names resolve against the scan's schema with the scan's case sensitivity and must name
+    /// primitive fields, which may be nested. By default, tasks carry no statistics.
+    ///
+    /// [`build`](Self::build) fails if a name matches no field, matches more than one field when
+    /// the scan is case-insensitive, or names a struct, list, map or metadata column.
+    pub fn with_column_stats(


Could with_column_stats get a doctest, like as_of_time on main? It's the entry point engines will copy, and an example that reads a bound from task.column_stats() would show the field-id keying and the None default in one place.

Added. It is compile-only, because there is no table with data files to run it against.

mbutrovich · 2026-10-27T22:52:05Z

+    /// [`TableScanBuilder::with_column_stats`](crate::scan::TableScanBuilder::with_column_stats).
+    /// `None` if no statistics were requested.
+    #[builder(default)]
+    column_stats: Option<FileScanTaskColumnStats>,


Could this be Option<Box<FileScanTaskColumnStats>>? Nothing is allocated when the field is None, but the field reserves room for six HashMaps inline, 288 bytes. So FileScanTask grows from 352 to 640 bytes on every scan, including scans that never call with_column_stats. I measured that with size_of on ba849a6 (the merge base) and fe1bf90. #3422 says the feature "should add nothing when it isn't used".

Boxing keeps the field at 8 bytes. The getter can return self.column_stats.as_deref(), and a transform on the builder setter keeps its signature, so the public API stays the same:

Suggested change

column_stats: Option<FileScanTaskColumnStats>,

#[builder(default, setter(transform = |stats: Option<FileScanTaskColumnStats>| stats.map(Box::new)))]

column_stats: Option<Box<FileScanTaskColumnStats>>,

The serializer at line 574 would change to value.column_stats.as_deref() too. The cost is one more allocation per task when stats are requested, next to the six maps that path already builds.

Done as you suggested. The setter and getter signatures are unchanged.

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/scan/mod.rs # crates/iceberg/src/test_utils/scan.rs

A manifest written before a type promotion keeps the narrower type, so one scan could return an int bound for one file and a long bound for another, and Datum cannot order values of different types. Promote bounds to the column's type in the scan schema (int to long, float to double, decimal to a wider precision) and omit a bound that cannot be promoted. Box the retained statistics so tasks of scans that never request them do not grow, and add a doctest for with_column_stats.

Resolve each requested column's type when the scan is built, so planning no longer looks the column up in the schema for every bound of every manifest entry. Rebuild the column stats tests on the builder, one behavior per test, and cover list names. Drop private field docs that repeated the getter docs, and state the decimal and date-to-timestamp promotion rules in the struct docs.

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/test_utils/scan.rs

A date bound under a timestamp column is stored in 4 bytes, which Datum::try_from_bytes cannot decode as a timestamp, so the manifest read fails before planning sees the bound. The docs described a case that planning never handles.

anuragmantri

Thanks @mbutrovich. I updated the PR with your suggestions.

anuragmantri · 2026-10-27T23:37:05Z

+///
+/// A requested column missing from a map has unknown statistics of that kind, for example when
+/// the table's metrics mode records no bounds. Bounds are inclusive even when truncated, and are
+/// typed with the schema the manifest was written with, so after type promotion an `int` bound


Agreed, and thanks for the trace through ManifestMergeManager. Planning now converts bounds to the column's type in the scan schema. The two-manifest case is a test.

A decimal is converted only to a wider precision at the same scale. Any other bound that differs in type is dropped, so it reads as unknown.

date to timestamp never gets this far. Datum::try_from_bytes fails on the 4-byte bound, so the manifest read errors before planning. That seems like a separate issue?

anuragmantri · 2026-10-27T23:37:07Z

+    /// Retains the data file's column statistics for the named columns on each planned
+    /// [`FileScanTask`], exposed as [`FileScanTask::column_stats`].
+    ///
+    /// Names resolve against the scan's schema with the scan's case sensitivity and must name
+    /// primitive fields, which may be nested. By default, tasks carry no statistics.
+    ///
+    /// [`build`](Self::build) fails if a name matches no field, matches more than one field when
+    /// the scan is case-insensitive, or names a struct, list, map or metadata column.
+    pub fn with_column_stats(


Added. It is compile-only, because there is no table with data files to run it against.

anuragmantri · 2026-10-27T23:37:08Z

+    /// [`TableScanBuilder::with_column_stats`](crate::scan::TableScanBuilder::with_column_stats).
+    /// `None` if no statistics were requested.
+    #[builder(default)]
+    column_stats: Option<FileScanTaskColumnStats>,


Done as you suggested. The setter and getter signatures are unchanged.

Backport of apache#3424 (net diff of PR head 86e35d3 against its merge base beba39c), adapted to base 5b82ab9. Adds TableScanBuilder::with_column_stats(names) and FileScanTask::column_stats(). Tasks carry the data file's column sizes, value/null/NaN counts and lower/upper bounds only for the named primitive columns. By default no statistics are kept and behaviour is unchanged. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: scan/mod.rs conflicts resolved for the older base: resolve_field_id keeps its Option return (no apache#3070 ambiguity check), TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is inlined as the same int->long / float->double widening in task.rs. Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and test_utils imports SchemaRef. Details in DFD_FORK.md. Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit Issue: apache#3422

Adds the ledger entry for 9832f9c: upstream links and head SHA, each scan/mod.rs conflict resolution, the base-rev adaptations, why DFD needs it and when to drop it. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: none (docs only) Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit

… ledger with_column_stats' doc promised rejection of ambiguous case-insensitive names, which needs apache#3070 (not in this base). It now says names resolve as select does. DFD_FORK.md links the DFD PR as the user of apache#3424 and explains the branch links in earlier Needed-by trailers. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3 Needed-by: Perseus/datafusion-distributed#7 Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit

Backport of apache#3424 (net diff of PR head 86e35d3 against its merge base beba39c), adapted to base 5b82ab9. Adds TableScanBuilder::with_column_stats(names) and FileScanTask::column_stats(). Tasks carry the data file's column sizes, value/null/NaN counts and lower/upper bounds only for the named primitive columns. By default no statistics are kept and behaviour is unchanged. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: scan/mod.rs conflicts resolved for the older base: resolve_field_id keeps its Option return (no apache#3070 ambiguity check), TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is inlined as the same int->long / float->double widening in task.rs. Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and test_utils imports SchemaRef. Details in DFD_FORK.md. Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit Issue: apache#3422

Adds the ledger entry for 9832f9c: upstream links and head SHA, each scan/mod.rs conflict resolution, the base-rev adaptations, why DFD needs it and when to drop it. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: none (docs only) Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit

… ledger with_column_stats' doc promised rejection of ambiguous case-insensitive names, which needs apache#3070 (not in this base). It now says names resolve as select does. DFD_FORK.md links the DFD PR as the user of apache#3424 and explains the branch links in earlier Needed-by trailers. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3 Needed-by: Perseus/datafusion-distributed#7 Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit

…canTask Bumps the [patch] rev for iceberg, iceberg-property-macro and iceberg-storage-opendal to Perseus/iceberg-rust@dacdc376, which adds a backport of apache/iceberg-rust#3424 and the DFD_FORK.md change ledger. No Rust changes.

github-actions Bot added the datafusion label Oct 12, 2026

JeonDaehong reviewed Oct 13, 2026

View reviewed changes

comphead reviewed Oct 13, 2026

View reviewed changes

anuragmantri commented Oct 13, 2026

View reviewed changes

JeonDaehong approved these changes Oct 14, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into file-scan-task-colu…

d689791

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/scan/mod.rs

JeonDaehong reviewed Oct 16, 2026

View reviewed changes

This was referenced Oct 16, 2026

Bound the sorted-scan merge by how many files' key ranges overlap apache/datafusion-iceberg#71

Open

[Epic] Sort-order reporting for iceberg-datafusion scans #3126

Open

JeonDaehong approved these changes Oct 17, 2026

View reviewed changes

This was referenced Oct 22, 2026

feat: plan Iceberg file tasks before execution apache/datafusion-iceberg#67

Open

feat: reconstruct eager Iceberg column statistics datafusion-contrib/datafusion-distributed#789

Draft

mbutrovich self-requested a review October 27, 2026 22:29

mbutrovich reviewed Oct 27, 2026

View reviewed changes

anuragmantri added 4 commits October 27, 2026 16:29

Merge remote-tracking branch 'upstream/main' into file-scan-task-colu…

93891ae

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/scan/mod.rs # crates/iceberg/src/test_utils/scan.rs

Merge remote-tracking branch 'upstream/main' into file-scan-task-colu…

c1de288

…mn-stats-f19b86d2 # Conflicts: # crates/iceberg/src/test_utils/scan.rs

anuragmantri commented Oct 29, 2026

View reviewed changes

Perseus mentioned this pull request Oct 30, 2026

build: point iceberg-rust fork at head carrying column stats on FileScanTask Perseus/datafusion-distributed#7

Draft

	column_stats: Option<FileScanTaskColumnStats>,
	#[builder(default, setter(transform = \|stats: Option<FileScanTaskColumnStats>\| stats.map(Box::new)))]
	column_stats: Option<Box<FileScanTaskColumnStats>>,

Conversation

anuragmantri commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

AI Disclosure

Uh oh!

JeonDaehong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeonDaehong commented Oct 14, 2026

Uh oh!

JeonDaehong left a comment

Choose a reason for hiding this comment

Uh oh!

JeonDaehong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Oct 12, 2026 •

edited

Loading