Skip to content

fix(scan): reject ambiguous case-insensitive fields - #3070

Merged
laskoviymishka merged 3 commits into
apache:mainfrom
manuzhang:fix-ambiguous-case-insensitive-fields
Oct 11, 2026
Merged

laskoviymishka merged 3 commits into
apache:mainfrom
manuzhang:fix-ambiguous-case-insensitive-fields

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

  • None.

What changes are included in this PR?

Case-insensitive schema lookup now records lower-cased name collisions instead of silently selecting one field. Scan projection and expression binding propagate a data-invalid error when a requested name is ambiguous, while case-sensitive lookup is unchanged.

This change was extracted from #2773 so the Unknown-type work can be reviewed independently.

Are these changes tested?

  • Added a regression covering ambiguous case-insensitive scan projection and expression binding.
  • cargo test -p iceberg --lib test_case_insensitive_scan_rejects_ambiguous_column_name
  • cargo fmt --all -- --check
  • cargo clippy -p iceberg --all-targets --all-features -- -D warnings

AI Disclosure

This PR was prepared with assistance from Codex.

@manuzhang
manuzhang force-pushed the fix-ambiguous-case-insensitive-fields branch from 1cfbee7 to aee375d Compare September 19, 2026 13:46
@manuzhang
manuzhang marked this pull request as ready for review September 19, 2026 13:58
Copilot AI lite review requested due to automatic review settings September 19, 2026 13:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@manuzhang

Copy link
Copy Markdown
Member Author

@CTTY @blackmwk please help review when you find time.

@Stefan-Dienst Stefan-Dienst left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @manuzhang,

I am not that familiar with the codebase, but your PR was a good read.

Generally I think that erroring in the case of ambiguity is a great change. I wondered if someone may unknowingly built upon this behavior, but as this ambiguity should lead to inconsistent field selection it's unlikely.

I just had concerns over test coverage. Maybe you could also cover that:

  • One can still select case insensitive unambiguous field names, even if a schema has case insensitive ambiguous field names.
  • One can still build a schema with case insensitive ambiguous field names.
  • General coverage for field_by_name_case_insensitive_checked

Comment thread crates/iceberg/src/scan/mod.rs Resolved
Comment thread crates/iceberg/src/scan/mod.rs Resolved
Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang force-pushed the fix-ambiguous-case-insensitive-fields branch from aee375d to 9a61a3e Compare September 23, 2026 16:52

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The direction is right — rejecting genuinely ambiguous case-insensitive names is the correct call, and the internal callers (bind, resolve_field_id) are converted cleanly. The tests Stefan asked for look like they've landed too.

What I'd want to settle before merge is the public surface. field_by_name_case_insensitive is a stable pub fn, and this quietly changes its contract: an ambiguous name now returns None, indistinguishable from "field not found" — where before it always handed back some field. The variant that can actually surface the ambiguity, _checked, is pub(crate), so external callers have no way to reach the new behavior. I'd promote _checked to pub and point the old doc comment at it, or at minimum document that None now means "missing or ambiguous."

A couple of smaller things while we're in here:

  • _checked has no doc comment, and it's the right spot to note that our per-name check is intentionally a Rust-only refinement — Java is all-or-nothing (any collision poisons every case-insensitive lookup) and PyIceberg silently keeps the last collider, so cross-engine users can see three different outcomes for the same table.
  • The sentinel drops the colliding IDs at build time, so the error can only name the lookup key, not what it collided with. If we want a Java-style "id and ID collide" message we'd need the map to hold the colliders — fine either way, but worth deciding now.

None of it is a big lift — sort out the public-surface piece and I think this is good to go.

Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/expr/term.rs Resolved
Comment thread crates/iceberg/src/scan/mod.rs Resolved
@Stefan-Dienst

Copy link
Copy Markdown

Thank you for changing the tests @manuzhang . Looks good 👍

Expose the checked lookup, document collision semantics, and cover the public fallback. Simplify the binding and scan callers.

Co-authored-by: Codex <codex@openai.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public-surface piece is sorted — _checked is pub now, the old field_by_name_case_insensitive doc points at it and spells out that None means missing-or-ambiguous, and the cross-engine note (Java all-or-nothing, PyIceberg keeps one collider) is exactly the caveat I was after. That was the thing blocking me last round, so from my side this is good to go.

Left some small stuff inline — the invalid_data! macro on the new arm, a readability tidy on the old fn. None of it blocks.

Overall LGTM, will left for some time to soak before merge. Nice work on this.

Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
Comment thread crates/iceberg/src/spec/schema/mod.rs Resolved
List sorted colliding field names with invalid_data!, simplify optional lookup, and loosen error assertions while checking the reported colliders.

Co-authored-by: Codex <codex@openai.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌬️ ⛵

@laskoviymishka
laskoviymishka added this pull request to the merge queue Oct 11, 2026
Merged via the queue into apache:main with commit d63e861 Oct 11, 2026
24 checks passed
Perseus pushed a commit to Perseus/iceberg-rust that referenced this pull request Oct 30, 2026
Backport of apache#3424 (net diff of PR head 86e35d3
against its merge base beba39c), adapted to base 5b82ab9.

Adds TableScanBuilder::with_column_stats(names) and
FileScanTask::column_stats(). Tasks carry the data file's column sizes,
value/null/NaN counts and lower/upper bounds only for the named
primitive columns. By default no statistics are kept and behaviour is
unchanged.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: scan/mod.rs conflicts resolved for the older base:
 resolve_field_id keeps its Option return (no apache#3070 ambiguity check),
 TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and
 apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is
 inlined as the same int->long / float->double widening in task.rs.
 Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and
 test_utils imports SchemaRef. Details in DFD_FORK.md.
Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Issue: apache#3422
Perseus added a commit to Perseus/iceberg-rust that referenced this pull request Oct 30, 2026
… ledger

with_column_stats' doc promised rejection of ambiguous case-insensitive
names, which needs apache#3070 (not in this base). It now
says names resolve as select does. DFD_FORK.md links the DFD PR as the
user of apache#3424 and explains the branch links in earlier Needed-by
trailers.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3
Needed-by: Perseus/datafusion-distributed#7
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Perseus pushed a commit to Perseus/iceberg-rust that referenced this pull request Oct 31, 2026
Backport of apache#3424 (net diff of PR head 86e35d3
against its merge base beba39c), adapted to base 5b82ab9.

Adds TableScanBuilder::with_column_stats(names) and
FileScanTask::column_stats(). Tasks carry the data file's column sizes,
value/null/NaN counts and lower/upper bounds only for the named
primitive columns. By default no statistics are kept and behaviour is
unchanged.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: scan/mod.rs conflicts resolved for the older base:
 resolve_field_id keeps its Option return (no apache#3070 ambiguity check),
 TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and
 apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is
 inlined as the same int->long / float->double widening in task.rs.
 Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and
 test_utils imports SchemaRef. Details in DFD_FORK.md.
Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Issue: apache#3422
Perseus added a commit to Perseus/iceberg-rust that referenced this pull request Oct 31, 2026
… ledger

with_column_stats' doc promised rejection of ambiguous case-insensitive
names, which needs apache#3070 (not in this base). It now
says names resolve as select does. DFD_FORK.md links the DFD PR as the
user of apache#3424 and explains the branch links in earlier Needed-by
trailers.

Upstream-PR: apache#3424
Upstream-head: 86e35d3
Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3
Needed-by: Perseus/datafusion-distributed#7
Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants