Repository navigation
fix(scan): reject ambiguous case-insensitive fields - #3070
Conversation
1cfbee7 to
aee375d
Compare
There was a problem hiding this comment.
Hi @manuzhang,
I am not that familiar with the codebase, but your PR was a good read.
Generally I think that erroring in the case of ambiguity is a great change. I wondered if someone may unknowingly built upon this behavior, but as this ambiguity should lead to inconsistent field selection it's unlikely.
I just had concerns over test coverage. Maybe you could also cover that:
- One can still select case insensitive unambiguous field names, even if a schema has case insensitive ambiguous field names.
- One can still build a schema with case insensitive ambiguous field names.
- General coverage for
field_by_name_case_insensitive_checked
Co-authored-by: Codex <codex@openai.com>
aee375d to
9a61a3e
Compare
laskoviymishka
left a comment
There was a problem hiding this comment.
The direction is right — rejecting genuinely ambiguous case-insensitive names is the correct call, and the internal callers (bind, resolve_field_id) are converted cleanly. The tests Stefan asked for look like they've landed too.
What I'd want to settle before merge is the public surface. field_by_name_case_insensitive is a stable pub fn, and this quietly changes its contract: an ambiguous name now returns None, indistinguishable from "field not found" — where before it always handed back some field. The variant that can actually surface the ambiguity, _checked, is pub(crate), so external callers have no way to reach the new behavior. I'd promote _checked to pub and point the old doc comment at it, or at minimum document that None now means "missing or ambiguous."
A couple of smaller things while we're in here:
_checkedhas no doc comment, and it's the right spot to note that our per-name check is intentionally a Rust-only refinement — Java is all-or-nothing (any collision poisons every case-insensitive lookup) and PyIceberg silently keeps the last collider, so cross-engine users can see three different outcomes for the same table.- The sentinel drops the colliding IDs at build time, so the error can only name the lookup key, not what it collided with. If we want a Java-style "id and ID collide" message we'd need the map to hold the colliders — fine either way, but worth deciding now.
None of it is a big lift — sort out the public-surface piece and I think this is good to go.
|
Thank you for changing the tests @manuzhang . Looks good 👍 |
Expose the checked lookup, document collision semantics, and cover the public fallback. Simplify the binding and scan callers. Co-authored-by: Codex <codex@openai.com>
laskoviymishka
left a comment
There was a problem hiding this comment.
The public-surface piece is sorted — _checked is pub now, the old field_by_name_case_insensitive doc points at it and spells out that None means missing-or-ambiguous, and the cross-engine note (Java all-or-nothing, PyIceberg keeps one collider) is exactly the caveat I was after. That was the thing blocking me last round, so from my side this is good to go.
Left some small stuff inline — the invalid_data! macro on the new arm, a readability tidy on the old fn. None of it blocks.
Overall LGTM, will left for some time to soak before merge. Nice work on this.
List sorted colliding field names with invalid_data!, simplify optional lookup, and loosen error assertions while checking the reported colliders. Co-authored-by: Codex <codex@openai.com>
Backport of apache#3424 (net diff of PR head 86e35d3 against its merge base beba39c), adapted to base 5b82ab9. Adds TableScanBuilder::with_column_stats(names) and FileScanTask::column_stats(). Tasks carry the data file's column sizes, value/null/NaN counts and lower/upper bounds only for the named primitive columns. By default no statistics are kept and behaviour is unchanged. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: scan/mod.rs conflicts resolved for the older base: resolve_field_id keeps its Option return (no apache#3070 ambiguity check), TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is inlined as the same int->long / float->double widening in task.rs. Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and test_utils imports SchemaRef. Details in DFD_FORK.md. Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit Issue: apache#3422
… ledger with_column_stats' doc promised rejection of ambiguous case-insensitive names, which needs apache#3070 (not in this base). It now says names resolve as select does. DFD_FORK.md links the DFD PR as the user of apache#3424 and explains the branch links in earlier Needed-by trailers. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3 Needed-by: Perseus/datafusion-distributed#7 Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Backport of apache#3424 (net diff of PR head 86e35d3 against its merge base beba39c), adapted to base 5b82ab9. Adds TableScanBuilder::with_column_stats(names) and FileScanTask::column_stats(). Tasks carry the data file's column sizes, value/null/NaN counts and lower/upper bounds only for the named primitive columns. By default no statistics are kept and behaviour is unchanged. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: scan/mod.rs conflicts resolved for the older base: resolve_field_id keeps its Option return (no apache#3070 ambiguity check), TableScanBuilder keeps snapshot_id (no apache#3260 snapshot_selection), and apache#3070's ambiguity tests are not carried. Datum::promote_to (apache#3483) is inlined as the same int->long / float->double widening in task.rs. Tests use Type::Primitive(..) (pre-apache#3527 NestedField signature) and test_utils imports SchemaRef. Details in DFD_FORK.md. Needed-by: https://github.com/Perseus/datafusion-distributed/tree/iceberg-dyn-filter/f2-fork-column-stats Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit Issue: apache#3422
… ledger with_column_stats' doc promised rejection of ambiguous case-insensitive names, which needs apache#3070 (not in this base). It now says names resolve as select does. DFD_FORK.md links the DFD PR as the user of apache#3424 and explains the branch links in earlier Needed-by trailers. Upstream-PR: apache#3424 Upstream-head: 86e35d3 Backport-notes: doc wording differs from upstream; see DFD_FORK.md entry 3 Needed-by: Perseus/datafusion-distributed#7 Remove-when: the pinned iceberg-rust rev contains apache#3424's merge commit
Which issue does this PR close?
What changes are included in this PR?
Case-insensitive schema lookup now records lower-cased name collisions instead of silently selecting one field. Scan projection and expression binding propagate a data-invalid error when a requested name is ambiguous, while case-sensitive lookup is unchanged.
This change was extracted from #2773 so the Unknown-type work can be reviewed independently.
Are these changes tested?
cargo test -p iceberg --lib test_case_insensitive_scan_rejects_ambiguous_column_namecargo fmt --all -- --checkcargo clippy -p iceberg --all-targets --all-features -- -D warningsAI Disclosure
This PR was prepared with assistance from Codex.