Conversation
_added_data_files and _deleted_data_files return iterators. Both validators called any() on the iterator and then built the error message from the same, now partially consumed, iterator. any() stops at the first truthy element, so the set comprehension saw only what remained: with a single conflicting entry it produced an empty set, and with several it silently dropped the first. The result was a ValidationException that could not name the snapshot it conflicted with -- 'Added data files were found matching the filter for snapshots set()!' -- which makes a real conflict hard to diagnose. Materialise the entries once before testing them. The existing tests did not catch this because they patch the helpers with a list, which can be iterated twice. The new tests patch with an iterator, as the real helpers return, and assert every conflicting snapshot id reaches the message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
_added_data_filesand_deleted_data_filesreturnIterator[ManifestEntry]. Both validators callany()on the iterator and then build the error message from that same, now partially consumed, iterator:any()stops at the first truthy element, so the set comprehension only sees what is left:set()The single-entry case is the common one, and it produces a
ValidationExceptionthat cannot name the snapshot it conflicted with:I hit this on a concurrent
Table.overwritewith anoverwrite_filter, where two writers do a read-modify-write of the same key. The validation itself is correct — there genuinely was a conflicting file — but the message gives nothing to debug with, and "snapshots set()" reads like an internal error rather than a real conflict.Affects
_validate_added_data_filesand_validate_deleted_data_files. Both are fixed here.Are these changes tested?
Yes. Two tests added to
tests/table/test_validate.py, one per validator. Both fail onmainand pass with the fix:With the fix,
tests/table/test_validate.pyis 18 passed.The existing
test_validate_added_data_files_raises_on_conflictdid not catch this because it patches the helper with a list, which can be iterated twice. The new tests patch with aniter(...), matching what the real helpers return, and assert that every conflicting snapshot id reaches the message — including the multi-entry case, which covers the silent drop as well as the empty set.Are there any user-facing changes?
Only the exception message, which now names the conflicting snapshots as it was always meant to. No behavioural change to when validation raises.