Repository navigation
Conversation
When replace_with_order_preserving_variants turns a CoalescePartitionsExec under GlobalLimitExec(skip=N) into a SortPreservingMergeExec, include N in the pushed fetch so the outer limit still has enough rows after skipping. Reuses sort_pushdown::input_fetch. Closes apache#26690.
Contributor
|
Thank you for the contribution. I am closing this under DataFusion's AI Dump policy: https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump Rationale: this account has opened 10 PRs against DataFusion since Oct 16, and 8 of them have already been closed under the AI policy this week (see #26919, #26914, #26912, #26905, #26823, #26808, #26432). Multiple AI-generated PRs in a short period from a new contributor are treated as spam per the policy. One high-quality PR worked through with maintainers is far more valuable to the project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
ORDER BY ... LIMITover anOFFSETsubquery returns too few rows when the table is sorted and has several files #26690.Rationale for this change
On a pre-sorted multi-file table,
SELECT * FROM (SELECT * FROM t OFFSET 5) ORDER BY a LIMIT 10can return fewer than 10 rows (sometimes none).replace_with_order_preserving_variantsturns theCoalescePartitionsExecunder thatOFFSETinto aSortPreservingMergeExecand gives it the TopKfetch(10). The merge therefore stops after 10 rows, and the outerGlobalLimitExecthen skips 5 of them, so only 5 come back.Sort pushdown already accounts for this with
input_fetch: when a fetch moves below aGlobalLimitExec, it addsskipfirst. This path did not.What changes are included in this PR?
plan_with_order_preserving_variantsnow passesinput_fetch(plan, fetch)to children instead of the bare parent fetch, so a merge underOFFSET 5with TopK 10 getsfetch=15.sort_pushdown::input_fetchpub(super)so both sites share the same rule.fetch=15for that case.What is the testing strategy for this PR?
test_plan_with_order_preserving_variants_accounts_for_limit_skip.replace_with_order_preserving_variantsandinput_fetchtests still pass.Are there any user-facing changes?
No API changes. Queries that previously under-returned after
OFFSET+ORDER BY ... LIMITon sorted multi-partition data should now return the expected number of rows.AI assistance
Used an AI coding assistant to help locate the bug site and draft the change. I reviewed the logic against #26690 and the existing
input_fetchhelper, ran the tests above, and wrote this description myself.