Skip to content

fix: account for OFFSET when pushing fetch into SortPreservingMergeExec - #26935

Closed
ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/26690-offset-limit-order-preserving-fetch
Closed

ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/26690-offset-limit-order-preserving-fetch

Conversation

@ump45nose

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

On a pre-sorted multi-file table, SELECT * FROM (SELECT * FROM t OFFSET 5) ORDER BY a LIMIT 10 can return fewer than 10 rows (sometimes none).

replace_with_order_preserving_variants turns the CoalescePartitionsExec under that OFFSET into a SortPreservingMergeExec and gives it the TopK fetch (10). The merge therefore stops after 10 rows, and the outer GlobalLimitExec then skips 5 of them, so only 5 come back.

Sort pushdown already accounts for this with input_fetch: when a fetch moves below a GlobalLimitExec, it adds skip first. This path did not.

What changes are included in this PR?

  • plan_with_order_preserving_variants now passes input_fetch(plan, fetch) to children instead of the bare parent fetch, so a merge under OFFSET 5 with TopK 10 gets fetch=15.
  • Made sort_pushdown::input_fetch pub(super) so both sites share the same rule.
  • Added a unit test that checks the merge receives fetch=15 for that case.

What is the testing strategy for this PR?

  • New unit test: test_plan_with_order_preserving_variants_accounts_for_limit_skip.
  • Existing replace_with_order_preserving_variants and input_fetch tests still pass.

Are there any user-facing changes?

No API changes. Queries that previously under-returned after OFFSET + ORDER BY ... LIMIT on sorted multi-partition data should now return the expected number of rows.

AI assistance

Used an AI coding assistant to help locate the bug site and draft the change. I reviewed the logic against #26690 and the existing input_fetch helper, ran the tests above, and wrote this description myself.

When replace_with_order_preserving_variants turns a CoalescePartitionsExec
under GlobalLimitExec(skip=N) into a SortPreservingMergeExec, include N in
the pushed fetch so the outer limit still has enough rows after skipping.

Reuses sort_pushdown::input_fetch. Closes apache#26690.
@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate labels Oct 30, 2026
@alamb alamb added the Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days label Oct 30, 2026
@alamb

alamb commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Thank you for the contribution. I am closing this under DataFusion's AI Dump policy: https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump

Rationale: this account has opened 10 PRs against DataFusion since Oct 16, and 8 of them have already been closed under the AI policy this week (see #26919, #26914, #26912, #26905, #26823, #26808, #26432). Multiple AI-generated PRs in a short period from a new contributor are treated as spam per the policy. One high-quality PR worked through with maintainers is far more valuable to the project.

@alamb alamb closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ORDER BY ... LIMIT over an OFFSET subquery returns too few rows when the table is sorted and has several files

2 participants