Repository navigation
fix: return no rows from SortExec with fetch=0 instead of panicking i… - #27000
Conversation
…n TopK A SortExec with fetch = Some(0) over unsorted input built a TopK with k = 0, and TopKHeap::new panicked on assert!(k > 0). LIMIT 0 is usually removed by eliminate_limit, but the plan is reachable with datafusion.optimizer.max_passes = 0 or when the plan is built directly. SortExec::execute now returns an empty stream for fetch = Some(0) and marks the partition as emitted on the shared TopK dynamic filter, so the filter still completes once every partition has run. Closes apache#26431.
| let filter = self.filter.as_ref(); | ||
| let filter = unwrap_or_internal_err!(filter); | ||
| filter.read().mark_topk_emitted(); |
There was a problem hiding this comment.
Thank you for the fix.
I can understand why enter this branch and return empty set, but I don't get those three lines, for example why don't the above LimitStream branch have to special handle dynamic filter like this
There was a problem hiding this comment.
I added them because the new arm takes the place of TopK, and TopK::emit() calls mark_topk_emitted() for each partition so the shared filter counts down and gets marked complete. I mirrored that so the filter ends in the same state as a normal TopK run, since I was worried something waiting on wait_complete() could hang otherwise.
Looking at it again, you have a point, nothing outside tests waits on completion, and the LimitStream branch already leaves the filter incomplete without issues, so it isn't needed. I'll remove those lines so this arm just returns an empty stream, which also lets mark_topk_emitted go back to private.
| /// a `SortExec` with `fetch = Some(0)` over unsorted input used to panic | ||
| /// in `TopKHeap::new`. It must return no rows, and the shared TopK | ||
| /// dynamic filter must still complete once every partition has run. | ||
| #[tokio::test] |
There was a problem hiding this comment.
See https://datafusion.apache.org/contributor-guide/testing.html#choosing-what-kind-of-test-to-write
This UT is asserting on low-level implementations (probing into the dynamic filter), make it hard to understand and maintain.
We could either:
- write it as only construct the SortExec and run
- or remove it, we already have slt I think it's good already
There was a problem hiding this comment.
Thanks! Since you're fine with the slt, I'll remove this.
The LimitStream branch already leaves the shared dynamic filter incomplete and nothing outside tests waits on completion, so the fetch=0 arm can just return an empty stream. Drop the unit test that checked the filter state; limit.slt covers the panic.
|
Thanks @shahwazreza @2010YOUY01 ! |
Which issue does this PR close?
ORDER BY ... LIMIT 0panics in TopK (assertion failed: k > 0) wheneliminate_limitdoes not run #26431Rationale for this change
SELECT ... ORDER BY ... LIMIT 0panics instead of returning an empty result when the limit makes it to the physical plan. Normallyeliminate_limitremovesLIMIT 0before that happens, so most people won't run into it. But it does reach execution withdatafusion.optimizer.max_passes = 0, or when someone builds the physical plan directly. In that caseSortExecwithfetch = Some(0)builds aTopKwithk = 0, andTopKHeap::newhitsassert!(k > 0).What changes are included in this PR?
SortExec::executehandlesfetch = Some(0)on unsorted input before it gets to the TopK path, and returns an empty stream.mark_topk_emittedis nowpub(crate)soSortExeccan call it.I left the
assert!(k > 0)inTopKHeapalone. Removing it wouldn't be enough anyway, sinceTopKHeap::addalso assumesk > 0(it callspeek_mut().unwrap()whenlen == k), so handling the case inSortExecis simpler.What is the testing strategy for this PR?
test_sort_fetch_zero_returns_empty_and_completes_filterinsort.rs. It runs a two-partitionSortExecwithfetch = Some(0)and checks that both partitions return no rows. It also checks that the dynamic filter completes only after the last partition has run.limit.sltthat setsmax_passes = 0, so theLIMIT 0survives into the physical plan. It checks theEXPLAINoutput (SortExec: TopK(fetch=0)) and that the query returns no rows.Are there any user-facing changes?
The query now returns an empty result instead of panicking.