Repository navigation
Add ProgressiveEval operator for optimize SortPreservingMerge #10488
Description
Is your feature request related to a problem or challenge?
Yes, this is a task of #10316. We will use the porting ProgressiveEval to optimize SortPreservingMerge
Describe the solution you'd like
In InfluxDB IOx, when the inputs of SortPreservingMerge are all sorted on the sort key and their data do not overlap, we replace SortPreservingMerge with ProgressiveEval which:
- Avoids starting all input streams at once
- Avoids having to compare any keys (doesn't actually do a merge)
We wrote about using this operator here: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/
This ticket is to port ProgressiveEval from InfluxDB to DataFusion
Describe alternatives you've considered
No response
Additional context
No response
Activity
NGA-TRAN commentedon May 13, 2024
I start working on this
[-]Porting `ProgressiveEval` from InfluxDB IOx to optimize `SortPreservingMerge`[/-][+]Add `ProgressiveEval` for optimize `SortPreservingMerge`[/+]alamb commentedon May 15, 2024
I updated this ticket's description with some additional background and linked: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/
There is a PR for adding this operator here: #10490
[-]Add `ProgressiveEval` for optimize `SortPreservingMerge`[/-][+]Add `ProgressiveEval` operator for optimize `SortPreservingMerge`[/+]alamb commentedon May 15, 2024
I think this ticket largely duplicates #10316 so let's use that instead
alamb commentedon Apr 1, 2025
Duplicated by #15191
SortPreservingMerge-->ProgressiveEval#15191