Repository navigation
Support for != predicate in pruning predicates #420
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
While evaluating queries against data stored in containers / multiple files, it is helpful to prune entire files using statistics (see #363 for more details). DataFusion has this logic already for ==, <, <=, > and >= operators in the pruning predicate.
However, as @NGA-TRAN noticed, there is no support for != at the moment.
Describe the solution you'd like
Add support and tests for != in predicate pruning logic
Activity
apache#420: Support for not_eq predicate in pruning predicates
jgoday commentedon Jun 10, 2021
@alamb Can I try to solve this issue ?
If I understand it correctly, for the non equal predicate the expression should be pruned if the literal value does not fall between the min and max values, Am I right ?
(I have already tried to implement it here (jgoday@3bb55a4), I can make a PR if you approve it)
alamb commentedon Jun 10, 2021
Thanks @jgoday ! That would be great
For
!=I think we would prune the container (aka return false or NULL) from the predicate if the constant value did fall within the min/max bounds but I might be misunderstanding what you are saying; I'll love to check out the PRjgoday commentedon Jun 11, 2021
@alamb I think that I express myself incorrectly :)
Created #544 PR.
#420: Support for not_eq predicate in pruning predicates (#544)
Revert "apache#420: Support for not_eq predicate in pruning predicates (