Replace Filter: Boolean(false) with EmptyRelation #3864

Activity

1. andygrove

   added

   enhancementNew feature or requestNew feature or request

   optimizerOptimizer rulesOptimizer rules

   on Oct 17, 2022
2. mingmwang commented on Oct 18, 2022

   

   mingmwang

   on Oct 18, 2022

   Contributor

   More actions

   We can have two general rules: FoldFilters and EmptyRelationPropagation.
   FoldFilters can remove the filters which can be evaluated trivially. If Filter condition is alway false, replace it with an EmptyRelation

   EmptyRelationPropagation can propagate the empty relation up to the tree, for example, inner join with EmptyRelation can return EmptyRelation.
3. mingmwang commented on Oct 18, 2022

   

   mingmwang

   on Oct 18, 2022

   Contributor

   More actions

   @andygrove Looks like there is already a rule EliminateFilter for this purpose.

   👍React with 👍1Reacted by Andrew Lamb
4. alamb commented on Oct 18, 2022

   

   alamb

   on Oct 18, 2022

   Contributor

   More actions

   Perhaps the changes in #3841 from @Dandandan made it possible to simplify the filter

   Perhaps we could solve this issue by adding another run of EliminateFilter at the end of the list in datafusion/optimizer/src/optimizer.rs
5. alamb commented on Oct 22, 2022

   

   alamb

   on Oct 22, 2022

   Contributor

   More actions

   @andygrove can you share a reproducer? We can then add it to the regression tests to make sure the empty filter is indeed eliminated and doesn't come back in
6. jackwener commented on Nov 12, 2022

   

   jackwener

   on Nov 12, 2022

   Last edited by jackwener

   Member

   More actions

   Datafusion only can propagate EmptyRelation just optimizer_config.max_passes times. Because we depend on multi-passes optimization to multiply elimination.

   In other words, we currently don't have really EmptyRelationPropagation.

   We can reproduce it.

   After 3 optimizations, the plan still can continue to be eliminated.

    CREATE TABLE IF NOT EXISTS t1 AS VALUES(1,'HELLO'),(12,'DATAFUSION');
    CREATE TABLE IF NOT EXISTS t2 AS VALUES(1,'HELLO'),(12,'DATAFUSION');
    CREATE TABLE IF NOT EXISTS t3 AS VALUES(1,'HELLO'),(12,'DATAFUSION');
   
   explain verbose  select column1 from t1 join ( select column1 from t2 join (select column1 from t3 where false ) as ta2 on t2.column1 = ta2.column1 ) as ta1 on t1.column1 = ta1.column1;
7. andygrove commented on Nov 14, 2022

   

   andygrove

   on Nov 14, 2022

   MemberAuthor

   More actions

   FWIW, This is no longer an issue for me in Dask SQL. I am not sure what fixed it but maybe the multiple optimizer passes.
8. alamb

   closed this as completedin #4192on Nov 16, 2022