Repository navigation
Break datafusion crate into smaller crates #1750
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
a new feature: to break datafusion crate into separate smaller crates.
It helps with code management and dependency reasoning
Describe the solution you'd like
- Create new
datafusion_commoncrate #1752 - Create new
datafusion_exprcrate #1753- [split/3] split datafusion expr module and move aggregate and window function expr #1759
- [split/5] move window frame and operator to datafusion-expr module #1761
- [split/6] move signature, type signature, volatility to datafusion-expr #1763
- [split/7] move built-in scalar function to datafusion-expr #1764
- [split/8] move
AccumulatorandColumnarValueto datafusion-expr #1765 - [split/9+12] move udf, udaf,
Exprto datafusion-expr module #1762 - [split/10] split up expr for rewriting, visiting, and simplification traits #1774
- [split/11] split expr type and null info to be expr-schemable #1784
- [split/9+12] move udf, udaf,
Exprto datafusion-expr module #1762 - [split/13] move rest of expr to expr_fn in datafusion-expr module #1794
- split datafusion-physical-expr module #1844
- split datafusion-datasource module #1772
- file format
- listing and object store
- split datafusion-logical-plan sub-module #1755
- move logical plan builder
- move logical plan structs
- move move optimizers
- split datafusion-physical-plan sub-module #1754
- move physical planner
- move execution exprs
- move physical plan optimizer
- split datafusion-execution sub-module
- move memory and disk manager
- move stats collection
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Reacted by QP Hou, Yijie Shen, Jon Mease, Andy Grove and jakevin
Activity
jimexist commentedon Feb 5, 2022
cc @alamb @houqp @Dandandan what do you think?
alamb commentedon Feb 5, 2022
I like this idea @jimexist 👍 There is some related commentary / information here: #348 as well
Some other crates that might be useful to consider;
datafusion_core(DataFusionError,DFSchema, etc)datafusion_datasource(the built in parquet, avro, csv and json readers and supporting logic)andygrove commentedon Feb 5, 2022
This will be helpful for use cases where users are looking to use DataFusion in a similar fashion to Calcite, for query parsing and planning, but not for execution. I like this idea.
houqp commentedon Feb 6, 2022
Thank you @jimexist for taking on this! I think this is the right path forward.
cc @jorgecarleitao and @yjshen since they have proposed the similar ideas before.
yahoNanJing commentedon Feb 7, 2022
Hi @jimexist, is it possible to make an exclusive crate for the data source so that it will be easy for the integration of the different kinds of remote object store?
Suppose I hope to introduce the hdfs as one of the remote object store. To be less intrusive, it's better to make an independent datafusion-objectstore-hdfs crate which depends on the datasource crate of datafusion. Then it would be much easier for other crates to decide whether to depend on the datafusion-hdfs crate or not without cyclic dependencies.
jimexist commentedon Feb 7, 2022
yes it's a good idea and i believe it's included in the list already
yahoNanJing commentedon Feb 8, 2022
Thanks @jimexist
jimexist commentedon Feb 8, 2022
my current plan is to finish items for:
datafusion-commondatafusion-exprbefore release 7:
and finish the rest after the release, to cap the amount of changes in a release
Igosuki commentedon Feb 9, 2022
@jimexist could this lead to a smaller ballista client crate as well ? Potentially, this could greatly speed up compilation of programs who just want to be datafusion clients and not run the entire stack.
jimexist commentedon Feb 9, 2022
i'm not sure as of now - but probably if we split up logical/physical planning further - but that won't happen soon
alamb commentedon Feb 9, 2022
Related comment: #1762 (comment)
jimexist commentedon Jul 20, 2022
i wanted to continue iterating on this split but obviously i didn't have time to keep up. a lot has happened in the last few months. i wonder if this is still relevant or it shall be closed now. @alamb and @andygrove any suggestions?
alamb commentedon Jul 20, 2022
@jimexist welcome back!
I would say that the main "break datafusion into smaller crates" has been completed. There are still some items in the description of this ticket that might not be done - I would personally recommend filing a new issue with any items that you think would still be good to work on and then closing this one.
alamb commentedon Oct 24, 2022
Closing this ticket and we can track further splitting in follow on issues. Again 👏
alamb commentedon Nov 11, 2022
I am thinking of trying to push us to the next level here #4181