Repository navigation
Library Guide: Extending DataFusion's operators: custom LogicalPlan and ExecutionPlans #7308
Description
Is your feature request related to a problem or challenge?
Part of #7014
If we want to have DataFusion used as the core of many new systems, we need it to be as easy as possible for someone to get their idea working on top of DataFusion.
Thanks to @tshauck we now have a basic Library Users Guide ❤️ and this ticket describes expanding it out
Describe the solution you'd like
Fill in the content of https://arrow.apache.org/datafusion/library-user-guide/extending-operators.html
We can draw inspiration from https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/user_defined/user_defined_plan.rs
Example Outline
- Introduce an example plan node that can not be expressed with existing relational operators (maybe pivot rows to columns, like here)
- Show how to define the Logical extension user defined node
- SHow how to use an extension planner physical planner to plan such a node (example here)
- Show how to create a simplified execution plan / stream
The examples directory holds a bunch more of examples: https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples
Describe alternatives you've considered
No response
Additional context
No response
Activity
brayanjuls commentedon Aug 19, 2024
I was investigating about pivoting in the DataFrame API and found some of the links in this issues are broken, leaving the replacement here for someone trying to work on this in the future
alamb commentedon Aug 19, 2024
Thanks @brayanjuls
Tangeroooo commentedon Feb 19, 2025
Can I take this issue?
alamb commentedon Feb 19, 2025
Yes of course, please do! Please ping me on the PR if you need a reviewer as improving the DataFusion documentation is high on my list of things to do
@Tangeroooo now that we are running the examples in the user guide, we can probably move the entire example into the docs (as they will still be tested)