Skip to content

Optimization rule CommonSubexprEliminate creates invalid projections #2907

Closed

Description

@andygrove

Describe the bug
I am trying to run some complex SQL and am seeing errors such as:

thread 'q11' panicked at 'index out of bounds: the len is 12 but the index is 12', /home/andy/.cargo/git/checkouts/arrow-datafusion-71ae82d9dec9a01c/9401d6d/datafusion/optimizer/src/projection_push_down.rs:159:39

With the changes in #2900 I see a slighter more informative error:

FAILURE: Plan("Projection has mismatch between number of expressions (12) and number of fields in schema (42)")

I can clearly see that some of the code in build_project_plan is incorrect. It is meant to be building the projection output schema but is including all the fields from the projection input. Unfortunately, I do not understand the code well enough to try and fix this issue.

    let mut schema = DFSchema::new_with_metadata(fields, HashMap::new())?;
    schema.merge(input.schema()); // <--- this is not correct

    Ok(LogicalPlan::Projection(Projection::try_new_with_schema(
        project_exprs,
        Arc::new(input),
        Arc::new(schema),
        None,
    )?))

To Reproduce
I do not have a repro to share currently. I will try and create one.

Expected behavior
Should not fail. Rules should avoid changing plans if there are errors rather than panic.

Additional context
I have created #2909 as a workaround so that we can at least fail gracefully and skip this rule when it produces an error and still apply the other rules.

Activity

  1. self-assigned this
    on Jul 14, 2022
  2. changed the title [-]Optimization rules create invalid projections[/-] [+]Optimization rule `CommonSubexprEliminate` creates invalid projections[/+] on Jul 14, 2022
  3. andygrove commented on Jul 14, 2022

    @andygrove
    MemberAuthor

    @waynexia fyi - if you have any time available maybe you can give some guidance on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions