Skip to content

Row Hash loads whole aggregation state to memory before sending #3460

Closed

Description

@milenkovicm

Describe the bug

Row Hash aggregation, loads whole aggregation state to memory before sending a single batch downstream. The resulting record batch will have more rows than predefined batch size

problematic part of code https://github.com/milenkovicm/arrow-datafusion/blob/17f069df4227b837cf2741a545c39a8b68d5fd76/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L438
where iterator without limits is crated, and whole state is cloned, which doubles memory needed for the aggregation state.

function poll_next creates single batch https://github.com/milenkovicm/arrow-datafusion/blob/17f069df4227b837cf2741a545c39a8b68d5fd76/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L146

To Reproduce

Run an aggregation

Expected behavior

Resulting aggregation should be chunked according to the predefined batch size

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions