Skip to content

Add API to read a Vec<RecordBatch> from SessionContext #9157

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

Many APIs in DataFusion produce Vec<RecordBatch> (e.g. DataFrame::collect)

However, there is no corresponding API to create a DataFrame from a Vec<RecordBatch> which is confusing to a first time user who just wants to do something like "sort my batches"

It is straightforward to scan Vec<RecordBatch> by create one with a MemTable, as is done here: https://docs.rs/datafusion/latest/src/datafusion/execution/context/mod.rs.html#939-950 but having to find that incantation puts a barrier to sue

There is a similar API for one batch SessionContext::read_batch but not one for the Vec<RecordBatch>

Describe the solution you'd like

I would like a SessionContext::read_batches API that takes an iterator of RecordBatches (like a Vec)

impl SessionContext
  /// Creates a [`DataFrame`] for reading a [`RecordBatch`]
    pub fn read_batch(&self, batches: impl IntoIter<Item = RecordBatch>) -> Result<DataFrame> {
      ...
    }
}

Along with:

  1. A doc example showing how to read a Vec<RecordBatch> (which would also serve as a test)

Describe alternatives you've considered

No response

Additional context

I was speaking with @carols10cents today, and she noted that m

Activity

  1. Lordworms commented on Feb 8, 2024

    @Lordworms
    Contributor

    I can do this one

  2. alamb commented on Feb 8, 2024

    @alamb
    ContributorAuthor

    Thank you @Lordworms

  3. added a commit that references this issue on Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions