Repository navigation
Consider adopting IOx ObjectStore abstraction #2489
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In another issue @alamb and @tustvold suggested we might want to use the IOx ObjectStore implementation.
A few nice points I'll mention about the IOx one:
- They have some nice path utilities, including a CloudPath struct. That seems nicer than the current one with
&strpaths. - Has implementations for S3, GCS, Azure Blob Storage included in the repo. There is no HDFS support yet.
- Has implementations of
put()for writing. There doesn't seem to be streaming write support (multi-part upload).
There are a few differences in the API:
- The IOx
list()implementation evaluated prefixes on path segments: "Prefixes are evaluated on a path segment basis, i.e.foo/bar/is a prefix offoo/bar/xbut not offoo/bar_baz/x." - IOx doesn't have a synchronous read implementation.
There of course exist other repos that this has implications for:
- https://github.com/datafusion-contrib/datafusion-objectstore-s3
- https://github.com/datafusion-contrib/datafusion-objectstore-hdfs
- https://github.com/datafusion-contrib/datafusion-objectstore-azure
From what I've seen, it seems like we could reasonably shift to simply use the IOx ObjectStore. But if there's a good reason, we could also reuse useful parts of the implementation to keep the existing API.
Activity
matthewmturner commentedon May 9, 2022
Indeed the IOx
ObjectStoreimplementation looks both robust and feature rich - and has the added benefit of being actively used. Whiledatafusion-objectstore-s3has been released on crates i am not aware of any production workloads leveraging it.I also prefer the more generic API interface in the IOx implementation, I had actually planned on proposing something similar on #2445.
From an s3 perspective the only things that come to mind are:
datafusion-objectstore-s3works without issue. In particular for connecting to services such as MinIO (which is S3 compatible storage) - I have spoken to them and know they were watching the project.As for the actual implementation, given the IOx implementation already has AWS, GCP, and Azure functionality features could we just create
datafusion-objectstoreindatafusion-contriband let people choose what they want by choosing the relevant feature?alamb commentedon May 9, 2022
I believe @tustvold is actively working on preparing the iox code for crates.io release in https://github.com/influxdata/influxdb_iox/pull/4534
tustvold commentedon May 9, 2022
Yeah as alluded to by @alamb, my plan is to get the iox code released to crates.io so that DataFusion could use it.
There would then be a couple of potential courses of action for DataFusion:
object_storecrate to fetch parquet files to local disk. This would potentially fetch more bytes from object storage, but as described in RFC: Spill-To-Disk Object Storage Download #2205 this may actually be faster than the current approach. It would also be temporary pending Push-Based Parquet Reader arrow-rs#1605object_storecratetustvold commentedon May 13, 2022
I've released the crate to crates.io - https://crates.io/crates/object_store, I'm going to take a stab at integrating this into Datafusion over this weekend. Hopefully I'll get something up as a workable draft
thinkharderdev commentedon May 16, 2022
Wrt fetching to local disk, we have an implementation of (datafusion)
ObjectStorein our project which adopts the S3A approach to minimize the number of small range requests. Basically, we set a minimum chunk size for S3 reads (usually 64K). If a read of less than 64K is requested, we go ahead and fetch 64K and buffer it in memory. Subsequent reads that fall within that buffer are returned from the in-memory buffer. This minimizes the overhead of small range requests from thePageIteratorwhile still avoiding reads of columns not required for the query.tustvold commentedon May 16, 2022
Yeah, buffered prefetch is one way to mitigate the small read problem. However, it does not allow for coalescing adjacent reads - i.e. you will still likely end up with one request per column chunk unless you have tiny columns.
TBC my preference is for 3, which mirrors the new vectored API if S3a, but I'm currently working on 2 first to ensure there aren't any fundamental integration issues.
alamb commentedon May 16, 2022
I believe @tustvold is working on this issue
thinkharderdev commentedon May 16, 2022
Cool. In our case the buffered prefetch helps marginally (but we also have a lot of sparse columns so it is a slightly special case which does a reasonable job at coalescing adjacent reads).
apache/arrow-rs#1605 looks like a really good idea. We're also working on trying to optimize S3 reads at the moment so if there's any way I can help please let me know!
jychen7 commentedon May 28, 2022
that's great, at a time, I want to create
datafusion-objectstore-gcssimilar to s3/azure, since I mostly play with GCP. And want to try how datafusion and also ballista can query GCSSwitch to object_store crate (apache#2489)
Switch to object_store crate (#2489) (#2677)