Repository navigation
Nested cloud storage #395
Description
Following on from #177, is it possible to also use the NestedDirectoryStorage with s3fs? I'm currently looking at code roughly equivalent to:
store = s3fs.S3Map(
root='example.zarr',
s3=s3, check=True, create=True
)
ds.to_zarr(store=store, mode='w', group=name)
and am missing if there's a parameter and/or a delegation pattern that I could use to go from . to / notation.
see also:
- http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html
- https://github.com/pydata/xarray/blob/6963164dbdc55f86f34b741073838f083a68a152/xarray/core/dataset.py#L1248
- https://github.com/pydata/xarray/blob/ee44478ed921ae3afdece9d16dc179a197727e8d/xarray/backends/api.py#L892
- https://github.com/pydata/xarray/blob/2667deb74a30dc3bd88752a3ce5da590cf7ddd48/xarray/backends/zarr.py#L222
- https://github.com/dask/s3fs/blob/master/s3fs/mapping.py#L8
Reacted by Eric Perlman
Activity
rabernat commentedon Jan 22, 2019
This is a good idea.
In theory, nesting and the storage medium are orthogonal. Nesting has to do with the structure of the keys. We should be able to make any store a nested store.
Just out of curiosity, what's your motivation here? In my understanding, object stores like S3 don't actually have directories...so what would be the advantage of nesting?
jakirkham commentedon Jan 22, 2019
Right the
NestedDirectoryStoremerely reinterprets the.s as nested directories. The keys remain the same as inDirectoryStore.joshmoore commentedon Jan 23, 2019
@jakirkham: primarily
Chunk storage zarr-developers/zarr-python#4from zarr-developers/zarr-specs#3jakirkham commentedon Jan 29, 2019
@joshmoore, I'm afraid I still don't follow. What is gained by using a
.instead of a/given the key/value pair is more or less the same?joshmoore commentedon Jan 29, 2019
@jakirkham: sorry, I'm trying to use
/instead of.to be a step closer to the N5 spec. (In general, I also think it's a sane default since it reduces the number of objects in a directory when working with the data locally, e.g. viaaws s3 sync)alimanfoo commentedon Jan 31, 2019
It should be pretty straightforward to pull out the functionality to transform chunk keys to use
/instead of.into a separate mapping, which you could then layer on top of any store. E.g., you'd then need to do something like:...a bit verbose, but illustrates the idea of composing the mappings.
(...and now I'm wondering if the N5 store being implemented in #309 should be implemented as a transformer layer over any other store, rather than fixed to file system storage. That would allow zarr to access N5 data stored in S3 or GCS, for example, with no extra work.)
joshmoore commentedon Feb 1, 2019
I'll give this a try next week while traveling if no one else has started.See #395 (comment) (2020-02-11)
jakirkham commentedon Mar 5, 2019
FWIW this came up in issue ( https://github.com/zarr-developers/zarr/issues/410 ) as well.
chrisroat commentedon Feb 19, 2020
I'd like to understand what the roadmap is here.
I came across this while assuming everything worked with cloud buckets, but painted myself into a corner in using N5Store (for neuroglancer viz), which does not. My first inclination was that the storage.py layer could be adjusted to replace
oscalls withfsspec-type code. But in reading this and zarr-developers/n5py#9, it looks like something bigger is in the works.How can I help?
joshmoore commentedon Feb 21, 2020
Sorry, this got backburner-ed from my side some time ago. Nothing substantial to report.
joshmoore commentedon Sep 22, 2021
With the
dimension_separatorepic complete (#707), nested cloud storage is now possible with the FSStore implementation. The default value has not been changed, but can be passed on creation. The setting will be stored in the.zarraymetadata.