Repository navigation
Nested storage detection in Zarr V2 #707
Description
see: ome/ngff#29 and bcdev/jzarr#17
In order to better handle Zarr arrays created with NestedDirectoryStorage or FSStore(key_separator="/"), SabineEmbacher and I have been working on a "protocol heuristic" that can be used by V2 implementations to detect nested chunking rather than requiring the user to specify it correctly.
tl;dr: This proposes a new key for .zarray which it would be good to have feedback on.
Proposal
When creating a zarr array:
- add to .zarray json:
{"dimension_separator": "/"} - always write a 0-position chunk
When opening an array:
- try to read the separator character from the .zarray json
- if not available, try to find a 0-position chunk
- if not available, at every read action try to find chunks with both standard variants until the situation is clarified. (Standard separator list
["/", "."])
Points for discussion:
- The name
dimension_separatordiffers from the code implementationkey_separatorto reduce confusion about whether every separator in the key name is effected. - There has been some discussion (community call, gitter) about whether or not "/" could become the default.
- Is an addition to the .zarray metadata sufficiently low impact to be rolled into v2?
Reacted by Constantin Pape and pmav99
Activity
SabineEmbacher commentedon Mar 7, 2021
Thoughts on the name of the separator:
What the separator separates is the respective chunk index per dimension.
See bcdev/jzarr#19 and bcdev/jzarr#17 (comment) and
Therefore the following suggestions:
jbms commentedon Mar 15, 2021
One issue with the detection protocol is that it does not work when writing. While auto-detection is nice it might be better to just assume "." unless the metadata key is present --- any existing arrays could be fixed up, and hopefully all future arrays would have the metadata key. That avoids the added complexity (and inefficiency) of auto-detection.
joshmoore commentedon Mar 16, 2021
Hi @jbms. You mean a
mode="a"-style writing? If so, I see the problem. I had initially intended the auto-detection only as a means of not needing to "standardize" a new key. Then I got greedy and went for both. Anyone else have thoughts?In the case of a
mode="w"situation I would assume the writer simply asserts where it will write.Guess there are some edge cases I worry about here. Maybe there are a number of strategies that can be enabled:
The real question is likely to be what should be the default.
joshmoore commentedon Mar 26, 2021
@SabineEmbacher, @axtimwalde suggests "dimension separator".
axtimwalde commentedon Mar 30, 2021
I used
dimensionSeparatorhttps://github.com/saalfeldlab/n5-zarr/blob/master/src/main/java/org/janelia/saalfeldlab/n5/zarr/N5ZarrReader.java#L89 or https://github.com/saalfeldlab/n5-universe/blob/main/src/main/java/org/janelia/saalfeldlab/n5/universe/N5Factory.java#L109 but the other choices sound reasonable too.SabineEmbacher commentedon Apr 6, 2021
I did not make my suggestions regarding the name in order to enforce them.
They were simply suggestions.
I agree with any decision. Even if a decision does not correspond to my personal preferences.
That is perfectly fine with me.
Please make a decision and I will put it in asap according to the specification.
joshmoore commentedon Apr 8, 2021
No worries, @SabineEmbacher. The suggestions were definitely useful. It's more just a matter of https://martinfowler.com/bliki/TwoHardThings.html ...
Since
key_separatorleads to confusion anddimensionSeparator/dimension_separatoris at least used in some other implementation, I've updated the description withdimension_separator.On the community call last night, there were no objections to moving forward with the .zarray addition, so I'll open a v2 spec PR now.
I'm a bit more hesitant about defining the heuristic as part of the specification (cf. @jbms comment) I'll leave this open for a discussion of where first-chunk writing falls on the MAY/SHOULD/MUST spectrum.
v2 spec: add optional dimension_separator (see zarr-developers#707)
v2 spec: add optional dimension_separator (see zarr-developers#707)
v2 spec: add optional dimension_separator (see #707) (#715)
joshmoore commentedon Sep 22, 2021
I consider the
dimension_separatorsaga complete! whew