Repository navigation
Re-evaluate the default normalize_keys for FSStore #739
Description
For nested filesets using FSStore, it's currently a requirement to set normalize_keys to False on construction. @shoyer pointed out in #546 (comment) that False is the default for the other stores.
Activity
martindurant commentedon May 12, 2021
As I said in the linked PR, I'm happy to have the default change for the sake of consistency
jakirkham commentedon May 12, 2021
cc @shoyer
d70-t commentedon May 19, 2021
I just stumbled across this behavior as well.
In order to create datasets on IPFS, I currently first create a dataset locally and then add it into IPFS. This meant that the datasets are written without normalization and read back with normalization.
While I might have to think a bit about how the need for normalization might affect my use case, it definitely surprised me to see this kind of different behavior.
One particular thing which I found interesting is related to consolidated metadata. Within my consolidated metadata, the variables showed up with capital letters and thus, they also showed up with capital letters in the listing of my dataset (via xarray). But when accessing the variable, all values have been read back as
nan, because none of the chunk files could be reached due to path normalization on access. I am wondering if the metadata of a dataset should (could?) tell the reader if reading the dataset requires to access it without normalization, but maybe that's also not the right way of thinking about the issue.joshmoore commentedon May 19, 2021
Opened #755, let's see how the tests do.