Repository navigation
Accented character in filename prevents payload verification on macOS #140
Description
Activity
I've added a test here, but TBH I'm having a lot of trouble getting this code to build with modern versions of Java — I had to update to a later version of Gradle 4 and comment out pretty much all the code quality plugins and tasks — so I'm reluctant to try to put together a full pull request.
@dmoles have you tried using https://github.com/jscancella/bagging ? I created it because it seems like LoC is no longer maintaining this repo.
@jscancella Thanks for that! Unfortunately I'm not using
bagit-javadirectly but via the Fedora Import/Export Utility.Trying your library, it looks like it has the same issue. I'll see if I can create a PR with a test case for you.
It looks like this may be a long-standing Java / macOS issue to do with how HFS+ does Unicode normalization.
If
toString()conversion seems too risky, an alternative would be to compare the paths withFiles.isSameFile(Path, Path), which does seem to admit they're the same.FWIW, I now think the correct fix is to normalize the paths before comparison, probably via canonical decomposition (
java.text.Normalizer.Form.NFD).- added a commit that references this issue
on Jan 10, 2023 FWIW, I now think the correct fix is to normalize the paths before comparison, probably via canonical decomposition (
java.text.Normalizer.Form.NFD).Normalization is definitely the way to handle comparisons but testing is a challenge for BagIt since the character names are stored in a manifest file with a known encoding process but are also subject to what tools and filesystems do for normalization (e.g. Git's core.precomposeunicode setting).
I'm almost certain that instinct is correct (IIRC some ZIP implementations normalize) so I'm wondering whether this needs to be created on checkout or if we want to have the Java codebase generate those names (possibly mocked?) in its test suite.
Reacted by David MolesJust wondering if this is an issue and also more generally following on to this comment whether this library is still being maintained. It appears that the CircleCi builds use Java 8 so I'm wondering if there was a plan to do testing against Java 11 and 17? If this does not work against either, are there cycles to get that work completed?
Environment
bagit-javaversion: v5.1.1Details
Given
When
Then
PayloadVerifiershould not raiseFileNotInManifestExceptionDiscussion
The underlying issue appears to be that
sun.nio.fs.UnixPathcompares paths based on their internal byte representation rather than their string representation, and that somehow, the internal byte representation of thePathproduced byPayloadVerifier.getAllFilesListedInManifests()differs from that in the manifest.I created a file with the name
contrôle.txt, where as near as I can tell, theôis represented as the single Unicode codepoint0xf4. Certainly that's what it is in the manifest, where if I open the file in a hex editor I can clearly see that it's correctly represented as the two-byte UTF-8 sequence0xc3 0xb4.However, when
getAllFilesListedInManifests()creates aPathobject, it somehow ends up representing it as a plaino(0x6f) plus a combining circumflex accent (codepointx0302), so the internal byte representation has the UTF-8 bytes0x6f 0xcc 0x82, the comparison fails, and the file appears not to be in the manifest.I'm not sure at just what point the conversion from
ôtoo-plus-combining^happens -- probably some kind of Unicode normalization that one is going through and the other isn'tHowever, oddly, when I call
toString(), the string representations of the paths are equal. So one workaround would be to convert thePathobjects toStringsbefore making the comparison.