SPDX output fails for scanned files with weird names

[sschuberth]

I'm scanning a file which has a colon (:) in its name, and when writing scan results to SPDX-TV, I get

  File "/home/jenkins/jobs/scancode/workspace/tools/scancode/src/scancode/cli.py", line 684, in save_results
    doc.package.verif_code = doc.package.calc_verif_code()
  File "/home/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/spdx/package.py", line 195, in calc_verif_code
    hashes.append(file_entry.calc_chksum())
  File "/home/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/spdx/file.py", line 151, in calc_chksum
    with open(self.name, 'rb') as file_handle:
IOError: [Errno 2] No such file or directory: u'data/response_24%3A00.xml'

I know @pombredanne has recently done an awesome work to fix issues with weird file names like this, and I know the colon is not supported on file name on e.g. Windows. However, I'm running the scan on Linux. Still, the file name seems to have the colon escaped as %3A.

My question to @pombredanne basically is: Is this on purpose, and am I supposed to unescape the file names in scanned_files before using them?

[pombredanne]

@sschuberth yes, escaping was done on purpose ass somehow files with non-ascii characters in their names were making the scan crash. So for now if would you need to unescape.

But beside that, the escaping should not be needed in the first place IMHO.

[pombredanne]

what I find weird is why would the spdx library need to recompute hashes that have been provided alright in the first place?

[sschuberth]

Well, I agree that there should be no need to use the (escaped) path from ScanCode's scanned_files to recompute the hash in this particular case, and I'm thinking about a fix as we speak. However, in the general case, if for some reason you need to open a file from a path as stored in scanned_files, it's good to know that you need to escape them.

[sschuberth]

spdx/tools-python#12 works around the issue by not using the path to recalculate ethe SHA1 at all. Once that is merged (and the packages is published), I'll change ScanCode to pass the pre-calculated SHA1 to the tool-python's file class.

[pombredanne]

I pushed the updated SDPX tools to scancode develop.

[sschuberth]

Perfect, I just saw that, great teamwork! :-)