scancode does not start on windows exec

[jeffmcaffer]

I exec'ing "c:\install\scancode-toolkit-2.9.2\scancode" from node fails looking for "c:\install\scancode-toolkit-2.9.2\bin\scancode" (note the "bin" segment. Seems that the bin folder is actually a symlink to Scripts. Using normal windows tools (i.e., drag and drop) to unzip the Release zip, you get a windows Junction for bin. That is fine in the UI and from the Command shell but when exec'ing from Node, it seems like the junction is not being followed (or is not available). I don't know enough about the internal workings of node and exec to know why this would be different.

Since the "Scripts" folder seems to include all the executable stuff, can it just called bin rather than using a symlink?

as a workaround right now I just deleted the bin link, and copied Scripts to bin. Not sure if there has to be something called Scripts. (i.e., could I have just renamed?)

@pombredanne FWIW, this is for ClearlyDefined.

[pombredanne]

@jeffmcaffer I do not know if node honours Windows junctions or not: it could be also that node may be less tested on Windows than POSIX

Now the reason why ScanCode generates a junction on first use (e.g. on the initial configure) is that the Python machinery that creates the bin and Scripts (e.g. setuptools) actually does generate a bin on POSIX and a Scripts dir on Windows. As a convenience, having a junction helps have the same paths referenced in help and documentation on every OS: e.g. bin.

These (the Scripts and the bin/junctions) are created either:

• on the first run of a new installation
• when calling the root configure.bat script (and this can be a combo of configure --clean then configure )

Can you point me to where in ClearlyDefined this happens?
Unless there is a bug in node I cannot fathom what happens exactly here.

Was this on the first run? or on other runs?
Was the 2.9.2 of scan a fresh extract in a clean directory? or was it overwritten on top of a previous installation?

[jeffmcaffer]

Thanks @pombredanne. this is all runs of a freshly installed 2.9.2 (was also happening on 2.9.1). This happens in ClearlyDefined if you setup the crawler. If you unzip scancode in c:\installs (default) and run, then queue a request by POSTing something like

curl -d '{"type":"npm", "url":"cd:/npm/npmjs/-/redie/0.3.0"}' -H "Content-Type: application/json" -H "X-token: secret" -X POST http://localhost:5000/requests

That will trigger the crawler to get the npm and its source iteratively and run scancode (and other tools).

See https://github.com/clearlydefined/crawler/blob/master/providers/process/scancode.js#L71 for details of the exec call in node.

Stepping back from the how/why, Is it possible to have the init code just make a copy or rename of Scripts to bin when setting up on Windows? I get that a symlink is easier but it has these problems.

[pombredanne]

Stepping back from the how/why, Is it possible to have the init code just make a copy or rename of Scripts to bin when setting up on Windows? I get that a symlink is easier but it has these problems.

The Python tools expect things on Windows to be under Scripts so using a copy instead of a link is likely to get things broken in many other use cases.

Now one way could be to still have a junction as a convenience but not use it in the root "scancode" script and reference \Scripts on windows and not the bin\junction.

[pombredanne]

@jeffmcaffer one thing that I do not understand is why would any of this matters to node.js?
The use of the bin->Scripts junction is only done by the scancode.bat script. And node exec has pass over control to scancode.bat when this failure occurs IMHO.

if it is failing then this means that the junction was not created in the first place possibly. Do you have a more complete failure/stack trace log of sorts?

[jeffmcaffer]

I don't know either but it just fails trying to run .../bin/scancode.exe. It is not a stack trace per se, just the exec gets an error. Are you able to reproduce? I'll see if anyone in the hackathon today can repo.

[pombredanne]

@jeffmcaffer the latest develop branch and tag 2.9.7 should have this resolved but I do not have an easy to to test this on hand. Can you check if this works now?

[pombredanne]

This has been fixed so I am closing short of other feedback. Please reopen if this is still an issue