Scancode failure on Windows and Linux

[Cypoman]

Windows:

C:\Users\slorber\Desktop\scancode-toolkit-1.6.1>scancode --format html-sap "z:\Documents\EBI4_Build_Local" EBI4_Build_Local.html
Scanning files...
[------------------------------------] 1 Scanning: com.extol.ebi
Scanning done.
Traceback (most recent call last):
File "C:\Users\slorber\Desktop\scancode-toolkit-1.6.1\bin\scancode-script.py", line 9, in load_entry_point('scancode-toolkit==1.6.1', 'console_scripts', 'scancode')()
File "C:\Users\slorber\Desktop\scancode-toolkit-1.6.1\lib\site-packages\click\core.py", line 664, in __call__return self.main(_args, *_kwargs)
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\scancode\utils.py", line 61, in main standalone_mode=standalone_mode, *_extra)
File "C:\Users\slorber\Desktop\scancode-toolkit-1.6.1\lib\site-packages\click\core.py", line 644, in main rv = self.invoke(ctx)
File "C:\Users\slorber\Desktop\scancode-toolkit-1.6.1\lib\site-packages\click\core.py", line 837, in invoke return ctx.invoke(self.callback, *_ctx.params)
File "C:\Users\slorber\Desktop\scancode-toolkit-1.6.1\lib\site-packages\click\core.py", line 464, in invoke return callback(_args, *_kwargs)
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\scancode\cli.py", line 248, in scancode results = scan(input, copyright, license, package, email, url, info, verbose, quiet)
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\scancode\cli.py", line 315, in scan
scan_result.update(scan_one(res, scanners))
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\scancode\cli.py", line 334, in scan_one
scan = list(scan)
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\scancode\api.py", line 56, in get_copyrights from cluecode.copyrights import detect_copyrights
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\cluecode\copyrights.py", line 36, in from textcode import analysis
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\textcode\analysis.py", line 31, in import typecode.contenttype
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\typecode\contenttype.py", line 48, in from typecode import magic2
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\typecode\magic2.py", line 208, in libmagic = load_lib()
File "c:\users\slorber\desktop\scancode-toolkit-1.6.1\src\typecode\magic2.py", line 192, in load_lib root_dir = command.get_base_dirs(typecode.bin_dir)[0]
IndexError: list index out of range

C:\Users\slorber\Desktop\scancode-toolkit-1.6.1>

Ubuntu:

./scancode: /home/slorber/Desktop/scancode-toolkit-1.6.1/bin/scancode: /Users/slorber/Desktop/scancode-toolkit-1.6.1/bin/python: bad interpreter: No such file or directory
slorber@ubuntu:~/Desktop/scancode-toolkit-1.6.1$ python --version
Python 2.7.6

slorber@ubuntu:~/Desktop/scancode-toolkit-1.6.1$

Centos

[e-team@localhost scancode-toolkit-1.6.1]$ ./scancode --help
./scancode: /home/e-team/Desktop/scancode-toolkit-1.6.1/bin/scancode: /Users/slorber/Desktop/scancode-toolkit-1.6.1/bin/python: bad interpreter: No such file or directory
[e-team@localhost scancode-toolkit-1.6.1]$ ./configure

• Installing components ...
  /bin/sh: /home/e-team/Desktop/scancode-toolkit-1.6.1/bin/pip: /Users/slorber/Desktop/scancode-toolkit-1.6.1/bin/python: bad interpreter: No such file or directory

Failed to execute command:
pip install --no-allow-external --use-wheel --no-index --find-links=/home/e-team/Desktop/scancode-toolkit-1.6.1/thirdparty/prod --find-links=/home/e-team/Desktop/scancode-toolkit-1.6.1/thirdparty/base -r /home/e-team/Desktop/scancode-toolkit-1.6.1/etc/conf/base.txt. Aborting...
[e-team@localhost scancode-toolkit-1.6.1]$

[pombredanne]

@Cypoman Thank you for this report: I am kind of puzzled why you have so many issues though!

1. On Windows, did you install Python 2.7.x for Windows 32? or another version, possibly for Windows 64? Which version of Windows do you run? The windows error should be made more explicit but likely means This is a Windows 64bits Python and I cannot do much with it
2. On Ubuntu, which version do you run and is this 32 bits or 64 bits? is this on Intel? Did you use a fresh installation from a release archive of https://github.com/nexB/scancode-toolkit/releases/tag/v1.6.0?
   It looks like you may have used an experimental release 1.6.1 rather than the stable 1.6.0
3. On Centos, which version do you run and is this 32 bits or 64 bits? is this on Intel? Do you have Python 2.7 available there as the default Python?

[Cypoman]

Philippe,

My problem (at least on Windows) was running the 64 bit version of Python.

I installed the 32-bit version and it's running now.

One question. I have about 100,000 files I'm scanning. It's running on a VM / SSD with 4 cores and it's running at about 25% and using less than 2GB of RAM.

It's been running for almost 9 hours and has done about 1/2 of our code base for one branch. Is this normal?

I am using the verbose option as it's my first time running the tool and I wanted to get an idea of throughput.

Thank you,
Stuart

Stuart Lorber | Release Engineer, Senior Software Engineer II
Cleo | Office: +1.570.621.7883 | slorber@extol.commailto:slorber@extol.com | www.cleo.comhttp://www.cleo.com/
Embrace the power of Move View Act(tm) integration.

From: Philippe Ombredanne <notifications@github.commailto:notifications@github.com>
Reply-To: nexB/scancode-toolkit <reply@reply.github.commailto:reply@reply.github.com>
Date: Monday, July 18, 2016 at 10:50 AM
To: nexB/scancode-toolkit <scancode-toolkit@noreply.github.commailto:scancode-toolkit@noreply.github.com>
Cc: Stuart Lorber <slorber@cleo.commailto:slorber@cleo.com>, Mention <mention@noreply.github.commailto:mention@noreply.github.com>
Subject: Re: [nexB/scancode-toolkit] Scancode failure on Windows and Linux (#279)

@Cypomanhttps://github.com/Cypoman Thank you for this report: I am kind of puzzled why you have so many issues though!

1. On Windows, did you install Python 2.7.x for Windows 32? or another version, possibly for Windows 64? Which version of Windows do you run? The windows error should be made more explicit but likely means This is a Windows 64bits Python and I cannot do much with it
2. On Ubuntu, which version do you run and is this 32 bits or 64 bits? is this on Intel? Did you use a fresh installation from a release archive of https://github.com/nexB/scancode-toolkit/releases/tag/v1.6.0?
   It looks like you may have used an experimental release 1.6.1 rather than the stable 1.6.0
3. On Centos, which version do you run and is this 32 bits or 64 bits? is this on Intel? Do you have Python 2.7 available there as the default Python?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/279#issuecomment-233351567, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSvR9W1J9JMJD0peDJHHyLPy1hULhf8ks5qW5KugaJpZM4JOxKk.

[pombredanne]

It's been running for almost 9 hours and has done about 1/2 of our code base for one branch. Is this normal?

The current release is single threaded and handles each file sequentially. There are also some areas that are rather hungry in memory which explains the 2GB usage. There is branch where we have an experimental multi processing scans https://github.com/nexB/scancode-toolkit/tree/219-multiprocessing which should land in the V2 release soon. It needs a little refinement first such that per-file scans are streamed to a file rather than kept in RAM.
About memory usage, the branch for approximate license detection that is due for merging in devel this week contains several optimizations that should speed things up and lower the memory footprint
See https://github.com/nexB/scancode-toolkit/tree/86-approximate-license-detection

Another approach could be to run multiple smaller scans in the meantime but this is essentially what the multiprocessing is about.
Note also that Windows is always slower tan Linux. I noticed this on a dual boot machine when running test suites on both: not sure why but this something to keep in mind. Might be due to Windows and Python more than ScanCode

[pombredanne]

I am using the verbose option as it's my first time running the tool and I wanted to get an idea of throughput.

This is a good idea indeed as there is no logging to a log file implemented yet!

[pombredanne]

One question. I have about 100,000 files I'm scanning. It's running on a VM / SSD with 4 cores and it's running at about 25% and using less than 2GB of RAM.

Note that 100 000 files is a significant codebase in many respect in anycase: for reference a Linux Kernel would be between 50 to 70K files ;)
My advice for now would be just to be patient. if 9 hours got you 1/2 18 hours is likely to get you to the end.

Which output format did you select btw?

[Cypoman]

I chose html.

Stuart Lorber | Release Engineer, Senior Software Engineer II
Cleo | Office: +1.570.621.7883 | slorber@extol.commailto:slorber@extol.com | www.cleo.comhttp://www.cleo.com/
Embrace the power of Move View Act(tm) integration.

From: Philippe Ombredanne <notifications@github.commailto:notifications@github.com>
Reply-To: nexB/scancode-toolkit <reply@reply.github.commailto:reply@reply.github.com>
Date: Wednesday, July 20, 2016 at 5:08 PM
To: nexB/scancode-toolkit <scancode-toolkit@noreply.github.commailto:scancode-toolkit@noreply.github.com>
Cc: Stuart Lorber <slorber@cleo.commailto:slorber@cleo.com>, Mention <mention@noreply.github.commailto:mention@noreply.github.com>
Subject: Re: [nexB/scancode-toolkit] Scancode failure on Windows and Linux (#279)

One question. I have about 100,000 files I'm scanning. It's running on a VM / SSD with 4 cores and it's running at about 25% and using less than 2GB of RAM.

Note that 100 000 files is a significant codebase in many respect in anycase: for reference a Linux Kernel would be between 50 to 70K files ;)
My advice for now would be just to be patient. if 9 hours got you 1/2 18 hours is likely to get you to the end.

Which output format did you select btw?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/279#issuecomment-234083802, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSvR2C6gPO9eHajJ33Bji0UfeCkPlLbks5qXo5cgaJpZM4JOxKk.

[pombredanne]

ok. This will create a fairly large data.json data file in the directory named _files if output.html was your selected file name. It might load OK in a browser but you will need to ensure you on a desktop will plenty of RAM. I hope this will not crash the browser. Please keep me abreast as there could be ways to possibly split the data.json in smaller chunks after the fact

[Cypoman]

I got a trial of dejacode that I want to start looking at tomorrow - assuming this finishes.

So I planned to import it into there.

Stuart Lorber | Release Engineer, Senior Software Engineer II
Cleo | Office: +1.570.621.7883 | slorber@extol.commailto:slorber@extol.com | www.cleo.comhttp://www.cleo.com/
Embrace the power of Move View Act(tm) integration.

From: Philippe Ombredanne <notifications@github.commailto:notifications@github.com>
Reply-To: nexB/scancode-toolkit <reply@reply.github.commailto:reply@reply.github.com>
Date: Wednesday, July 20, 2016 at 5:13 PM
To: nexB/scancode-toolkit <scancode-toolkit@noreply.github.commailto:scancode-toolkit@noreply.github.com>
Cc: Stuart Lorber <slorber@cleo.commailto:slorber@cleo.com>, Mention <mention@noreply.github.commailto:mention@noreply.github.com>
Subject: Re: [nexB/scancode-toolkit] Scancode failure on Windows and Linux (#279)

ok. This will create a fairly large data.json data file in the directory named _files if output.html was your selected file name. It might load OK in a browser but you will need to ensure you on a desktop will plenty of RAM. I hope this will not crash the browser. Please keep me abreast as there could be ways to possibly split the data.json in smaller chunks after the fact

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/279#issuecomment-234085072, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSvR73wxaFAQ1Z2O5O8HGZxUd5qtUdPks5qXo96gaJpZM4JOxKk.

[pombredanne]

@Cypoman did your scan eventually finish?

[Cypoman]

After 24 hours it didn't so I killed it.

I tried to run it on a subset but it failed on a large jar with an out of memory error so the overall scan would never have completed anyway.

I then ran it on a tiny subset of code to see the html output of ScanCode.

I requested a DejaCode trial but evidently it resides on your servers and I would have to upload my code base - I'm not sure I can do that.

Stuart Lorber | Release Engineer, Senior Software Engineer II
Cleo | Office: +1.570.621.7883 | slorber@extol.commailto:slorber@extol.com | www.cleo.comhttp://www.cleo.com/
Embrace the power of Move View Act(tm) integration.

From: Philippe Ombredanne <notifications@github.commailto:notifications@github.com>
Reply-To: nexB/scancode-toolkit <reply@reply.github.commailto:reply@reply.github.com>
Date: Thursday, July 21, 2016 at 3:39 PM
To: nexB/scancode-toolkit <scancode-toolkit@noreply.github.commailto:scancode-toolkit@noreply.github.com>
Cc: Stuart Lorber <slorber@cleo.commailto:slorber@cleo.com>, Mention <mention@noreply.github.commailto:mention@noreply.github.com>
Subject: Re: [nexB/scancode-toolkit] Scancode failure on Windows and Linux (#279)

@Cypomanhttps://github.com/Cypoman did your scan eventually finish?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/279#issuecomment-234360854, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSvR2AAaPVhIVMLadksiJw1Dfgn6Bkbks5qX8ragaJpZM4JOxKk.

[Cypoman]

I'm assuming that DejaCode will not run without the output from ScanCode?

Stuart Lorber | Release Engineer, Senior Software Engineer II
Cleo | Office: +1.570.621.7883 | slorber@extol.commailto:slorber@extol.com | www.cleo.comhttp://www.cleo.com/
Embrace the power of Move View Act(tm) integration.

From: Philippe Ombredanne <notifications@github.commailto:notifications@github.com>
Reply-To: nexB/scancode-toolkit <reply@reply.github.commailto:reply@reply.github.com>
Date: Thursday, July 21, 2016 at 3:39 PM
To: nexB/scancode-toolkit <scancode-toolkit@noreply.github.commailto:scancode-toolkit@noreply.github.com>
Cc: Stuart Lorber <slorber@cleo.commailto:slorber@cleo.com>, Mention <mention@noreply.github.commailto:mention@noreply.github.com>
Subject: Re: [nexB/scancode-toolkit] Scancode failure on Windows and Linux (#279)

@Cypomanhttps://github.com/Cypoman did your scan eventually finish?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/279#issuecomment-234360854, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJSvR2AAaPVhIVMLadksiJw1Dfgn6Bkbks5qX8ragaJpZM4JOxKk.

[pombredanne]

@Cypoman There is new code that I pushed in #354 in a branch to fix fix your issue of RAM exhaustion. It also addresses other issues such as #267 that should fix your issue of RAM exhaustion. I am closing this issue in favor of #267