After running for more than 24 hours, errored out with pool.py

[publicst]

Description

After running following command over 36 hours, scan code removed temporary files, then it crashed without generating report.
F:\WS\FG1.5-venv>scancode -lpceiu -n 4 --csv FG15-Int.csv F:\WS\FG1.5-Int-Installer --license-text

...
[#-------------------] 107580 Scanned: Syncfusion.Grid.Grouping.Windows.dll
Removing temporary files...done.
Traceback (most recent call last):
File "C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 720, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 726, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "f:\ws\FG1.5-venv\Scripts\scancode.exe_main.py", line 9, in
File "f:\ws\fg1.5-venv\lib\site-packages\click\core.py", line 722, in call
return self.main(*args, **kwargs)
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\utils.py", line 70, in main
standalone_mode=standalone_mode, **extra)
File "f:\ws\fg1.5-venv\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "f:\ws\fg1.5-venv\lib\site-packages\click\core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "f:\ws\fg1.5-venv\lib\site-packages\click\core.py", line 535, in invoke
return callback(*args, **kwargs)
File "f:\ws\fg1.5-venv\lib\site-packages\click\decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\cli.py", line 513, in scancode
*args, **kwargs)
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\cli.py", line 903, in run_scan
quiet=quiet, verbose=verbose, kwargs=kwargs, echo_func=echo_func,
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\cli.py", line 1093, in run_scanners
with_timing=timing, progress_manager=progress_manager)
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\cli.py", line 1187, in scan_codebase
location, rid, scan_errors, scan_time, scan_result, scan_timings = next(scans)
File "f:\ws\fg1.5-venv\lib\site-packages\click_termui_impl.py", line 259, in next
rv = next(self.iter)
File "f:\ws\fg1.5-venv\lib\site-packages\scancode\pool.py", line 48, in wrap
return func(self, timeout=timeout or 3600)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 730, in next
raise TimeoutError
multiprocessing.context.TimeoutError

How To Reproduce

Running scan code with huge code base with following command
F:\WS\FG1.5-venv>scancode -lpceiu -n 4 --csv FG15-Int.csv F:\WS\FG1.5-Int-Installer --license-text

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? (Windows/MacOS/Linux) Windows
• What version of scancode-toolkit was used to generate the scan file? 3.1.1
• What installation method was used to install/run scancode? (pip/source download/other) pip

[pombredanne]

Thank you and sorry for the late reply. Somehow it may be the same or related to this other bug #2106

[publicst]

It may be related but I am only running with 4-processes instead of 100 like the other person's issue.
I am running it under Intel Core i7-5930K CPU @ 3.5 GHz with 32 GB.

Under what kind of system is this program expected to run and complete?

[pombredanne]

@publicst you wrote:

Under what kind of system is this program expected to run and complete?

I am sorry you are getting issues!
It routinely runs on machines with a few GB or RAM. It must be a bug with Windows. Can you try the latest version with Python on Windows 64 instead?
It could be a 32 vs. 64 bits thing

[raratiru]

What would be a "best practice" for scanning a virtualenv? In my case I initiated a full scan in a low end machine and bumped on this error in Debian Bullseye - Python-3.9.9, Intel I5-2400@3.10GHz, 8GB Ram :

After 12 hours: (Of course scancode, scans itself, would that be an issue?)

$ pwd
/home/user/project

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "*"
argon2-cffi = "*"
psycopg2-binary = "*"
diskcache = "*"
sentry-sdk = "*"
gunicorn = "*"
uvicorn = "*"
yalm = "*"

[dev-packages]
ipdb = "*"
pdbpp = "*"
pytest-django = "*"
pytest-cov = "*"
pytest-factoryboy = "*"
django-debug-toolbar = "*"
pytest-splinter = "*"
pytest-bdd = "*"
pylint = "*"
flake8 = "*"
bandit = "*"
black = "==21.12b0"
safety = "*"
scancode-toolkit = "*"

[requires]
python_version = "3.9"

$ scancode -clpeui -n 2 --json-pp sample.json ./
...
Removing temporary files...done.
Traceback (most recent call last):
  File "/home/user/project/.venv/bin/scancode", line 8, in <module>
    sys.exit(scancode())
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/project/.venv/lib/python3.9/site-packages/commoncode/cliutils.py", line 69, in main
    return click.Command.main(
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user/project/.venv/lib/python3.9/site-packages/scancode/cli.py", line 451, in scancode
    success, _results = run_scan(
  File "/home/user/project/.venv/lib/python3.9/site-packages/scancode/cli.py", line 887, in run_scan
    scan_success = run_scanners(
  File "/home/user/project/.venv/lib/python3.9/site-packages/scancode/cli.py", line 1125, in run_scanners
    scan_success = scan_codebase(
  File "/home/user/project/.venv/lib/python3.9/site-packages/scancode/cli.py", line 1236, in scan_codebase
    scan_timings) = next(scans)
  File "/home/user/project/.venv/lib/python3.9/site-packages/click/_termui_impl.py", line 116, in __next__
    return next(iter(self))
  File "/home/user/project/.venv/lib/python3.9/site-packages/commoncode/cliutils.py", line 172, in generator
    for rv in self.iter:
  File "/home/user/project/.venv/lib/python3.9/site-packages/scancode/pool.py", line 52, in wrap
    return func(self, timeout=timeout or 3600)
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 865, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError

[pombredanne]

After 12 hours: (Of course scancode, scans itself, would that be an issue?)

yes, this would likely be a significant issue, as there are some many licenses in there that it would take forever and exhaust all resources.

BTW, I do not think it is the same issue as the one experienced here before.

In your case, you could --ignore the paths where ScanCode modules are installed, but there are several top level ones in a typical site-packages.

That could be a short term work around... but in the end there are possibly a few better ways:

1. install ScanCode separately and elsewhere as an app and not as a wheel, such that it is not living inside your current venv. I am toying with using AppImage which would make it much simpler as it would have its own bundled Python and would be a single file.

2. have scancode be self-aware, such that it detects code of itself to skip itself (which could be extended to be aware of a few other license-related tools, though most are handled OK by ScanCode these days with the latest develop branch)

Alternatively in the short term you could also run with only --package if what you care for are only top-level package data?
Help would be much appreciate in any case!

[raratiru]

@pombredanne

BTW, I do not think it is the same issue as the one experienced here before.

Indeed.

I followed the idea of a separate virtual environment for ScanCode, ran it with -cl --package and got a firm "first-time result" I can work with.

Thank you!