Question: dealing with timeouts during the scan

[nnobelis]

I run Scancode within ORT on a big repository (+ 3.5 GB) containing .json test data (a lot a big files).

Scan crashes with a timeout:

Running '/opt/python/shims/scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 7 /tmp/ort-DefaultProvenanceDownloader6273558335820335397 --json-pp /tmp/ort-ScanCode9471182827864591694/result.json' in '/home/ort'...

/opt/python/versions/3.10.6/lib/python3.10/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
  File "/opt/python/versions/3.10.6/bin/scancode", line 8, in <module>
    sys.exit(scancode())
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 69, in main
    return click.Command.main(
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 451, in scancode
    success, _results = run_scan(
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 887, in run_scan
    scan_success = run_scanners(
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1125, in run_scanners
    scan_success = scan_codebase(
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1236, in scan_codebase
    scan_timings) = next(scans)
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/_termui_impl.py", line 116, in __next__
    return next(iter(self))
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 170, in generator
    yield from self.iter
  File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/pool.py", line 52, in wrap
    return func(self, timeout=timeout or 3600)
  File "/opt/python/versions/3.10.6/lib/python3.10/multiprocessing/pool.py", line 868, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError

Is this occurring because of the --timeout 300 parameter I specified ?
I know the clean solution would be to use --ignore parameter but is there another way to alleviate this error ?

Also the memory usage goes through the roof so I tried to see if --max-in-memory -1 helps to prevent scancode putting the big files in memory by no luck here.

Any help is welcome.

OS: eclipse-temurin:11-jdk-jammy
Scancode: 30.1.0.

[pombredanne]

@nnobelis Thank for the report. --max-in-memory -1 prevents data to be kept in memory during the scan, but not at the last step when writing the scan results.

We discussed this on Monday... the overall scancode process is roughly this:

• collect the list of files to scan
• for each file run in a separate process the scan and write the results to a small JSON data file (or keep in memory) (think of it as a map in map/reduce)
• once all scans are complete, assemble all the small JSON data files in the output

That last part is the likely problem. We are collating a bunch of related issues to find a proper strategy.

Is this occurring because of the --timeout 300 parameter I specified ?

This is unlikely.

Some questions:

• eclipse-temurin:11-jdk-jammy means you run this in Docker. What is the base image for eclipse-temurin?

• while this is unlikely to fix your issue, I wonder if you could try running the very latest of ScanCode?

• if we patch a branch based 30.1.0 to test something, will be able to rerun this scan easily?

• is this reproducible consistently? (I know it is difficult to rerun big scans like this.)

[nnobelis]

Thanks @pombredanne for your answer.

• ubuntu:22.04 (See https://github.com/adoptium/containers/blob/fa65b46d19f4db0e1c8736c3d7b2a8392dd58b1b/11/jdk/ubuntu/jammy/Dockerfile.releases.full).
• I could try running the latest ScanCode. I will comment on this ticket when it's done.
• Not really :(
  Since I run ORT Docker image, I install ScanCode through PIP. Hence, I can only use the officially released ScanSode versions.
  Or can you release such an experimental branch through PIP ?
• This is happening each run on our customer repository. Unfortunately the latter is proprietary code so I cannot give you a sample.

[oheger-bosch]

Just an update: We encountered the problem again on a different project, with the same stack trace. I tried to reproduce the error locally, but was not able to do so. ScanCode could successfully scan the affected project files on a Windows machine with Python 3.6 and on a Linux box with Python 3.8. It fails reproducible in our pipeline setup which uses Python 3.10. So could this be related to the Python version?

IIUC, the timeout for a single file passed via the --timeout parameter is not effective, and then after 1 hour the job gets killed by the pool.

[pombredanne]

@oheger-bosch re:

So could this be related to the Python version?

that's a promising avenue to investigate! Thank you ++