ScanCode.io failure on systemd-252.tar.gz

[DennisClark]

I created a new project called systemdv252 to scan systemd-252.tar.gz which is available from
https://github.com/systemd/systemd/archive/refs/tags/v252.tar.gz

The ScanCode.io screen simply shows a red "Failure" indicator and I do not see any additional information.

[tdruez]

After investigation ScanCode-toolkit is failing (timeout) on this file systemd-252/test/fuzz/fuzz-unit-file/oss-fuzz-8827
oss-fuzz-8827.zip

On the ScanCode.io, we should improve the reporting on those type of failure (hard to catch as long as we use subprocess in the scan_package pipeline)

A proper fix is also required on the toolkit side, it's a made up fuzz text for testing and it's rather small, it should not reach the 1h timeout during scanning.

---

Traceback (most recent call last):
  File "/usr/local/bin/scancode", line 8, in <module>
    sys.exit(scancode())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/commoncode/cliutils.py", line 70, in main
    return click.Command.main(
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/scancode/cli.py", line 452, in scancode
    success, _results = run_scan(
  File "/usr/local/lib/python3.9/site-packages/scancode/cli.py", line 905, in run_scan
    scan_success = run_scanners(
  File "/usr/local/lib/python3.9/site-packages/scancode/cli.py", line 1144, in run_scanners
  File "/usr/local/lib/python3.9/site-packages/scancode/cli.py", line 1252, in scan_codebase
    scan_timings) = next(scans)
  File "/usr/local/lib/python3.9/site-packages/click/_termui_impl.py", line 116, in __next__
    return next(iter(self))
  File "/usr/local/lib/python3.9/site-packages/click/_termui_impl.py", line 328, in generator
    for rv in self.iter:
  File "/usr/local/lib/python3.9/site-packages/scancode/pool.py", line 52, in wrap
    return func(self, timeout=timeout or 3600)
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 865, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError

[tdruez]

The underlying issue is that the ScanPackage pipeline depends on a subprocess call to the scancode command.
If scancode fails, there's no easy way to save this as a ProjectError and continue the pipeline run.

A better approach would be to replace this approach by proper api calls, as we did for the Scanners in the ScanCodebase pipeline (For example: get_copyrights, get_licenses, get_package_data)

The 2 missing pieces as callable API for the ScanPackage pipeline are:

• --classify: FileClassifier(PreScanPlugin)
• --summary: ScanSummary(PostScanPlugin)
• --is-license-text: IsLicenseText(PostScanPlugin)

[tdruez]

With the changes from #798 , the scan_package pipeline on https://github.com/systemd/systemd/archive/refs/tags/v252.tar.gz now completes successfully and reports the file timeout as a Project message: