A process in the process pool was terminated abruptly

[bwjohnson-ss]

When running scancode.io locally on my mac, I get this odd error when trying to run a scan_codebase pipeline. It seems it fails, but doesn't provide enough info on why. Here's the information it provides:

2022-09-09 21:52:44.04 Pipeline [scan_codebase] starting
2022-09-09 21:52:44.15 Step [copy_inputs_to_codebase_directory] starting
2022-09-09 21:52:44.24 Step [copy_inputs_to_codebase_directory] completed in 0.09 seconds
2022-09-09 21:52:44.25 Step [extract_archives] starting
2022-09-09 21:52:45.00 Step [extract_archives] completed in 0.76 seconds
2022-09-09 21:52:45.01 Step [collect_and_create_codebase_resources] starting
2022-09-09 21:52:49.75 Step [collect_and_create_codebase_resources] completed in 4.74 seconds
2022-09-09 21:52:49.75 Step [tag_empty_files] starting
2022-09-09 21:52:49.81 Step [tag_empty_files] completed in 0.06 seconds
2022-09-09 21:52:49.81 Step [scan_for_application_packages] starting
2022-09-09 21:53:17.12 Step [scan_for_application_packages] completed in 27.30 seconds
2022-09-09 21:53:17.13 Step [scan_for_files] starting
2022-09-09 21:53:20.83 Pipeline failed
Task output
A process in the process pool was terminated abruptly while the future was running or pending.

Traceback:
  File "/app/scanpipe/pipelines/__init__.py", line 115, in execute
    step(self)
  File "/app/scanpipe/pipelines/scan_codebase.py", line 99, in scan_for_files
    scancode.scan_for_files(self.project)
  File "/app/scanpipe/pipes/scancode.py", line 310, in scan_for_files
    _scan_and_save(resource_qs, scan_file, save_scan_file_results)
  File "/app/scanpipe/pipes/scancode.py", line 297, in _scan_and_save
    scan_results, scan_errors = future.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception

Mac Specs:

Docker Desktop version:

Repro Steps:

1. git clone https://github.com/nexB/scancode.io.git && cd scancode.io

• 815411a

2. make envfile
3. docker-compose build
4. docker-compuse up
5. navigate to localhost
6. add a new project
7. upload source (https://github.com/bcicen/ctop) and start scan_codebase pipeline
8. it fails with error

Here are the logs from docker:
docker-logs.txt

[tdruez]

Hi @bwjohnson-ss this is likely an issue with the multiprocessing on macOS (fork() vs spawn())
Can you try to turn off the multiprocessing by adding SCANCODEIO_PROCESSES=-1 to your scancode.io/.env and scancode.io/docker.env files.
You can restart from 3. docker-compose build after that.

[bwjohnson-ss]

That seems to have fixed my issue. Thanks! So for this specific issue, is there something that can be done to the bootstrapping/configure process that would make it so that the environment support for SCANCODEIO_PROCESSES could be auto-detected and configured?

[tdruez]

@bwjohnson-ss yes we should set the SCANCODEIO_PROCESSES=-1 during the make env when ran on a Mac.

[bwjohnson-ss]

Do you mean the envfile target? Or is there one called env?

[tdruez]

@bwjohnson-ss thanks for the patch, this was the approach I was originally suggesting.
But rethinking this, I've actually enforced this behavior directly in the multiprocessing code to disabled multiprocessing and threading on all OS that do not use "fork" as the default multiprocessing start method.
See f73d913
The setting is not needed anymore.