Migrate the ScanCodebase pipeline from run_scancode subprocess...

[tdruez]

... to the scancode.scan_for_application_packages and scancode.scan_for_files.

The run_scancode does not allow much progressive and customized approach which is the main purpose of ScanCode.io.

[tdruez]

Note that there's no impact on the performance side after this change:

Subprocess: Execution time 5307 seconds (1.5 hours)

2021-08-26 13:12:33.15 Pipeline [scan_codebase] starting
2021-08-26 13:12:33.15 Step [copy_inputs_to_codebase_directory] starting
2021-08-26 13:12:33.37 Step [copy_inputs_to_codebase_directory] completed in 0.22 seconds
2021-08-26 13:12:33.37 Step [run_extractcode] starting
2021-08-26 13:13:07.30 Step [run_extractcode] completed in 33.92 seconds
2021-08-26 13:13:07.30 Step [run_scancode] starting
2021-08-26 14:37:15.18 Step [run_scancode] completed in 5047.87 seconds
2021-08-26 14:37:15.18 Step [build_inventory_from_scan] starting
2021-08-26 14:40:03.28 Step [build_inventory_from_scan] completed in 168.09 seconds
2021-08-26 14:40:03.28 Step [csv_output] starting
2021-08-26 14:41:00.39 Step [csv_output] completed in 57.10 seconds
2021-08-26 14:41:00.40 Pipeline completed

New system: Execution time 5497 seconds (1.5 hours)

2021-10-11 16:58:05.70 Pipeline [scan_codebase] starting
2021-10-11 16:58:05.71 Step [copy_inputs_to_codebase_directory] starting
2021-10-11 16:58:05.86 Step [copy_inputs_to_codebase_directory] completed in 0.15 seconds
2021-10-11 16:58:05.86 Step [extract_archives] starting
2021-10-11 16:58:39.22 Step [extract_archives] completed in 33.36 seconds
2021-10-11 16:58:39.23 Step [collect_and_create_codebase_resources] starting
2021-10-11 17:05:05.21 Step [collect_and_create_codebase_resources] completed in 385.97 seconds
2021-10-11 17:05:05.21 Step [tag_empty_files] starting
2021-10-11 17:05:05.24 Step [tag_empty_files] completed in 0.02 seconds
2021-10-11 17:05:05.24 Step [scan_for_application_packages] starting
2021-10-11 17:05:26.85 Step [scan_for_application_packages] completed in 21.60 seconds
2021-10-11 17:05:26.85 Step [scan_for_files] starting
2021-10-11 18:29:42.75 Step [scan_for_files] completed in 5055.89 seconds
2021-10-11 18:29:42.76 Pipeline completed

On the benefit side, we now have proper status for each codebase resource (not set with the subprocess)

Resources status
application-package: 29
ignored-empty-file: 7
scanned: 50770
scanned-with-error: 19