ScanCode 31.2.1 fail to scan a clusterfuzz testcase file

[sschuberth]

Description

Scanning https://github.com/grpc/grpc/blob/b72da316bb2e718bec19c267e1427628a183ac6b/test/core/transport/chttp2/hpack_parser_corpus/upgraded-clusterfuzz-testcase-5298216461402112 with --copyright --license --info --strip-root -n 4 --timeout 300 --processes 7 results in

Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Scanning done.
Some files failed to scan properly:
Path: ort-DefaultProvenanceDownloader15154544898313057407/test/core/transport/chttp2/hpack_parser_corpus/upgraded-clusterfuzz-testcase-5298216461402112
Summary:        info, licenses, copyrights with 7 process(es)
Errors count:   1
Scan Speed:     5.40 files/sec. 26.08 KB/sec.
Initial counts: 10567 resource(s): 9183 file(s) and 1384 directorie(s) 
Final counts:   10566 resource(s): 9183 file(s) and 1383 directorie(s) for 43.28 MB
Timings:
  scan_start: 2022-12-15T045214.192114
  scan_end:   2022-12-15T052043.558346
  setup_scan:licenses: 3.76s
  setup: 3.76s
  inventory: 3.07s
  scan: 1699.46s
  output:json-pp: 4.42s
  output: 4.42s
  total: 1714.19s
Removing temporary files...done.

and exit code 1.

How To Reproduce

See above.

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Linux
• What version of scancode-toolkit was used to generate the scan file? 31.2.1
• What installation method was used to install/run scancode? pip

[sschuberth]

/cc @borisbaldassari

[AyanSinhaMahapatra]

This is failing just because of the timeout specified:

I could not reproduce at first on my local machine, but a reduced timeout (--timeout 100) was able to reproduce. Showing both scan results here for context:

scancode --copyright --license --info --strip-root -n 4 --timeout 100  upgraded-clusterfuzz-testcase-5298216461402112 --json-pp clusterfuzz.json
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
[####################] 2
Scanning done.
Some files failed to scan properly:
Path: upgraded-clusterfuzz-testcase-5298216461402112
Summary:        info, licenses, copyrights with 4 process(es)
Errors count:   1
Scan Speed:     0.01 files/sec. 6.50 KB/sec.
Initial counts: 1 resource(s): 1 file(s) and 0 directorie(s)
Final counts:   1 resource(s): 1 file(s) and 0 directorie(s) for 672.05 KB
Timings:
  scan_start: 2022-12-15T141551.926515
  scan_end:   2022-12-15T141736.647727
  setup_scan:licenses: 1.32s
  setup: 1.32s
  scan: 103.40s
  total: 104.72s

timeout: 100

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "31.2.1",
      "options": {
        "input": [
          "upgraded-clusterfuzz-testcase-5298216461402112"
        ],
        "--copyright": true,
        "--info": true,
        "--json-pp": "clusterfuzz.json",
        "--license": true,
        "--processes": "4",
        "--strip-root": true,
        "--timeout": "100.0"
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2022-12-15T141551.926515",
      "end_timestamp": "2022-12-15T141736.647727",
      "output_format_version": "2.0.0",
      "duration": 104.72122406959534,
      "message": null,
      "errors": [
        "Path: upgraded-clusterfuzz-testcase-5298216461402112"
      ],
      "warnings": [],
      "extra_data": {
        "system_environment": {
          "operating_system": "linux",
          "cpu_architecture": "64",
          "platform": "Linux-5.14.0-1054-oem-x86_64-with-glibc2.29",
          "platform_version": "#61-Ubuntu SMP Fri Oct 14 13:05:50 UTC 2022",
          "python_version": "3.8.10 (default, Nov 14 2022, 12:59:47) \n[GCC 9.4.0]"
        },
        "spdx_license_list_version": "3.17",
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "upgraded-clusterfuzz-testcase-5298216461402112",
      "type": "file",
      "name": "upgraded-clusterfuzz-testcase-5298216461402112",
      "base_name": "upgraded-clusterfuzz-testcase-5298216461402112",
      "extension": "",
      "size": 688179,
      "date": "2022-12-15",
      "sha1": "4271e5059b58be25d0845678c8b1c3f12c0040c3",
      "md5": "3fe4430ff25b408b7516b09433361050",
      "sha256": "64f26b6a338731cf1ebf7c3c166999150052e4947e405de8c84f88a2b0c1813b",
      "mime_type": "text/plain",
      "file_type": "ASCII text, with very long lines",
      "programming_language": null,
      "is_binary": false,
      "is_text": true,
      "is_archive": false,
      "is_media": false,
      "is_source": false,
      "is_script": false,
      "licenses": [],
      "license_expressions": [],
      "percentage_of_license_text": 0,
      "copyrights": [],
      "holders": [],
      "authors": [],
      "files_count": 0,
      "dirs_count": 0,
      "size_count": 0,
      "scan_errors": [
        "ERROR: for scanner: copyrights:\nERROR: Processing interrupted: timeout after 100 seconds."
      ]
    }
  ]
}

timeout: 300

scancode --copyright --license --info --strip-root -n 4 --timeout 300  upgraded-clusterfuzz-testcase-5298216461402112 --json-pp clusterfuzz.json
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
[####################] 2
Scanning done.
Summary:        info, licenses, copyrights with 4 process(es)
Errors count:   0
Scan Speed:     0.01 files/sec. 6.21 KB/sec.
Initial counts: 1 resource(s): 1 file(s) and 0 directorie(s)
Final counts:   1 resource(s): 1 file(s) and 0 directorie(s) for 672.05 KB
Timings:
  scan_start: 2022-12-15T143000.658938
  scan_end:   2022-12-15T143150.160877
  setup_scan:licenses: 1.32s
  setup: 1.32s
  scan: 108.18s
  total: 109.50s
Removing temporary files...done.

Results:

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "31.2.1",
      "options": {
        "input": [
          "upgraded-clusterfuzz-testcase-5298216461402112"
        ],
        "--copyright": true,
        "--info": true,
        "--json-pp": "clusterfuzz.json",
        "--license": true,
        "--processes": "4",
        "--strip-root": true,
        "--timeout": "300.0"
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2022-12-15T143000.658938",
      "end_timestamp": "2022-12-15T143150.160877",
      "output_format_version": "2.0.0",
      "duration": 109.50194954872131,
      "message": null,
      "errors": [],
      "warnings": [],
      "extra_data": {
        "system_environment": {
          "operating_system": "linux",
          "cpu_architecture": "64",
          "platform": "Linux-5.14.0-1054-oem-x86_64-with-glibc2.29",
          "platform_version": "#61-Ubuntu SMP Fri Oct 14 13:05:50 UTC 2022",
          "python_version": "3.8.10 (default, Nov 14 2022, 12:59:47) \n[GCC 9.4.0]"
        },
        "spdx_license_list_version": "3.17",
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "upgraded-clusterfuzz-testcase-5298216461402112",
      "type": "file",
      "name": "upgraded-clusterfuzz-testcase-5298216461402112",
      "base_name": "upgraded-clusterfuzz-testcase-5298216461402112",
      "extension": "",
      "size": 688179,
      "date": "2022-12-15",
      "sha1": "4271e5059b58be25d0845678c8b1c3f12c0040c3",
      "md5": "3fe4430ff25b408b7516b09433361050",
      "sha256": "64f26b6a338731cf1ebf7c3c166999150052e4947e405de8c84f88a2b0c1813b",
      "mime_type": "text/plain",
      "file_type": "ASCII text, with very long lines",
      "programming_language": null,
      "is_binary": false,
      "is_text": true,
      "is_archive": false,
      "is_media": false,
      "is_source": false,
      "is_script": false,
      "licenses": [],
      "license_expressions": [],
      "percentage_of_license_text": 0,
      "copyrights": [],
      "holders": [],
      "authors": [],
      "files_count": 0,
      "dirs_count": 0,
      "size_count": 0,
      "scan_errors": []
    }
  ]
}

Another point: you don't need to use both:

-n 4
--processes 7
as they are both the same CLI option, scancode will take only one.

This is same as #3175 also.

[sschuberth]

Thanks. Let us verify that increasing the timeout indeed solves the issue, and eventually close it afterwards.