Exception when creating cyclonedx output

[jkwasny]

There is an exception when generating cyclonedx output and external references are merged. The error is:
seen = set(x)
TypeError: unhashable type: 'CycloneDxExternalRef'

It is not reproduceable using sample files but I've managed to slightly modify json output from scanning the samples and got the same error. Json file is attached, please try it with --cyclonedx and --from-json options.
I've run it on Windows 10 using python 3.8.10.

Thank you,
Janusz

scancode-samples-mod.txt

[AyanSinhaMahapatra]

Thanks for the report, I could reproduce from this JSON file you attached, and looking into this.

[MochonDeVuelta]

I have the same problem, when I try to generate a cyclonedx or cyclonedx-xml I get the following error message:

ERROR: failed to run output plugin: cyclonedx:

Traceback (most recent call last):
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\scancode\cli.py", line 1077, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 794, in process_codebase
bom = CycloneDxBom.from_codebase(codebase)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 681, in from_codebase
components = list(CycloneDxComponent.from_packages(packages))
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 339, in from_packages
base_component.merge(other_component)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 375, in merge
merge_lists(self.externalReferences, other.externalReferences)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 431, in merge_lists
seen = set(x)
TypeError: unhashable type: 'CycloneDxExternalRef'

Using scancode scancode-toolkit-v31.1.1 and phyton 3.8.10

[tgodau]

Hi there,

got a similar issue with the cyclonedx-output when scanning a very large package...

Used version:

ScanCode version: 31.0.1
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.17

Download, prepare package and start scanning:

> wget https://download.qt.io/official_releases/qt/5.15/5.15.7/submodules/qtwebengine-everywhere-opensource-src-5.15.7.zip
> unzip qtwebengine-everywhere-opensource-src-5.15.7.zip
> mv qtwebengine-everywhere-src-5.15.7/ qtwebengine-5.15.7/
> scancode -n 2 -clip qtwebengine-5.15.7/ --json-pp qtwebengine-5.15.7.json --cyclonedx qtwebengine-5.15.7_cdxSBOM.json

Gives the following error message:

:
:
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last):
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 794, in process_codebas
e
    bom = CycloneDxBom.from_codebase(codebase)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 681, in from_codebase
    components = list(CycloneDxComponent.from_packages(packages))
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 339, in from_packages
    base_component.merge(other_component)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 370, in merge
    merge_lists(self.licenses, other.licenses)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 431, in merge_lists
    seen = set(x)
TypeError: unhashable type: 'CycloneDxLicenseExpression'

Scanning done.
Some files failed to scan properly:
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtb.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtl.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/ios/icudtl.dat
Summary:        info, licenses, copyrights, packages with 2 process(es)
Errors count:   3
Scan Speed:     2.45 files/sec. 27.69 KB/sec.
Initial counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s)
Final counts:   183592 resource(s): 167376 file(s) and 16216 directorie(s) for 1.80 GB
Timings:
  scan_start: 2022-11-21T121128.360738
  scan_end:   2022-11-22T072026.947766
  setup_scan:licenses: 2.22s
  setup: 2.22s
  inventory: 185.09s
  scan:packages: 180.89s
  scan: 68326.72s
  output:json-pp: 123.34s
  output: 123.37s
  total: 69087.68s
Removing temporary files...done.

Maybe this helps...

Regards, Thorsten

[pombredanne]

@tgodau Thanks for the report. I am planning this for the next milestone