Scancode for licenses scans the same package from different sources

[tarungunampalli1]

Description

I am only looking to use scancode to check licenses in my project, and it runs extremely slowly, even when limited to only looking through the package.json files in my repo. I believe scancode doesn't optimize for when multiple installed packages have the same dependency, and it will scan that dependency many times. This causes it to take ~30 minutes to run, whereas this license compliance package runs in under 10 seconds. I would prefer using scancode over this package, however I can't justify a 30 minute run.

How To Reproduce

Tell us how to reproduce the issue.

Install multiple npm packages that share the same peer dependency and run scancode --licenses

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? (Windows/MacOS/Linux)
• What version of scancode-toolkit was used to generate the scan file?
• What installation method was used to install/run scancode? (pip/source download/other)

[pombredanne]

@tarungunampalli1 do you have an example to reproduce? If you want something that only looks at package.json, you can use --package and nothing else, this will detect license as declared only in packages manifests (which is what https://www.npmjs.com/package/license-compliance does) but we should be better at scale. Though if you do not detect with --license too, you will not get detection for the correct licenses for things like:

  "license": "SEE LICENSE IN <filename>"

that scancode happily follows and reports but no other tool I know of does process.

[tarungunampalli1]

The runtime is still incredibly slow. When I manually filter out the input paths to just package.jsons, scancode takes around 30 minutes. When I just let it run through our whole repo, I'm not sure because it hasn't even finished yet, but >2 hours. The command I am running is scancode --license --package --json-pp scan_results.json ..

[tarungunampalli1]

After running with --verbose, I can see the files it's scanning, and despite providing the --package flag, it seems to be scanning all files, not just package manifests.

[pombredanne]

After running with --verbose, I can see the files it's scanning, and despite providing the --package flag, it seems to be scanning all files, not just package manifests.

The option "--license" scan all files indeed, and "--package" alone would also check all files to collect which file belong to a package.

Something that may help is the new option --package-only that will only look at manifests:

    --package-only    Scan for system and application package data and skip
                      license/copyright detection and top-level package
                      creation.

This will not be a comprehensive scan but will be fast, if you just need that.

[pombredanne]

Also do you have a small reproducible sample project?

[tarungunampalli1]

Also do you have a small reproducible sample project?

I think the size of my repo is part of the issue, so a small reproduction wouldn't really get the effect. However, based on what you said, it seems like this behavior is expected, since scancode will just scan all files.
The --package-only option seems reasonable, but the statement skip license/copyright detection and top-level package creation isn't desirable for my use case. We want license detection, but I feel that the only files needed to scan for that is just the package manifests.