Running scancode in a npm based repository takes long

[hiwit]

Description

• we want to include scancode in our pipeline but are running into timing issues
• scanning a code base with node_modules for a react / npm project is very slow (>5h).

How To Reproduce

• npx create-react-app test
• install some more npm packages
• scancode -cl -n 4 --html-app /project/result.html /project

Small workaround

• we were able to reduce the time by including only text files (difficult to perform as the npm packages differ in structure / language and other types of files)

System configuration

• running on macos (catalina/big sur)
• scancode version 21.3.31
• as an Application // scancode --help AND docker based

[pombredanne]

@hiwit Thank you for the report!

Can you provide a few more details?

1. please confirm you are using Python 3.6

2. What is your CPU model? how many CPU cores?

3. How much RAM do you have?

4. when you say "install some more npm packages" can you be more explicit? There could be 10 or 10,000 of these as npm can have some hidden "iceberg"-like tree of dependencies . Could you run this on your test app directory: find . -name "package.json" | wc -l to find out how many packages there are. And find . -type f | wc -l to find out how many files you have.

5. Have you tried to use --package scan first for a start?

6. Note that --html-app is deprecated, you should consider using JSON and the workbench instead

7. When you say "docker based" you mean you run Docker on macOS? that's several levels of indirection as this is running a VM that is running Docker that is running scancode. ScanCode runs natively on macOS FWIW.

8. can you paste the time and file number stats printed at the end of your run?

[hiwit]

Hi, thanks for your quick reply.

1. yes, we were using python 3.6
2. 2,6 GHz 6-Core Intel Core i7
3. 16 GB 2400 MHz DDR4
4. for our target application we have the following dependencies
   "dependencies": { "@types/clipboard": "^2.0.1", "@types/cookie": "^0.3.3", "@types/jest": "26.0.15", "@types/lodash.replace": "^4.1.6", "@types/node": "12.0.8", "@types/react-dom": "16.8.4", "@types/react-router": "^5.1.8", "@types/react-router-dom": "^4.3.4", "@types/uuid": "^8.3.0", "ajv": "^6.12.6", "clipboard": "^2.0.6", "core-js": "^3.7.0", "date-fns": "^2.16.1", "i18next": "^17.3.1", "jwt-decode": "^3.1.2", "lodash.debounce": "^4.0.8", "lodash.replace": "^4.1.4", "query-string": "^6.13.7", "react": "^16.14.0", "react-app-polyfill": "^1.0.6", "react-dom": "^16.14.0", "react-markdown": "^5.0.3", "react-router": "^5.2.0", "react-router-dom": "^5.2.0", "styled-components": "^5.2.1", "uuid": "^3.3.3", "xml-js": "^1.6.11" }
   npm install --only=prod (no dev dependencies)
   find . -name "package.json" | wc -l =1337
   find . -type f | wc -l = 23027
5. actually, no. I was not sure what the package option is really for :/
6. will check that
7. we first tried it on docker (and it probably will in the pipeline), but the problem also occures on the "native" python app
8. without include/ignore options we were not able to complete the scan.

[hiwit]

small update: we were able to run scancode in 1362.15s when heavily using the include command.
we filtered all text files before the scan and added the distinct filenames as a long list of includes...
find node_modules -type f -exec grep -Iq . {} \; -print
scancode --include "*.js" --include "*.cc" ... -cl node_modules
but I'm not sure how valid the result is

[pombredanne]

@hiwit these includes should not be needed, but that said, 23K files in about 20 minutes and 4 processes does NOT sound too crazy.

You should try a plain --package for a start instead and without includes or excludes.

[hiwit]

20 minutes is ok, i guess... :) but that was after optimization / filtering :) but it seems that using the json output is also somewhere in that timeframe :) i'll run some more tests to validate

[pombredanne]

@hiwit ping, any update on your side?

[hiwit]

hi @pombredanne, sorry for the late answers.

we're now able to generate a report in about 25 minutes in a concourse job. that's fine. I'm not sure what did the trick in the end. Probably switching to json improved a lot (filtering files was also a good idea :) ).

[DanielRuf]

For me switching to json did not do the trick in ORT.

Unfortunately license texts can be in txt, rst, md, LICENSE, COPYRIGHT and more files. What do all your include parameters look like @hiwit?

[pombredanne]

@DanielRuf should I reopen this issue or may be it would be better if you create a new one instead, with your details and inputs that are likely different?

[DanielRuf]

@pombredanne I guess I should create a new issue since in my case I use scancode through ORT(with and without Docker).

Inside Docker it takes about 45 minutes, outside of it only 39 minutes with empty caches. Not sure if this is slow or fast for the following setup (package.json and yarn.lock):

package-files.zip

[pombredanne]

@DanielRuf your lock files has links to 227 npms.

• Fetching them with wget: Total wall clock time: 20s Downloaded: 227 files, 8.6M in 8.9s (994 KB/s)

• Extracting them recursively with extractcode: 2m10.239s

• $ find . -type f | wc -l: 11350

• Scanning the packages manifests on 6 cores on my laptop with scancode --package --processes 6 --json daniel.json: 6216 seconds with 227 packages and 2294 dependency instances

• Scanning all the package files in 227 packages and 2294 dependency instances is likely to take several hours alright IMHO.

[DanielRuf]

@pombredanne thanks for checking this. I had a small typo, outside of Docker on my 8 core (16 with hyper threading) it took 39 (not 29) minutes, in Docker (ORT image) it was 45 minutes.

So this is a quite normal duration? Sorry for my naive question as I am new to ORT and scancode.
I guess we can not speed this much up besides scaling our server vertically (more CPU power)?

Will scancode 32 improve the performance further?

[wzrdtales]

I have to wonder, why it seems "alright" to take hours for such small amount of data?