Parallel or distributed

[darcyfdu]

Can this tool be parallelized or distributed? Now the scanning speed is too slow ... not very practical？If it can,how should I do?

[pombredanne]

@darcyfdu you can run on X multiple processes with -n X or --processes X And it is fairly easy to distribute otherwise. Note that while the tool may not be as fast as it likely could be, yet the results are accurate.... and we value correctness over speed for sure.

You can surely find other tools that are faster and spit a lot of junk and or inaccurate results ;)

Now to give some perspective using multi processes on a fat/fast machine, a whole Debian pool (e.g. 27K packages and many million files ) takes about 11 hours to scan with ScanCode.

[pombredanne]

@darcyfdu did this work for you?

[pombredanne]

Ping?

[pombredanne]

I am closing this since there was no reply in several weeks. Feel free to reopen though.

[Jeeppler]

@pombredanne I am struggling to understand how many processes would be a sensible value to improve the speed of the scan? The default is 1, I guess that is a conservative value to ensure Scancode-Toolkit works even in resource restricted environments (e.g. low powered VM).

Should the processes be equal to the number of cores of the CPU? Is there an amount of processes which is just too much? Because not every part of the code can be parallelized.

Now to give some perspective using multi processes on a fat/fast machine, a whole Debian pool (e.g. 27K packages and many million files ) takes about 11 hours to scan with ScanCode.

Pretty impressive. Are there more information about this setup?

[darcyfdu]

你好。darcy已经收到你的邮件，谢谢！