performance issues when scanning files individually

[armijnhemel]

I ran two tests: one on scanning a directory, and one on scanning all the files of the directory separately. The performance difference is quite big.

real 6m57.747s
user 6m2.996s
sys 0m48.011s

when doing:

time find /tmp/bzip2-1.0.1/ -type f -print0 | xargs -0 -n 1 ./bin/scancode

and:

real 1m0.930s
user 0m59.054s
sys 0m1.309s

when scanning the directory normally.

I did some (unprofessional) benchmarks: on my system loading the index pickle from disk takes about 3 seconds.

Maybe other alternatives, like msgpack, might work?

[armijnhemel]

I ran a test with msgpack. It seems it would require some changes to the internal model:

TypeError: can't serialize <licensedcode.index.LicenseIndex object at 0x7f9b7ae1f600>

Feel free to ignore.

[pombredanne]

So there is a load time incurred in the cached/pickled index. This was optimized somehow but not to run on individual files. What would be the use case to scan file by file? I guess a server-based approach would make more sense this that case.

Of notes is that some native libs (e.g. for the ahocorasick automatons) are in C and have pickle built-in and are not likely serializable directly with something like message pack.

[armijnhemel]

The use case that I have is that I want to do a delta scan: for example, differences between two stable versions of the Linux kernel is usually just a handful of files and if I only have to scan 30 files instead of 40,000, then that's a big win.

Right now I have worked around this problem by first identifying the files that I don't want to scan, recreate the directory structure, copy the files that I want to scan and then scan the directory with the copied files. It works, but it feels like a bit of a hack.

[pombredanne]

I wish I could change the serialization format of the Aho-Corasick automaton away from pickle... b ut this is native C++ code and that's rather engaged. A better way would be to use a memory-mapped automaton and not have this loading at all.

[pombredanne]

Some updates:

1. Eval and benchmark alternative ahocorasick implementation #2397 is a possible alternative library for building the license index that comes possibly with mmap support and this would likely improve/eschew load time (and it may be a tighter smaller index too)
2. there is an experimental client/server setting in https://github.com/nexB/scancode-toolkit/blob/develop/etc/scripts/scanserv.README which keeps a daemon-like server running for a cli client.
3. the latest release bundles a pre-built pickled DB. This may improve in a very small way the startup time