When scanning a docker image, it make take forever to complete run.
The key reason is that when we actually scan files, each resource is scanned sequentially in a single process.
This was initially done this way because of some incompatibility between the various libraries we use for tasking and scancode-toolkit joint use of multiprocessing and multithreading.
We should instead actively use multiprocessing and multithreading to reach scan speeds similar to an equivalent command line run.
For reference, it took about 5 days :| to complete a scan for a mid-size Docker image of about 500MB. In contrast, running a -clipeu scan on -n5 processes from the CLI on the same code completed in a couple hours on the same machine. All that on a fairly underpowered standard laptop with 5 year old specs.
We can do better: we should be able to complete such a scan in a couple hours in scancode.io too.
When scanning a docker image, it make take forever to complete run.
The key reason is that when we actually scan files, each resource is scanned sequentially in a single process.
This was initially done this way because of some incompatibility between the various libraries we use for tasking and scancode-toolkit joint use of multiprocessing and multithreading.
We should instead actively use multiprocessing and multithreading to reach scan speeds similar to an equivalent command line run.
For reference, it took about 5 days :| to complete a scan for a mid-size Docker image of about 500MB. In contrast, running a
-clipeuscan on-n5processes from the CLI on the same code completed in a couple hours on the same machine. All that on a fairly underpowered standard laptop with 5 year old specs.We can do better: we should be able to complete such a scan in a couple hours in scancode.io too.