evaluating scancode

[valeriocos]

Hi @pombredanne,

I've embedded scancode (a really nice tool) in Graal and now I'm evaluating scancode against nomos (another popular tool for license analysis) wrt precision and performance.

In a nutshell, the evaluation consists in iterating over the commits of a set of git repositories, for each commit graal performs a checkout and launches scancode/nomos on each file present in the commit, finally the results are persisted on disk.
While nomos is pretty fast (it processed 5 repos of around 3000 commits each in 2 hours), scancode is stilll processing the first repo. I'm wondering if I'm missing some parameters (or if you have some suggestions) to make the analysis faster. Currently I'm using the release 3.0.0 and I launch it with the following params:
https://github.com/chaoss/grimoirelab-graal/blob/master/graal/backends/core/analyzers/scancode.py#L58

Thank you

[pombredanne]

@valeriocos Hi! it was nice to meet at FOSDEM and thank you for the report.
You want to use 3.0.2 but that minor. You would likely to run on multiple processes with --processes X where X would the number of parallel processes. There could be some other flags too
Can you tell me some examples of what you scan too? A whole checkout for each commit? or something else? And is this something you run on one file at a time?

[valeriocos]

Thank you for answering @pombredanne and indeed It was nice to meet you too at FOSDEM.
I'm going to try with the version 3.0.2 and the --processes param and let you know.

Scancode is currently being executed on https://github.com/xiph/vorbis, graal performs a whole checkout of the commit, and then scancode is launched for each file in the commit (one file at a time)

if you want to execute it in your machine, you can install graal and then execute from command line:

graal colic 
https://github.com/xiph/vorbis (the URL of the repo)
--git-path /home/test-scancode-vorbis (where the repo is going to be downloaded)
--exec-path /home/scancode-toolkit-3.0.0/scancode (the exec path of scancode)
--category code_license_scancode (the category to activate scancode)

[pombredanne]

@valeriocos there are two issues:

1. there some pathological files in vorbis (and that's a bug tracked in It takes too long to scan Vorbis sources  #1404)
2. running a scan file-by-file is the worst case scenario for ScanCode.

Now if I run the quick comparison where I run not file-by-file but a directory at a time, and both using 8 processes:

$ time ~/w421/scancode-toolkit-master2/scancode --license vorbis-9eadeccdc4247127d91ac70555074239f5ce3529 -n8 --json-pp sc.json
Setup plugins...
Collect file inventory...
Scan files for: licenses with 8 process(es)...
[####################] 413                                          
Scanning done.
Summary:        licenses with 8 process(es)
Errors count:   0
Scan Speed:     13.29 files/sec. 
Initial counts: 453 resource(s): 413 file(s) and 40 directorie(s) 
Final counts:   453 resource(s): 413 file(s) and 40 directorie(s) 
Timings:
  scan_start: 2019-02-28T140649.087502
  scan_end:   2019-02-28T140722.627997
  setup_scan:licenses: 2.33s
  setup: 2.33s
  scan: 31.07s
  total: 33.62s
Removing temporary files...done.

real	0m34.809s
user	2m41.974s
sys	0m2.723s

and :

$ time ./nomossa -d ~/tmp/bnch/vorbis-9eadeccdc4247127d91ac70555074239f5ce3529 -n8  > out.txt

real	0m3.184s
user	0m20.022s
sys	0m0.076s

So we have 34.809s vs. 3.184s which is about 11 times slower in this case (which is still too slow and #1400 would help). In practice because of bugs nomossa should only be running on a single thread to avoid munging the output. In this case the elapsed time is 6.1s which means this is still about 5 times faster.

If you were to run file by file, it would take about 18s vs. well ... ~39 minutes?
There is a fixed startup time of about ~2s to load the license index but some overhead: everything in ScanCode is designed more for batch rather one-at-a-time.

Now if you run on Python 2, you could directly invoke the license detection as a function call on one file at a time. There, the overhead of the index load would happen only once. You would also bypass any JSON serialization/deserialization.

Of course, the bug I mentioned in #1404 in also making this more problematic but I happen to a work in progress fix in a local branch

The other thing to consider is the accuracy of detection (it does not help to be fast if this is to be incorrect).
A quick comparison shows a few false positive GPL and several imprecise detections in nomos. I will post a comparison table tomorrow

[pombredanne]

then scancode is launched for each file in the commit (one file at a time)

the short answer is that this is the worst case for ScanCode (one file at a time)

[pombredanne]

@valeriocos are you dealing with only the files changed in a commit ? or all the files are rescanned at every commit? because there is a new feature being worked on to provide multiple paths as args to ScanCode in #1399 based on a report by @nicobucher which seems quite related.

[valeriocos]

thank you @pombredanne for your detailed explanation. I'm dealing with only the files changed in a commit. I'll have a look at #1399 and try to use it.

Now if you run on Python 2, you could directly invoke the license detection as a function call on one file at a time. There, the overhead of the index load would happen only once. You would also bypass any JSON serialization/deserialization.

Unfortunately I'm using Python 3, thus scancode is executed by command line.

The other thing to consider is the accuracy of detection (it does not help to be fast if this is to be incorrect). A quick comparison shows a few false positive GPL and several imprecise detections in nomos. I will post a comparison table tomorrow

I attach the results for nomos and scancode obtained from the analysis of the vorbis repo. In a nutshell, each line in the files is a JSON representing the analysis for a given commit. In the attribute data.analysis you will find the output of scancode/nomos.

vorbis-analysis-nomos-vs-scancode.zip

(other repos are under analysis, once they are done I can share them with you if you want)

[pombredanne]

ok... so there is something that @armijnhemel was asking me about a while back which would be a way to have a pre-fork daemon of sorts such that there is always a pre-loaded process ready to scan.

@armijnhemel how would you do this?

Alternatively scanning a list of paths may amortize the startup costs too.
And I have some fixes for the #1404 issues that cuts down the scan time in half in some special cases (code files with most numeric data)

[armijnhemel]

I would agree with what @pombredanne says: running scancode file by file is the worst case scenario. What I could imagine is something similar to clamd where you can send a path and a persistent process scans that path.

[jgbarah]

Can scancode analyze lists of files (in a single invocation, I mean). Maybe that could speed up the thing, since each commit may touch several files, and in that case, running would be sort of linear with the number of commits instead of number of files touched for all commits. (I think this is what you mention above as "list of paths").

On a related note, do you have plans to support Python3 in the near future? That could make things easier too...

[pombredanne]

@jgbarah Hey!

Can scancode analyze lists of files (in a single invocation, I mean). Maybe that could speed up the thing, since each commit may touch several files, and in that case, running would be sort of linear with the number of commits instead of number of files touched for all commits. (I think this is what you mention above as "list of paths").

Not yet but I started a branch that can do that now at #1399 and this supports doing things such as git diff --name-only master | xargs scancode -i --json-pp -
and passing multiple paths in one call... it should soon land in develop
See also #875 and #1397 ... this last one by @nicobucher is kinda timely with your evaluation.

On a related note, do you have plans to support Python3 in the near future? That could make things easier too...

Yes, see #295 ... this is going to be a GSoC project

I also have a simple remoting solution using execnet until then and this works nicely: I will push it later today for your enjoyment.

[pombredanne]

@valeriocos re

I attach the results for nomos and scancode obtained from the analysis of the vorbis repo. In a nutshell, each line in the files is a JSON representing the analysis for a given commit. In the attribute data.analysis you will find the output of scancode/nomos.

Thank you! I will post my eval later.

[pombredanne]

@valeriocos do you mind trying this 8afa686 in the 1397-multiple-inputs branch? There is a README

[valeriocos]

thank you @pombredanne for this, I'll use it and report.