Support multiple input path in CLI #875 #1397

[pombredanne]

This adds support for #875 #1397 aka. multiple inputs and a new --include option.

• you can now enter multiple input paths with the constraint that they all must be relative path and share a common root directory.
• also add a new enabling option for --include that works in tandem with the --ignore and has the same semantics. Includes are processed first then ignores are applied on top.
• it is now possible to run this:
  find samples/ | grep zlib |grep -v ada | grep -v iostrea | xargs scancode --include="*JGroup*" --ignore "*.S" --json-pp -
  or this:
  git diff --name-only master | xargs scancode -i --json-pp -

Reported-by: Nico Bucher @nicobucher
Signed-off-by: Philippe Ombredanne pombredanne@nexb.com

[pombredanne]

@nicobucher ping... were you able to test this branch a bit? feedback is mucho welcomed!

[nicobucher]

Hi @pombredanne, I was able to quickly try this out.
It seems to do exactly what I was looking for!

I will play around with this branch in the next days. Perhaps I can give more detailed feedback then.

[pombredanne]

@nicobucher thank you for the feedback. Note that the current implementation is subpar performance wise because the logic goes this way:

1. find what is the root, shared ancestor directory of all the files arguments (and they must all be relative paths)
2. walk the whole tree for this
3. keep only the ones that match the requested paths.

If you have a large codebase (say a Linux kernel) and you modified only one file, this would walk 70K files first. So this can be optimized later if needed.

[codecov]

Codecov Report

Merging #1399 into develop will increase coverage by 0.17%.
The diff coverage is 70.55%.

@@             Coverage Diff             @@
##           develop    #1399      +/-   ##
===========================================
+ Coverage    83.65%   83.83%   +0.17%     
===========================================
  Files          119      119              
  Lines        13949    14245     +296     
===========================================
+ Hits         11669    11942     +273     
- Misses        2280     2303      +23

Impacted Files	Coverage Δ
src/formattedcode/output_jsonlines.py	100% <ø> (ø)	⬆️
src/plugincode/__init__.py	80% <0%> (-2.76%)	⬇️
src/scancode/__init__.py	75% <100%> (+0.3%)	⬆️
src/commoncode/fileutils.py	82.5% <100%> (+1.38%)	⬆️
src/commoncode/ignore.py	72.97% <100%> (ø)	⬆️
src/scancode/plugin_ignore.py	76.36% <68.42%> (-20.52%)	⬇️
src/scancode/cli.py	77.11% <69.69%> (+1.01%)	⬆️
src/commoncode/fileset.py	70.58% <75.86%> (-3.86%)	⬇️
src/formattedcode/output_json.py	77.58% <90%> (+0.22%)	⬆️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 887ae13...96069fd. Read the comment docs.

[pombredanne]

@johnmhoran I moved the script to create one SPDX doc per license to this branch... https://github.com/nexB/scancode-toolkit/pull/1399/files#diff-09a4d8eb1415f0acf8c0e24ec59fc3d3

[johnmhoran]

Thanks for the heads-up @pombredanne .

[pombredanne]

All green ... merging now.