Use case: Scanning files contained in pull request

[nicobucher]

For projects using Git: "git diff --name-only master" shows a list of files which changed when comparing the current branch to master.

It would be great if the scanner could only scan those files in order to provide feedback about foss and licenses contained in the pull request. In bash-like environments, this task could be achieved by a one-liner (E.g. "git diff --name-only master | ./scancode --licenses --json-pp") if the scanner could take a list of files as input.

[pombredanne]

@nicobucher thank you ++.
So the output of a git diff --name-only is something like:

$ git diff --name-only
src/formattedcode/output_csv.py
src/packagedcode/models.py

And we have one path per line and each path is relative to the root of the git repository.

Here is an idea for an implementation:

• ScanCode has already an option to --ignore some files with this plugin https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/plugin_ignore.py
• there is an open ticket Need specific file type pattern option and License Version #875 to also have the opposite e.g a way to include file patterns
• the underlying fileset module used for --ignore already supports both inclusions and exclusions https://github.com/nexB/scancode-toolkit/blob/develop/src/commoncode/fileset.py

Therefore, an approach could be:

• the current single input directory argument would be kept as it is today
• a new option to include paths that would match a pattern would be added to the ignore plugin , and the logic would be to use this in conjunction with ignore
• the logic when working with combos of excludes/ignores and includes would be: 1. keep only included paths, 2. further exclude from these the ignored paths (this is the way of the fileset module)

This first step would not support yet something such asgit diff --name-only master | xargs scancode -i --json-pp - (note the use of xargs and the trailing dash to print the scan results) .... but a step towards this.

In fact the command may look more like git diff --name-only master | xargs scancode -i --json-pp - btw (note the use of xargs and the trailing dash to print the scan results)

To get there, the process could be:

1. allow multiple args in the scancode function CLI definition https://github.com/nexB/scancode-toolkit/blob/a7de3052d540b2f005960521635e7816f50485a6/src/scancode/cli.py#L263 by adding nargs= -1

2. handle the case where there is only a single path provided as usual, processing the input[0] item.

3. if more than one paths are provided, we need to ensure first that all the paths share a common root (this would otherwise require quite a few changes to the codebase/ virtual filesystem model used internally by ScanCode)

• first case: the provided input paths share a common first path segment. In this case we would transform the list of input path arguments as:

  • one input path which is the shared root
  • a list of include options (as created above), one per included path

• second case: the provided input paths do NOT share a common first path segment.

  • We can use the current directory as a root if all provided paths exist in the current directory (as is in the git diff cases). We then fall back to the 3.1 above.
  • Otherwise, fail the command and return a message stating we cannot run a scan with multiple input paths that do not share a common parent.

This is not the only way of course, but this would provide the least disruption and blend in the current code

[pombredanne]

@nicobucher see the code in the branch for #1399
@MaJuRG @JonoYang @chinyeungli you may be interested too.

[pombredanne]

@valeriocos ping. This may be relevant to your use case in https://github.com/Bitergia/graal ?

[pombredanne]

@nicobucher there is a new option in the develop branch called include that works this way --include="*JGroup*" --ignore "*.S" ... and it is also possible to enter multiple paths arguments.
I am closing now and your feedback here is welcomed.