configurable ignores with a `--ignore` option

[pombredanne]

This is a follow up of #35 from @K-Rex
We should add an --ignore command line option to point to an ignore file

[balusarakesh]

this ignore option can take either file types or the filenames itself separated by a comma
examples:

--ignore "file1.doc, file2.pdf" 
--ignore ".doc, .pdf"

[yashdsaraf]

@pombredanne I've added the --ignore and -I options to add files which are to be ignored while scanning. Here are the commits d9bc14b and 69e2167
Glob is also supported so 'file1.doc, file2.doc' or '*.doc' both should work,
Kindly tell if these changes satisfy the goal and follow the coding ethics of this project so I can make a PR.

[pombredanne]

@yashdsaraf Thank you! Let me look into and post some reviews comments. Did you look at #214 and #520 too? the way we want to do this would likely be using a config file of sorts, either a .gitgnore-like file or a general YAML file.

From a quick look, I am not sure that using a comma-separated list is the best approach. And using a global mapping does not feel entirely right.

@sschuberth @K-Rex @chinyeungli What do you think would be a good approach for "ignores" ?

[chinyeungli]

What is the use case here. Ignoring scans for .xml, .txt, .json, .yml, .html etc... may not be a very good idea as these files can contains license/copyright information.

Will the code automatically ignore scan on media files such as images/videos?

Instead of having ignore scan on certain file_type, perhaps the ignore option should ignore scan on resource that contains the user-defined value in its path.
For instance,
we may not want to scan the huge and well-known OSS components such as kernel/, busybox/, apache-common/
OR if we already know a subdirectory contains all the well-known OSS which we don't want scancode to scan it

[yashdsaraf]

@pombredanne So perhaps a --config option to specify a YAML file with keys like ignore, license with an optional subkey copyright which can further be divided into only and except options to fix #106
Also would the paths provided in the config support Glob?
@chinyeungli The code by default only ignores some VCS related files, can be found at https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/cli.py#L535 and https://github.com/nexB/scancode-toolkit/blob/develop/src/commoncode/ignore.py#L175
We could also add well-known OSS component dirs to https://github.com/nexB/scancode-toolkit/blob/develop/src/commoncode/ignore.py and import those from cli.py based on the options provided in the config by the user.
Maybe the application could throw an alert if some important dir was added to the config to be ignored.

[chinyeungli]

@yashdsaraf adding well-known OSS component dirs in the code is not a good idea as, perhaps, sometimes users or other people may want to see if these components been modified and/or if it contains any proprietary notice/company copyright notice if it's been modified.

[yashdsaraf]

@chinyeungli I meant giving the users an option, for e,g in a YAML config they could add something like
ignore: @oss
This would ignore a predefined set of well-known OSS component dirs or any other file (defined by the user) for that matter.

[mjherzog]

Whatever we do to enable ignore options must be primarily handled by a configuration file. A command line list is not practical in most cases where your ignore list may be tens or more arguments.

[sschuberth]

I don't necessarily think the ignore option must be managed by a configuration file. I do like the idea of passing glob patterns to ignore (but not just .doc to ignore all documents, that has to be something like **/*.doc then) as most likely very few patterns would do. Like for example **/src/test/** to ignore all test source code, which most of the time does not get delivered, and thus might be omitted from the scan.

[armijnhemel]

Alternatively, just some .ini file (easy to parse using ConfigParser) could be a solution and is probably more intuitive than a YAML file.