Add glob matching comparable to gitignore files

[skoehler]

Short Description

Add glob matching comparable to .gitignore files

Possible Labels

• new feature

Select Category

• Enhancement
• Add License/Copyright
• Scan Feature
• Packaging
• Documentation
• Expand Support
• Other

Describe the Update

How This Feature will help you/your organization

I don't see how I can tell scancode to ignore/include all *.java files in a particular folder and its subfolders, for example.
Currently, I would have to write an --include for each folder and for each subfolder and for each subsubfolder, etc.

Possible Solution/Implementation Details

Here are some possible patterns and their meaning

*.java                java files at any level in the folder hierarchy
tests/*.java          java files in a folder called tests
tests/**/*.java       java files in a folder called tests or any subfolder of tests
/tests/**/*.java      java files in a folder called tests or any subfolder of tests, the folder tests has to be in the root directory

Example/Links if Any

Can you help with this Feature

[pombredanne]

Thank you. This should behave the same as gitignore alright and this is quite timely as we want to have configuration file(s) in ScanCode.io aboutcode-org/scancode.io#685 and here #520 and a primary use case is ignoring alright

Some other references:

• the spec https://github.com/git/git/blob/7e27bd589d328b9daf154c2444d1a86ec3afedb0/Documentation/gitignore.txt
• some implementations that claim spec compliance:
  • @jelmer dulwich https://github.com/jelmer/dulwich/blob/master/dulwich/ignore.py and its derivative: @benkehoe https://github.com/benkehoe/ignorelib
  • @snark https://github.com/snark/ignorance/
  • @mherrmann https://github.com/mherrmann/gitignore_parser derived from @snark
  • @valohai https://github.com/valohai/gitignorant
  • @bitranox https://github.com/bitranox/igittigitt
  • @excitoon https://github.com/excitoon/gitignorefile derived from mostly from @mherrmann but also crediting @bitranox and @cpburnz https://github.com/cpburnz/python-pathspec ... not sure sure what is borrowed from these two
• test suites to consider:
  • https://github.com/git/git/blob/master/t/t0008-ignores.sh
  • https://github.com/libgit2/libgit2/tree/main/tests/libgit2/ignore
  • https://github.com/jelmer/dulwich/blob/master/dulwich/tests/test_ignore.py
• some other local isssues:
  • the --ignore behaviour needs refinement  #726
  • configurable ignores with a --ignore option #36

[skoehler]

I forgot to mention that gitignore has a special syntax for folders:

test/          a folder called test anywhere in the hierarchy
/test/         a folder called test in the root directory
a/**/b/        a folder called b that is a subfolder (at any level) of a folder called a anywhere in the hierarchy
/a/**/b/       a folder called b that is a subfolder (at any level) of a folder called a in the root directory

To the best of my knowledge, a line without the slash in the end should only match files.

Also, it's a good idea to look for implementations. There should be plenty as you have pointed out!

Also a big "yes please!" to having configuration files. Having to specify everything on the command line is tedious.

[pombredanne]

@tdruez ping, as this may be something of interest for scancode.io