RFC: retire pre-scan plugin stage

[pombredanne]

We could consider retiring pre-scan.

1. this is seldom used
2. this forces walking the whole codebase multiple times
3. this would be replace by a way to plugin in the filtering of the codebase, so that this can be walked only once
4. facets and classification can be made into regular scans that other can depend on

The plugins that use it:

• https://github.com/aboutcode-org/scancode-plugins/blob/9fcc50df36afdeb7b876eea4ec46fd2d2332772f/misc/scancode-ignore-binaries/setup.py#L48
• builtins classify = summarycode.classify:FileClassifier
• builtins facet = summarycode.facet:AddFacet
• builtins ignore = scancode.plugin_ignore:ProcessIgnore

The benefits:

• simpler code
• enable faster codebase walking

Based on a chat with @armijnhemel

[armijnhemel]

This would potentially positively impact #2725

[AyanSinhaMahapatra]

Working on this as a part of #4065:

Few things to note:

• classify plugin was already moved to a post-scan step with Add get_top_level_resources() to DatafileHandler class  #3315 : see latest plugins
• the --include option and the multiple input paths are essentially treated the same through code, this might not be a good idea.
• we are currently walking the codebase from the common prefix for multiple paths (i.e. walking from /, the entire filesystem, if inputs are /home/ayan/code/ and /tmp/other_code/) this is a major performance issue, see Scanning multiple directories scans too much #3452 for details
• I'm removing the --include --exclude plugins, but these options would be available as core scan options, and these are passed and used during codebase creation directly
• potentially parallelizing becomes possible after this, we just have to parallelize this depth_walk function, collect list of paths seperately, and then merge these and create a codebase at the end.
• I'm moving commoncode back into scancode-toolkit after discussion with @pombredanne, as the code is tightly linked and moving this out was a mistake. We will still build and release the wheel separately too just in case
• ignoring while walking based on include/exclude was already implemented in depth_walk but was never used. Enabling this
• this is also related to the much deliberated upon root of codebase issue. This is fine to technically have one root (common prefix for multiple inputs) but then prune the tree based on includes and excludes.
• --include/--exclude collisions are not documented properly and the precedence now is for the exclude, but IMHO we should provide a way to exclude larger directories and include smaller parts of the tree inside. This leads to better experience and less complex input options. --includes should also be seperate from input paths, such as there are 3 levels of scan paths specified: 1. inputs (codebase walks start from these, and these are sort of the main includes) 2. excludes (exclusions by filetypes, directories, specified by glob path) 3. includes (specific paths inside excludes which should be scanned). The 1st include is currently present as --inlcude and the 3rd include is currently not supported
• we should also support setting these exclude/includes via a file, similar to how we do this at scancode.io: https://scancodeio.readthedocs.io/en/latest/project-configuration.html#ignored-patterns (potentially adopting the same config file spec, scan_max_file_size and ignored_dependency_scopes is useful here and can be added later)

@pombredanne @armijnhemel please let me know your opinions on this.