From 27d10852c9c45e65867e117a49b3cb888ce07b51 Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Mon, 19 Sep 2022 16:45:03 +0530 Subject: [PATCH] Update plugins docs and fix links Reference: https://github.com/nexB/scancode-toolkit/issues/3103 Signed-off-by: Ayan Sinha Mahapatra --- docs/scripts/sphinx_build_link_check.sh | 0 docs/source/misc/faq.rst | 2 +- .../how_to_add_a_post_scan_plugin.rst | 86 +++++++++++++++---- 3 files changed, 72 insertions(+), 16 deletions(-) mode change 100644 => 100755 docs/scripts/sphinx_build_link_check.sh diff --git a/docs/scripts/sphinx_build_link_check.sh b/docs/scripts/sphinx_build_link_check.sh old mode 100644 new mode 100755 diff --git a/docs/source/misc/faq.rst b/docs/source/misc/faq.rst index 3456b09b83e..2ec5fa6d085 100644 --- a/docs/source/misc/faq.rst +++ b/docs/source/misc/faq.rst @@ -68,7 +68,7 @@ Can I install ScanCode in a Unicode path? ----------------------------------------- Yes but but it is best to void this for now. See -https://github.com/nexB/scancode- toolkit/issues/867 +https://github.com/nexB/scancode-toolkit/issues/867 There is a bug in virtualenv https://github.com/pypa/virtualenv/issues/457 that is now fixed but has not been extensively tested for ScanCode. diff --git a/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst b/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst index b2b0873cdc5..ea6aa728f3a 100644 --- a/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst +++ b/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst @@ -3,14 +3,71 @@ Add A Post-Scan Plugin ====================== +Scan plugins in ``scancode-toolkit`` +------------------------------------ + +A lot of scancode features are built-in plugins which are present with scancode-toolkit source code +and are usually enabled via the different scancode-toolkit CLI options and are grouped by the types +of plugins. + +Here are the major types of plugins: + +1. Pre-scan plugins (`scancode_pre_scan` in entry points) + + These plugins are run before the main scanning steps and are usually + filtering of input files, or file classification steps, on whose results + the main scan plugins depend on. The base plugin class to be extended is ``PreScanPlugin`` at + `/src/plugincode/pre_scan.py `_. + +2. Scan plugins (`scancode_scan` in entry points) + + The are the scancode plugins which does the file scanning for useful + information like license, copyrights, packages and others. These are + run on multiprocessing for speed as they are done on a per-file basis, + but there can also be post-processing steps on these which are run afterwards + and have access to all the per-file scan results. The base plugin class to be extended is + ``ScanPlugin`` at `/src/plugincode/scan.py `_. + +3. Post-scan plugins (`scancode_post_scan` in entry points) + + These are mainly data processing, summerizing and reporting plugins which + depend on all the results for the scan plugins. These add new codebase level + or file-level attributes, and even removes/modifies data as required + for consolidation or summarization. The base plugin class to be extended is ``PostScanPlugin`` + at `/src/plugincode/post_scan.py `_. + +4. Output plugins (`scancode_output` in entry points) + + Supported output options in scancode-toolkit are all plugins and + these can also be multiple output options selected. These convert, process + and writes the data in the specific file format as the output of the scanning + procedures. The base plugin class to be extended is ``OutputPlugin`` at + `/src/plugincode/output.py `_. + +5. Output Filter Plugins (`scancode_output_filter` in entry points) + + There are also output filter plugins which apply filters to the outputs + and is modified. These filters can be based on whether resources had any + detections, ignorables present in licenses and others. + The base plugin class to be extended is ``OutputFilterPlugin`` at + `/src/plugincode/output_filter.py `_. + +6. Location Provider Plugins + + These plugins provide pre-built binary libraries and utilities and their locations which + are packaged to be used in scancode-toolkit. The base plugin class to be extended is + ``LocationProviderPlugin`` at `/src/plugincode/location_provider.py `_. + + Built-In vs. Optional Installation ---------------------------------- Built-In ^^^^^^^^ -Some post-scan plugins are installed when ScanCode itself is installed, e.g., the -:ref:`license_policy_plugin`, whose code is located here:: +Some post-scan plugins are installed when ScanCode itself is installed, and they are specified at +``[options.entry_points]`` in the `setup.cfg `_ file. +For example, the :ref:`license_policy_plugin` is a built-in plugin, whose code is located here:: https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/plugin_license_policy.py @@ -23,7 +80,7 @@ Optional ScanCode is also designed to use post-scan plugins that must be installed separately from the installation of ScanCode. The code for this sort of plugin is located here:: - https://github.com/nexB/scancode-toolkit/tree/develop/plugins/ + https://github.com/nexB/scancode-plugins This wiki page will focus on optional post-scan plugins. @@ -45,18 +102,17 @@ We'll start by creating three folders: 1. Top-level folder -- ``/scancode-hello/`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- In the ``/scancode-toolkit/plugins/`` directory, add a folder with a relevant name, e.g., - ``scancode-hello``. This folder will hold all of your plugin code. +- In the ``scancode-plugins`` repository, in the ``misc`` directory, add a folder with + a relevant name, e.g., ``scancode-hello``. This folder will hold all of your plugin code. - Inside the ``/scancode-hello/`` folder you'll need to add a folder named ``src`` and 7 files. - -1. ``/src/`` -- This folder will contain your primary Python code and is discussed in more detail - in the following section. + ``/src/`` -- This folder will contain your primary Python code and is discussed in more detail + in the following section. The 7 Files are: 1. ``.gitignore`` -- See, e.g., - `/plugins/scancode-ignore-binaries/.gitignore `_ + `/scancode-ignore-binaries/.gitignore `_ :: @@ -64,7 +120,7 @@ The 7 Files are: /dist/ 2. ``apache-2.0.LICENSE`` -- See, e.g., - `/plugins/scancode-ignore-binaries/apache-2.0.LICENSE `_ + `/scancode-ignore-binaries/apache-2.0.LICENSE `_ 3. ``MANIFEST.in`` @@ -83,7 +139,7 @@ The 7 Files are: global-exclude *.py[co] __pycache__ *.*~ 4. ``NOTICE`` -- See, e.g., - `/plugins/scancode-ignore-binaries/NOTICE `__ + `/scancode-ignore-binaries/NOTICE `__ 5. ``README.md`` @@ -129,7 +185,7 @@ The 7 Files are: long_description=desc, author='nexB', author_email='info@aboutcode.org', - url='https://github.com/nexB/scancode-toolkit/plugins/scancode-categories', + url='https://github.com/nexB/scancode-plugins/blob/main/misc/scancode-hello/', packages=find_packages('src'), package_dir={'': 'src'}, py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')], @@ -201,11 +257,11 @@ Create a ``PostScanPlugin`` class """"""""""""""""""""""""""""""""" The ``PostScanPlugin`` class -`PostScanPlugin code `_) +`PostScanPlugin code `_) inherits from the ``CodebasePlugin`` class (see -`CodebasePlugin code `_), +`CodebasePlugin code `_), which inherits from the ``BasePlugin`` class (see -`BasePlugin code `_). +`BasePlugin code `_). ::