diff --git a/docs/custom-pipelines.rst b/docs/custom-pipelines.rst index 73fdf8312d..8bbf5d38d6 100644 --- a/docs/custom-pipelines.rst +++ b/docs/custom-pipelines.rst @@ -3,11 +3,16 @@ Custom Pipelines ================ -- A pipeline is a **Python class** that lives in a Python module as a ``.py`` **file**. +A Pipeline is a Python script that performs code analysis by executing a +sequence of steps. + +- A pipeline is a **Python class** that lives in a Python module as a ``.py`` + **file**. - A pipeline class **always inherits** from the ``Pipeline`` base class - :ref:`pipeline_base_class`, or from another existing pipeline class, such as the - :ref:`built_in_pipelines`. -- It **defines steps** using the ``steps`` classmethod. + :ref:`pipeline_base_class`, or from other existing pipeline classes, such as + the :ref:`built_in_pipelines`. +- It **defines steps** - execution order of the steps - using the ``steps`` + classmethod. See :ref:`pipelines_concept` for more details. @@ -15,17 +20,18 @@ Pipeline registration --------------------- Built-in pipelines are located in :guilabel:`scanpipe/pipelines/` directory and -registered during the ScanCode.io installation. +are registered during the ScanCode.io installation. -Custom pipelines can be added as Python files ``.py`` in the directories defined in -the :ref:`scancodeio_settings_pipelines_dirs` setting and will be automatically -registered at runtime. +Whereas custom pipelines are added as Python files ``.py`` in the directories +defined in the :ref:`scancodeio_settings_pipelines_dirs` setting. Custom +pipelines are registered at runtime. Create a Pipeline ----------------- -Create a new Python file ``my_pipeline.py`` in the and make sure the directory is -registered in the :ref:`scancodeio_settings_pipelines_dirs` setting. +Create a new Python file ``my_pipeline.py``, and make sure to include the full +path of the new pipeline directory in the :ref:`scancodeio_settings_pipelines_dirs` +setting. .. code-block:: python @@ -48,14 +54,15 @@ registered in the :ref:`scancodeio_settings_pipelines_dirs` setting. .. tip:: - Have a look in the :guilabel:`scanpipe/pipelines/` directory for more pipeline + You can view the :guilabel:`scanpipe/pipelines/` directory for more pipeline examples. Modify existing Pipelines ------------------------- -Any existing pipeline can be reused as a base and customized. -You may want to override existing steps, add new ones, and remove some. +Existing pipelines are flexible and can be reused as a base for custom pipelines +, i.e. be customized. For instance, you can override existing steps, add new +ones, or remove any of them. .. code-block:: python @@ -87,14 +94,14 @@ You may want to override existing steps, add new ones, and remove some. pass -Report step example -------------------- +Custom Pipeline example +----------------------- -Example of a custom pipeline based on the built-in :ref:`pipeline_scan_codebase` one -with an extra reporting step. +The example below shows a custom pipeline that is based on the built-in +:ref:`pipeline_scan_codebase` pipeline with an extra reporting step. -Add the following content to a Python file and register its directory in the -:ref:`scancodeio_settings_pipelines_dirs`. +Add the following code snippet to a Python file and register the path of +the file's directory in the :ref:`scancodeio_settings_pipelines_dirs`. .. code-block:: python diff --git a/docs/scanpipe-command-line.rst b/docs/scanpipe-command-line.rst index 0c1aa03998..b2ee113b91 100644 --- a/docs/scanpipe-command-line.rst +++ b/docs/scanpipe-command-line.rst @@ -1,7 +1,7 @@ .. _scanpipe_command_line: -Management Commands -=================== +Command Line Interface +====================== The main entry point is the :guilabel:`scanpipe` command which is available directly when you are in the activated virtualenv or at this path: diff --git a/docs/scanpipe-concepts.rst b/docs/scanpipe-concepts.rst index 70c38458e4..1203128945 100644 --- a/docs/scanpipe-concepts.rst +++ b/docs/scanpipe-concepts.rst @@ -8,12 +8,13 @@ Project A **project** encapsulates the analysis of software code: -- it has a **workspace** which is a directory that contains the software code files under - analysis -- it is related to one or more **code analysis pipelines** scripts to automate its analysis -- it tracks ``Codebase Resources`` e.g. its **code files and directories** -- it tracks ``Discovered Packages`` e.g. its the **system and application packages** origin and - license discovered in the codebase +- It has a **workspace**, which is a directory that contains the software code + files under analysis. +- It makes use of one or more **code analysis pipelines** scripts to automate + the code analysis process. +- It tracks ``Codebase Resources``, i.e. its **code files and directories** +- It tracks ``Discovered Packages``, i.e. **system and application packages** + origin and license discovered in the codebase. In the database, **a project is identified by its unique name**. @@ -25,42 +26,47 @@ In the database, **a project is identified by its unique name**. Project workspace ----------------- -A project workspace is the root directory where **all the project files are stored**. +A project workspace is the root directory where **a project's files are stored**. -The following directories exists under this workspace directory: +The following directories exist under the workspace directory: -- :guilabel:`input/` contains all the original uploaded and input files used of the project. - For instance, it could be a codebase archive. -- :guilabel:`codebase/` contains the files and directories (aka. resources) tracked as - CodebaseResource records in the database. -- :guilabel:`output/` contains all output files created by the pipelines: reports, - scan results, etc. -- :guilabel:`tmp/` is a scratch pad for temporary files generated during the pipelines runs. +- :guilabel:`input/` contains all uploaded files used as the input of a project, + such as a codebase archive. +- :guilabel:`codebase/` contains files and directories - i.e. resources - + tracked as CodebaseResource records in the database. +- :guilabel:`output/` contains any output files created by the pipelines, + including reports, scan results, etc. +- :guilabel:`tmp/` is a scratch pad for temporary files generated during + pipelines runs. .. _pipelines_concept: Pipelines --------- -A pipeline is a Python script that contains a series of steps from start to end -to execute in order to **perform a code analysis**. +A pipeline is a Python script that contains a series of steps, which are +executed sequentially to **perform a code analysis**. -It usually starts from the uploaded input files, and may extract these then -generates ``CodebaseResource`` records in the database accordingly. +It usually starts with the uploaded input files, which might need to be +extracted first. Then, it generates ``CodebaseResource`` records in the database +accordingly. Those resources can then be **analyzed, scanned, and matched** as needed. Analysis results and reports are eventually posted at the end of a pipeline run. -All pipelines are located in the ``scanpipe.pipelines`` module. -Each pipeline consist of a Python script including one subclass of the ``Pipeline`` class. +All :ref:`built_in_pipelines` are located in the ``scanpipe.pipelines`` module. +Each pipeline consists of a Python script and includes one subclass of the +``Pipeline`` class. Each step is a method of the ``Pipeline`` class. -The execution order of the steps is declared through the ``steps`` class attribute -which is a sequence of steps to execute. +The execution order of the steps - or the sequence of steps execution - is +declared through the ``steps`` class attribute. + +.. tip:: + Refer to :ref:`custom_pipelines` for adding pipelines to ScanCode.io. .. note:: One or more pipelines can be assigned to a project as a sequence. - Codebase Resources ------------------ @@ -68,35 +74,34 @@ A project ``Codebase Resources`` are records of its **code files and directories ``CodebaseResource`` is a database model and each record is identified by its path under the project workspace. -Some of the ``CodebaseResource`` interesting attributes are: +The following are some of the ``CodebaseResource`` attributes: -- a **status** used to track the analysis status for this resource. -- a **type** (such as file, directory or symlink) -- various attributes to track detected **copyrights**, **license expressions**, +- A **status**, which is used to track the analysis status for this resource. +- A **type**, such as a file, a directory or a symlink +- Various attributes to track detected **copyrights**, **license expressions**, **copyright holders**, and **related packages**. .. note:: - In general the attributes and their names are the same that are used in - `ScanCode-toolkit `_ for files. - + Please note that `ScanCode-toolkit `_ + use the same attributes and attribute names for files. Discovered Packages ------------------- A project ``Discovered Packages`` are records of the **system and application packages** -discovered in its code. +discovered in the code unedr analysis. ``DiscoveredPackage`` is a database model and each record is identified by its ``Package URL``. -``Package URL`` is a grassroot efforts to create informative identifiers for software -packages such as Debian, RPM, npm, Maven, or PyPI packages. -See https://github.com/package-url for details. +``Package URL`` is a fundamental effort to create informative identifiers for +software packages, such as Debian, RPM, npm, Maven, or PyPI packages. +See https://github.com/package-url for more details. -Some of the ``DiscoveredPackage`` interesting attributes are: +The following are some of the ``DiscoveredPackage`` attributes: -- type, name, version (all Package URL attributes) -- homepage_url, download_url and other URLs -- checksums (such as SHA1, MD5) -- copyright, license_expression, declared_license +- A type, name, version (all Package URL attributes) +- A homepage_url, download_url, and other URLs +- Checksums, such as SHA1, MD5 +- Copyright, license_expression, and declared_license .. note:: - In general the attributes and their names are the same that are used in - `ScanCode-toolkit `_ for packages. + Please note that `ScanCode-toolkit `_ + use the same attributes and attribute names for packages.