Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 26 additions & 19 deletions docs/custom-pipelines.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,35 @@
Custom Pipelines
================

- A pipeline is a **Python class** that lives in a Python module as a ``.py`` **file**.
A Pipeline is a Python script that performs code analysis by executing a
sequence of steps.

- A pipeline is a **Python class** that lives in a Python module as a ``.py``
**file**.
- A pipeline class **always inherits** from the ``Pipeline`` base class
:ref:`pipeline_base_class`, or from another existing pipeline class, such as the
:ref:`built_in_pipelines`.
- It **defines steps** using the ``steps`` classmethod.
:ref:`pipeline_base_class`, or from other existing pipeline classes, such as
the :ref:`built_in_pipelines`.
- It **defines steps** - execution order of the steps - using the ``steps``
classmethod.

See :ref:`pipelines_concept` for more details.

Pipeline registration
---------------------

Built-in pipelines are located in :guilabel:`scanpipe/pipelines/` directory and
registered during the ScanCode.io installation.
are registered during the ScanCode.io installation.

Custom pipelines can be added as Python files ``.py`` in the directories defined in
the :ref:`scancodeio_settings_pipelines_dirs` setting and will be automatically
registered at runtime.
Whereas custom pipelines are added as Python files ``.py`` in the directories
defined in the :ref:`scancodeio_settings_pipelines_dirs` setting. Custom
pipelines are registered at runtime.

Create a Pipeline
-----------------

Create a new Python file ``my_pipeline.py`` in the and make sure the directory is
registered in the :ref:`scancodeio_settings_pipelines_dirs` setting.
Create a new Python file ``my_pipeline.py``, and make sure to include the full
path of the new pipeline directory in the :ref:`scancodeio_settings_pipelines_dirs`
setting.

.. code-block:: python

Expand All @@ -48,14 +54,15 @@ registered in the :ref:`scancodeio_settings_pipelines_dirs` setting.


.. tip::
Have a look in the :guilabel:`scanpipe/pipelines/` directory for more pipeline
You can view the :guilabel:`scanpipe/pipelines/` directory for more pipeline
examples.

Modify existing Pipelines
-------------------------

Any existing pipeline can be reused as a base and customized.
You may want to override existing steps, add new ones, and remove some.
Existing pipelines are flexible and can be reused as a base for custom pipelines
, i.e. be customized. For instance, you can override existing steps, add new
ones, or remove any of them.

.. code-block:: python

Expand Down Expand Up @@ -87,14 +94,14 @@ You may want to override existing steps, add new ones, and remove some.
pass


Report step example
-------------------
Custom Pipeline example
-----------------------

Example of a custom pipeline based on the built-in :ref:`pipeline_scan_codebase` one
with an extra reporting step.
The example below shows a custom pipeline that is based on the built-in
:ref:`pipeline_scan_codebase` pipeline with an extra reporting step.

Add the following content to a Python file and register its directory in the
:ref:`scancodeio_settings_pipelines_dirs`.
Add the following code snippet to a Python file and register the path of
the file's directory in the :ref:`scancodeio_settings_pipelines_dirs`.

.. code-block:: python

Expand Down
4 changes: 2 additions & 2 deletions docs/scanpipe-command-line.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _scanpipe_command_line:

Management Commands
===================
Command Line Interface
======================

The main entry point is the :guilabel:`scanpipe` command which is available
directly when you are in the activated virtualenv or at this path:
Expand Down
89 changes: 47 additions & 42 deletions docs/scanpipe-concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,13 @@ Project

A **project** encapsulates the analysis of software code:

- it has a **workspace** which is a directory that contains the software code files under
analysis
- it is related to one or more **code analysis pipelines** scripts to automate its analysis
- it tracks ``Codebase Resources`` e.g. its **code files and directories**
- it tracks ``Discovered Packages`` e.g. its the **system and application packages** origin and
license discovered in the codebase
- It has a **workspace**, which is a directory that contains the software code
files under analysis.
- It makes use of one or more **code analysis pipelines** scripts to automate
the code analysis process.
- It tracks ``Codebase Resources``, i.e. its **code files and directories**
- It tracks ``Discovered Packages``, i.e. **system and application packages**
origin and license discovered in the codebase.

In the database, **a project is identified by its unique name**.

Expand All @@ -25,78 +26,82 @@ In the database, **a project is identified by its unique name**.
Project workspace
-----------------

A project workspace is the root directory where **all the project files are stored**.
A project workspace is the root directory where **a project's files are stored**.

The following directories exists under this workspace directory:
The following directories exist under the workspace directory:

- :guilabel:`input/` contains all the original uploaded and input files used of the project.
For instance, it could be a codebase archive.
- :guilabel:`codebase/` contains the files and directories (aka. resources) tracked as
CodebaseResource records in the database.
- :guilabel:`output/` contains all output files created by the pipelines: reports,
scan results, etc.
- :guilabel:`tmp/` is a scratch pad for temporary files generated during the pipelines runs.
- :guilabel:`input/` contains all uploaded files used as the input of a project,
such as a codebase archive.
- :guilabel:`codebase/` contains files and directories - i.e. resources -
tracked as CodebaseResource records in the database.
- :guilabel:`output/` contains any output files created by the pipelines,
including reports, scan results, etc.
- :guilabel:`tmp/` is a scratch pad for temporary files generated during
pipelines runs.

.. _pipelines_concept:

Pipelines
---------

A pipeline is a Python script that contains a series of steps from start to end
to execute in order to **perform a code analysis**.
A pipeline is a Python script that contains a series of steps, which are
executed sequentially to **perform a code analysis**.

It usually starts from the uploaded input files, and may extract these then
generates ``CodebaseResource`` records in the database accordingly.
It usually starts with the uploaded input files, which might need to be
extracted first. Then, it generates ``CodebaseResource`` records in the database
accordingly.

Those resources can then be **analyzed, scanned, and matched** as needed.
Analysis results and reports are eventually posted at the end of a pipeline run.

All pipelines are located in the ``scanpipe.pipelines`` module.
Each pipeline consist of a Python script including one subclass of the ``Pipeline`` class.
All :ref:`built_in_pipelines` are located in the ``scanpipe.pipelines`` module.
Each pipeline consists of a Python script and includes one subclass of the
``Pipeline`` class.
Each step is a method of the ``Pipeline`` class.
The execution order of the steps is declared through the ``steps`` class attribute
which is a sequence of steps to execute.
The execution order of the steps - or the sequence of steps execution - is
declared through the ``steps`` class attribute.

.. tip::
Refer to :ref:`custom_pipelines` for adding pipelines to ScanCode.io.

.. note::
One or more pipelines can be assigned to a project as a sequence.


Codebase Resources
------------------

A project ``Codebase Resources`` are records of its **code files and directories**.
``CodebaseResource`` is a database model and each record is identified by its path
under the project workspace.

Some of the ``CodebaseResource`` interesting attributes are:
The following are some of the ``CodebaseResource`` attributes:

- a **status** used to track the analysis status for this resource.
- a **type** (such as file, directory or symlink)
- various attributes to track detected **copyrights**, **license expressions**,
- A **status**, which is used to track the analysis status for this resource.
- A **type**, such as a file, a directory or a symlink
- Various attributes to track detected **copyrights**, **license expressions**,
**copyright holders**, and **related packages**.

.. note::
In general the attributes and their names are the same that are used in
`ScanCode-toolkit <https://github.com/nexB/scancode-toolkit>`_ for files.

Please note that `ScanCode-toolkit <https://github.com/nexB/scancode-toolkit>`_
use the same attributes and attribute names for files.

Discovered Packages
-------------------

A project ``Discovered Packages`` are records of the **system and application packages**
discovered in its code.
discovered in the code unedr analysis.
``DiscoveredPackage`` is a database model and each record is identified by its ``Package URL``.
``Package URL`` is a grassroot efforts to create informative identifiers for software
packages such as Debian, RPM, npm, Maven, or PyPI packages.
See https://github.com/package-url for details.
``Package URL`` is a fundamental effort to create informative identifiers for
software packages, such as Debian, RPM, npm, Maven, or PyPI packages.
See https://github.com/package-url for more details.

Some of the ``DiscoveredPackage`` interesting attributes are:
The following are some of the ``DiscoveredPackage`` attributes:

- type, name, version (all Package URL attributes)
- homepage_url, download_url and other URLs
- checksums (such as SHA1, MD5)
- copyright, license_expression, declared_license
- A type, name, version (all Package URL attributes)
- A homepage_url, download_url, and other URLs
- Checksums, such as SHA1, MD5
- Copyright, license_expression, and declared_license

.. note::
In general the attributes and their names are the same that are used in
`ScanCode-toolkit <https://github.com/nexB/scancode-toolkit>`_ for packages.
Please note that `ScanCode-toolkit <https://github.com/nexB/scancode-toolkit>`_
use the same attributes and attribute names for packages.