Rewrite the CLI tutorials for a Docker-based installation #440 (#633)

* Add --print option to the output scanpipe command #440

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Rewrite the Analyze Codebase CLI tutorial for a Docker setup #440

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Rewrite the Analyze Docker Image CLI tutorial for a Docker setup #440

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Author: tdruez

CHANGELOG.rst

@@ -18,6 +18,9 @@ v33.0.0 (unreleased)
   integration in the documentation.
   https://github.com/nexB/scancode.io/issues/600
 
+- Rewrite the CLI tutorials for a Docker-based installation.
+  https://github.com/nexB/scancode.io/issues/440
+
 - Use CodebaseResource ``path`` instead of ``id`` as slug_field in URL navigation.
   https://github.com/nexB/scancode.io/issues/242
 

docs/installation.rst

@@ -53,7 +53,9 @@ create an **environment file**, and **build the Docker image**::
 
 .. note::
     You need to rebuild the image whenever ScanCode.io's source code has been
-    modified or updated.
+    modified or updated::
+
+        docker compose build
 
 Run the App
 ^^^^^^^^^^^
@@ -87,7 +89,7 @@ An overview of the web application usage is available at :ref:`user_interface`.
     and ``CSRF_TRUSTED_ORIGINS`` settings need to be provided in your ``.env`` file,
     for example::
 
-        ALLOWED_HOSTS=.your-domain.com,127.0.0.1
+        ALLOWED_HOSTS=.your-domain.com,localhost,127.0.0.1
         CSRF_TRUSTED_ORIGINS=https://*.your-domain.com,http://127.0.0.1:8001
 
     Refer to `ALLOWED_HOSTS settings <https://docs.djangoproject.com/
@@ -288,7 +290,7 @@ you can start the local webserver and access the app with::
 
     make run
 
-Then open your web browser and visit: http://127.0.0.1:8001/ to access the web
+Then open your web browser and visit: http://localhost:8001/ to access the web
 application.
 
 .. warning::

docs/output-files.rst

@@ -5,10 +5,11 @@ Output Files
 
 Whether you use the command line or the web application to run your
 scans, the generated results are available for review or export in
-**JSON** and **Excel (XLSX)** file formats.
+**JSON**, **Excel (XLSX)**, **SPDX**, and **CycloneDX** file formats.
 
 .. tip::
-    Check our :ref:`data_model` section for more details about all fields included in the output files.
+    Check our :ref:`data_model` section for more details about all fields included
+    in the output files.
 
 Creating Output Files
 ---------------------
@@ -20,11 +21,11 @@ the output file format with the ``–-format`` option:
 
 .. code-block:: console
 
-    $ scanpipe output --project PROJECT --format {json,xlsx}
+    $ scanpipe output --project PROJECT --format {json,xlsx,spdx,cyclonedx}
 
 .. note::
-    The previous command will output the scan results in a JSON or XLSX file
-    format – as specified – with the output files created in the
+    The previous command will output the scan results in a file format
+    – as specified – with the output files created in the
     ``PROJECT``'s :guilabel:`output/` directory. By default, JSON output
     files are created when no file format is given.
 

docs/rest-api.rst

@@ -4,7 +4,7 @@ REST API
 ========
 
 To get started with the REST API, visit the **projects' API endpoint** at
-http://localhost/api/projects/ or http://127.0.0.1:8001/api/projects/ if you run on a
+http://localhost/api/projects/ or http://localhost:8001/api/projects/ if you run on a
 local development setup.
 
 .. _rest_api_authentication:
@@ -316,7 +316,7 @@ the :guilabel:`input/` directory.
 Errors
 ^^^^^^
 
-This action lists all errors that were logged during any pipeline(s) execution
+This action lists all errors that were logged during any pipeline execution
 on a given ``project``.
 
 ``GET /api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/errors/``
@@ -423,11 +423,11 @@ The run details view returns all information available about a pipeline run.
 .. code-block:: json
 
     {
-        "url": "http://127.0.0.1:8001/api/runs/8d5c3962-5fca-47d7-b8c8-47a19247714e/",
+        "url": "http://localhost/api/runs/8d5c3962-5fca-47d7-b8c8-47a19247714e/",
         "pipeline_name": "scan_package",
         "status": "success",
         "description": "A pipeline to scan a single package archive with ScanCode-toolkit.",
-        "project": "http://127.0.0.1:8001/api/projects/cd5b0459-303f-4e92-99c4-ea6d0a70193e/",
+        "project": "http://localhost/api/projects/cd5b0459-303f-4e92-99c4-ea6d0a70193e/",
         "uuid": "8d5c3962-5fca-47d7-b8c8-47a19247714e",
         "created_date": "2021-10-01T08:44:05.174487+02:00",
         "task_exitcode": 0,

docs/tutorial_api_analyze_package_archive.rst

@@ -40,7 +40,7 @@ Using cURL
     curl -X POST "$api_url" -H "$content_type" -d "$data"
 
 .. note::
-    You have to set the api_url to http://127.0.0.1:8001/api/projects/ if you run on a
+    You have to set the api_url to http://localhost:8001/api/projects/ if you run on a
     local development setup.
 
 .. tip::

docs/tutorial_cli_analyze_codebase.rst

@@ -7,31 +7,41 @@ The focus of this tutorial is to guide you through scanning a codebase package
 using ScanCode.io.
 
 .. note::
-    This tutorial assumes you have a current version of ScanCode.io installed
-    locally on your machine. If you do not have it installed,
-    see our :ref:`installation` guide for instructions.
+    This tutorial assumes you have a recent version of ScanCode.io installed
+    locally on your machine and **running with Docker**.
+    If you do not have it installed, see our :ref:`installation` guide for instructions.
 
 Requirements
 ------------
+
 Before you follow the instructions in this tutorial, you need to:
 
 - Install **ScanCode.io** locally
-- Download the following **package archive** and save it to your home directory: `asgiref-3.3.0-py3-none-any.whl <https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl>`_
 - Have **Shell access** on the machine where ScanCode.io is installed
 
 Instructions
 ------------
 
-- Open a shell in the ScanCode.io installation directory and activate the
-  virtual environment - **virtualenv**:
+- Create a new directory in your home directory that will be used to put the input code
+  to be scanned.
+
+.. code-block:: console
+
+    $ mkdir -p ~/codedrop/
+
+- Download the following **package archive** and save it to the :guilabel:`~/codedrop/`
+  directory: `asgiref-3.3.0-py3-none-any.whl <https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl>`_
 
 .. code-block:: console
 
-    $ source bin/activate
+    $ curl https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl --output ~/codedrop/asgiref-3.3.0-py3-none-any.whl
+
+- Create an alias to the ``scanpipe`` command executed through the
+  ``docker compose`` command line interface with:
 
 .. code-block:: console
 
-    >> (scancodeio) $
+    $ alias scanpipe="docker compose -f ${PWD}/docker-compose.yml run --volume ~/codedrop/:/codedrop:ro web scanpipe"
 
 - Create a new project named ``asgiref``:
 
@@ -41,18 +51,18 @@ Instructions
 
 .. code-block:: console
 
-    >> Project asgiref created with work directory projects/asgiref-072c89db
+    >> Project asgiref created with work directory /var/scancodeio/workspace/projects/asgiref-35519104
 
 - Add the package archive to the project workspace's :guilabel:`input/`
   directory:
 
 .. code-block:: bash
 
-    $ scanpipe add-input --project asgiref --input-file ~/asgiref-3.3.0-py3-none-any.whl
+    $ scanpipe add-input --project asgiref --input-file /codedrop/asgiref-3.3.0-py3-none-any.whl
 
 .. code-block:: console
 
-    >> File(s) copied to the project inputs directory:
+    >> File copied to the project inputs directory:
       - asgiref-3.3.0-py3-none-any.whl
 
 - Add the ``scan_codebase`` pipeline to your project:
@@ -63,7 +73,7 @@ Instructions
 
 .. code-block:: console
 
-    >> Pipeline(s) added to the project
+    >> Pipeline scan_codebase added to the project
 
 .. note::
     The content of the :guilabel:`input/` directory will be copied in the
@@ -83,26 +93,38 @@ Instructions
 .. code-block:: console
 
     >> Pipeline scan_codebase run in progress..
-       2021-07-12 17:45:53.85 Pipeline [scan_codebase] starting
-       2021-07-12 17:45:53.85 Step [copy_inputs_to_codebase_directory] starting
-       2021-07-12 17:45:53.86 Step [copy_inputs_to_codebase_directory] completed in 0.00 seconds
-       2021-07-12 17:45:53.86 Step [extract_archives] starting
+       Pipeline [scan_codebase] starting
+       Step [copy_inputs_to_codebase_directory] starting
+       Step [copy_inputs_to_codebase_directory] completed in 0.00 seconds
+       Step [extract_archives] starting
        [...]
-       2021-07-12 17:46:01.61 Pipeline completed
+       Pipeline completed
+       scan_codebase successfully executed on project asgiref
 
-- Finally, you can view your scan results in JSON or CSV file formats inside
-  the project's :guilabel:`output/` directory.
+- Finally, export the scan results as JSON format::
+
+  $ scanpipe output --project asgiref --format json --print > asgiref-3.3.0_results.json
 
 .. tip::
     The ``inputs`` and ``pipelines`` can be provided at the same time when
     calling the ``create-project`` command. For instance, the following command
-    will create a new project named ``asgiref``, add the package archive as the
+    will create a new project named ``asgiref2``, add the package archive as the
     project input, add the ``scan_codebase`` pipeline to the project, and
     execute it:
 
 .. code-block:: bash
 
-    $ scanpipe create-project asgiref \
-        --input-file ~/asgiref-3.3.0-py3-none-any.whl \
+    $ scanpipe create-project asgiref2 \
+        --input-file /codedrop/asgiref-3.3.0-py3-none-any.whl \
         --pipeline scan_codebase \
         --execute
+
+.. code-block:: console
+
+    >> Project asgiref2 created with work directory /var/scancodeio/workspace/projects/asgiref2-bea7a5e9
+       File copied to the project inputs directory:
+       - asgiref-3.3.0-py3-none-any.whl
+       Start the scan_codebase pipeline execution...
+       [...]
+       Pipeline completed
+       scan_codebase successfully executed on project asgiref2

docs/tutorial_cli_analyze_docker_image.rst

@@ -8,32 +8,42 @@ a test Docker image by following the steps below and, along the way,
 learn some of the ScanCode.io basic commands.
 
 .. note::
-    This tutorial assumes you have a current version of ScanCode.io installed
-    locally on your machine. If you do not have it installed,
-    see our :ref:`installation` guide for instructions.
+    This tutorial assumes you have a recent version of ScanCode.io installed
+    locally on your machine and **running with Docker**.
+    If you do not have it installed, see our :ref:`installation` guide for instructions.
 
 Requirements
 ------------
 
 To successfully complete this tutorial, you first need to:
 
 - Install **ScanCode.io** locally
-- Download the following **test Docker image** and save it to your home directory: `30-alpine-nickolashkraus-staticbox-latest.tar <https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar>`_
 - Have **Shell access** on the machine where ScanCode.io is installed
 
 Instructions
 ------------
 
-- Open a shell in the ScanCode.io installation directory and activate the
-  virtual environment - **virtualenv**:
+- Create a new directory in your home directory that will be used to put the input code
+  to be scanned.
 
 .. code-block:: console
 
-    $ source bin/activate
+    $ mkdir -p ~/codedrop/
+
+- Download the following **test Docker image** and save it to the :guilabel:`~/codedrop/`
+  directory: `30-alpine-nickolashkraus-staticbox-latest.tar
+  <https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar>`_
+
+.. code-block:: console
+
+    $ curl https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar --output ~/codedrop/30-alpine-nickolashkraus-staticbox-latest.tar
+
+- Create an alias to the ``scanpipe`` command executed through the
+  ``docker compose`` command line interface with:
 
 .. code-block:: console
 
-    >> (scancodeio) $
+    $ alias scanpipe="docker compose -f ${PWD}/docker-compose.yml run --volume ~/codedrop/:/codedrop:ro web scanpipe"
 
 - Create a new project named ``staticbox``:
 
@@ -43,30 +53,29 @@ Instructions
 
 .. code-block:: console
 
-    >> Project staticbox created with work directory projects/staticbox-d4ed9405
+    >> Project staticbox created with work directory /var/scancodeio/workspace/projects/staticbox-d4ed9405
 
 .. note::
     New projects work directory are created inside the location defined in
     :ref:`scancodeio_settings_workspace_location` setting.
-    Default to a :guilabel:`var/` directory in the local ScanCode.io codebase.
+    Default to the :guilabel:`/var/scancodeio/workspace/` directory.
 
 - Add the test Docker image tarball to the project workspace's :guilabel:`input/`
   directory:
 
 .. code-block:: bash
 
     $ scanpipe add-input --project staticbox \
-      --input-file ~/30-alpine-nickolashkraus-staticbox-latest.tar
+        --input-file /codedrop/30-alpine-nickolashkraus-staticbox-latest.tar
 
 .. code-block:: console
 
-    >> File(s) copied to the project inputs directory:
+    >> File copied to the project inputs directory:
        - 30-alpine-nickolashkraus-staticbox-latest.tar
 
 .. note::
     The command output will let you know that the Docker image file was
     copied to the project's :guilabel:`input/` directory.
-    You can also navigate to this directory and confirm your file is there.
     Alternatively, you can copy files manually to the :guilabel:`input/`
     directory to include entire directories.
 
@@ -78,7 +87,7 @@ Instructions
 
 .. code-block:: console
 
-    >> Pipeline(s) added to the project
+    >> Pipeline docker added to the project
 
 - Check the status of the pipeline added to your project:
 
@@ -107,12 +116,13 @@ Instructions
 .. code-block:: console
 
     >> Pipeline docker run in progress...
-       2021-07-07 10:39:26.49 Pipeline [docker] starting
-       2021-07-07 10:39:26.53 Step [extract_images] starting
-       2021-07-07 10:39:26.71 Step [extract_images] completed in 0.18 seconds
-       2021-07-07 10:39:26.71 Step [extract_layers] starting
+       Pipeline [docker] starting
+       Step [extract_images] starting
+       Step [extract_images] completed in 0.18 seconds
+       Step [extract_layers] starting
        [...]
-       2021-07-07 10:39:31.39 Pipeline completed
+       Pipeline completed
+       docker successfully executed on project staticbox
 
 - Executing the ``show-pipeline`` command again will also confirm the success
   of the pipeline execution - **"[SUCCESS] docker"** status:
@@ -129,14 +139,10 @@ Instructions
 
 .. code-block:: console
 
-    $ scanpipe output --project staticbox --format json
-
-.. code-block:: console
-
-    >> projects/staticbox-d4ed9405/output/results-2021-07-07-08-54-02.json
+    $ scanpipe output --project staticbox --format json --print > staticbox_results.json
 
-- Finally, open the ``output/results-<timestamp>.json`` file in your preferred
-  text editor/file viewer.
+- Finally, open the ``staticbox_results.json`` file in your preferred text
+  editor/file viewer.
 
 .. note::
     To understand the output of the pipeline execution, see our :ref:`output_files`
@@ -145,15 +151,15 @@ Instructions
 .. tip::
     The ``inputs`` and ``pipelines`` can be provided directly at once when
     calling the ``create-project`` command.
-    An ``execute`` option is also available to start the pipeline execution right
+    The ``--execute`` option is also available to start the pipeline execution right
     after the project creation.
     For example, the following command will create a project named ``staticbox2``,
     download the test Docker image to the project's :guilabel:`input/`
     directory, add the docker pipeline, and execute the pipeline in one operation:
 
     .. code-block:: bash
 
-      $ scanpipe create-project staticbox2 \
-        --input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar \
-        --pipeline docker \
-        --execute
+        $ scanpipe create-project staticbox2 \
+            --input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar \
+            --pipeline docker \
+            --execute

docs/tutorial_vulnerablecode_integration.rst

@@ -60,7 +60,3 @@ Open any of your existing projects containing a few detected packages.
 - Click red bug icon to reach the vulnerability details for this package:
 
 .. image:: images/tutorial-find-vulnerabilities-extra-data.png
-
-.. tip::
-    Refer to the complementary :ref:`tutorial_web_ui_review_scan_results` page, to
-    understand this tutorial's scan results/output.