diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 237d9e0d62..a0f52a6c21 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -18,6 +18,9 @@ v33.0.0 (unreleased) integration in the documentation. https://github.com/nexB/scancode.io/issues/600 +- Rewrite the CLI tutorials for a Docker-based installation. + https://github.com/nexB/scancode.io/issues/440 + - Use CodebaseResource ``path`` instead of ``id`` as slug_field in URL navigation. https://github.com/nexB/scancode.io/issues/242 diff --git a/docs/installation.rst b/docs/installation.rst index 4bec4f3a31..af768b3a4c 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -53,7 +53,9 @@ create an **environment file**, and **build the Docker image**:: .. note:: You need to rebuild the image whenever ScanCode.io's source code has been - modified or updated. + modified or updated:: + + docker compose build Run the App ^^^^^^^^^^^ @@ -87,7 +89,7 @@ An overview of the web application usage is available at :ref:`user_interface`. and ``CSRF_TRUSTED_ORIGINS`` settings need to be provided in your ``.env`` file, for example:: - ALLOWED_HOSTS=.your-domain.com,127.0.0.1 + ALLOWED_HOSTS=.your-domain.com,localhost,127.0.0.1 CSRF_TRUSTED_ORIGINS=https://*.your-domain.com,http://127.0.0.1:8001 Refer to `ALLOWED_HOSTS settings `_ - Have **Shell access** on the machine where ScanCode.io is installed Instructions ------------ -- Open a shell in the ScanCode.io installation directory and activate the - virtual environment - **virtualenv**: +- Create a new directory in your home directory that will be used to put the input code + to be scanned. + +.. code-block:: console + + $ mkdir -p ~/codedrop/ + +- Download the following **package archive** and save it to the :guilabel:`~/codedrop/` + directory: `asgiref-3.3.0-py3-none-any.whl `_ .. code-block:: console - $ source bin/activate + $ curl https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl --output ~/codedrop/asgiref-3.3.0-py3-none-any.whl + +- Create an alias to the ``scanpipe`` command executed through the + ``docker compose`` command line interface with: .. code-block:: console - >> (scancodeio) $ + $ alias scanpipe="docker compose -f ${PWD}/docker-compose.yml run --volume ~/codedrop/:/codedrop:ro web scanpipe" - Create a new project named ``asgiref``: @@ -41,18 +51,18 @@ Instructions .. code-block:: console - >> Project asgiref created with work directory projects/asgiref-072c89db + >> Project asgiref created with work directory /var/scancodeio/workspace/projects/asgiref-35519104 - Add the package archive to the project workspace's :guilabel:`input/` directory: .. code-block:: bash - $ scanpipe add-input --project asgiref --input-file ~/asgiref-3.3.0-py3-none-any.whl + $ scanpipe add-input --project asgiref --input-file /codedrop/asgiref-3.3.0-py3-none-any.whl .. code-block:: console - >> File(s) copied to the project inputs directory: + >> File copied to the project inputs directory: - asgiref-3.3.0-py3-none-any.whl - Add the ``scan_codebase`` pipeline to your project: @@ -63,7 +73,7 @@ Instructions .. code-block:: console - >> Pipeline(s) added to the project + >> Pipeline scan_codebase added to the project .. note:: The content of the :guilabel:`input/` directory will be copied in the @@ -83,26 +93,38 @@ Instructions .. code-block:: console >> Pipeline scan_codebase run in progress.. - 2021-07-12 17:45:53.85 Pipeline [scan_codebase] starting - 2021-07-12 17:45:53.85 Step [copy_inputs_to_codebase_directory] starting - 2021-07-12 17:45:53.86 Step [copy_inputs_to_codebase_directory] completed in 0.00 seconds - 2021-07-12 17:45:53.86 Step [extract_archives] starting + Pipeline [scan_codebase] starting + Step [copy_inputs_to_codebase_directory] starting + Step [copy_inputs_to_codebase_directory] completed in 0.00 seconds + Step [extract_archives] starting [...] - 2021-07-12 17:46:01.61 Pipeline completed + Pipeline completed + scan_codebase successfully executed on project asgiref -- Finally, you can view your scan results in JSON or CSV file formats inside - the project's :guilabel:`output/` directory. +- Finally, export the scan results as JSON format:: + + $ scanpipe output --project asgiref --format json --print > asgiref-3.3.0_results.json .. tip:: The ``inputs`` and ``pipelines`` can be provided at the same time when calling the ``create-project`` command. For instance, the following command - will create a new project named ``asgiref``, add the package archive as the + will create a new project named ``asgiref2``, add the package archive as the project input, add the ``scan_codebase`` pipeline to the project, and execute it: .. code-block:: bash - $ scanpipe create-project asgiref \ - --input-file ~/asgiref-3.3.0-py3-none-any.whl \ + $ scanpipe create-project asgiref2 \ + --input-file /codedrop/asgiref-3.3.0-py3-none-any.whl \ --pipeline scan_codebase \ --execute + +.. code-block:: console + + >> Project asgiref2 created with work directory /var/scancodeio/workspace/projects/asgiref2-bea7a5e9 + File copied to the project inputs directory: + - asgiref-3.3.0-py3-none-any.whl + Start the scan_codebase pipeline execution... + [...] + Pipeline completed + scan_codebase successfully executed on project asgiref2 diff --git a/docs/tutorial_cli_analyze_docker_image.rst b/docs/tutorial_cli_analyze_docker_image.rst index 05eefefdda..81fbc2e735 100644 --- a/docs/tutorial_cli_analyze_docker_image.rst +++ b/docs/tutorial_cli_analyze_docker_image.rst @@ -8,9 +8,9 @@ a test Docker image by following the steps below and, along the way, learn some of the ScanCode.io basic commands. .. note:: - This tutorial assumes you have a current version of ScanCode.io installed - locally on your machine. If you do not have it installed, - see our :ref:`installation` guide for instructions. + This tutorial assumes you have a recent version of ScanCode.io installed + locally on your machine and **running with Docker**. + If you do not have it installed, see our :ref:`installation` guide for instructions. Requirements ------------ @@ -18,22 +18,32 @@ Requirements To successfully complete this tutorial, you first need to: - Install **ScanCode.io** locally -- Download the following **test Docker image** and save it to your home directory: `30-alpine-nickolashkraus-staticbox-latest.tar `_ - Have **Shell access** on the machine where ScanCode.io is installed Instructions ------------ -- Open a shell in the ScanCode.io installation directory and activate the - virtual environment - **virtualenv**: +- Create a new directory in your home directory that will be used to put the input code + to be scanned. .. code-block:: console - $ source bin/activate + $ mkdir -p ~/codedrop/ + +- Download the following **test Docker image** and save it to the :guilabel:`~/codedrop/` + directory: `30-alpine-nickolashkraus-staticbox-latest.tar + `_ + +.. code-block:: console + + $ curl https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar --output ~/codedrop/30-alpine-nickolashkraus-staticbox-latest.tar + +- Create an alias to the ``scanpipe`` command executed through the + ``docker compose`` command line interface with: .. code-block:: console - >> (scancodeio) $ + $ alias scanpipe="docker compose -f ${PWD}/docker-compose.yml run --volume ~/codedrop/:/codedrop:ro web scanpipe" - Create a new project named ``staticbox``: @@ -43,12 +53,12 @@ Instructions .. code-block:: console - >> Project staticbox created with work directory projects/staticbox-d4ed9405 + >> Project staticbox created with work directory /var/scancodeio/workspace/projects/staticbox-d4ed9405 .. note:: New projects work directory are created inside the location defined in :ref:`scancodeio_settings_workspace_location` setting. - Default to a :guilabel:`var/` directory in the local ScanCode.io codebase. + Default to the :guilabel:`/var/scancodeio/workspace/` directory. - Add the test Docker image tarball to the project workspace's :guilabel:`input/` directory: @@ -56,17 +66,16 @@ Instructions .. code-block:: bash $ scanpipe add-input --project staticbox \ - --input-file ~/30-alpine-nickolashkraus-staticbox-latest.tar + --input-file /codedrop/30-alpine-nickolashkraus-staticbox-latest.tar .. code-block:: console - >> File(s) copied to the project inputs directory: + >> File copied to the project inputs directory: - 30-alpine-nickolashkraus-staticbox-latest.tar .. note:: The command output will let you know that the Docker image file was copied to the project's :guilabel:`input/` directory. - You can also navigate to this directory and confirm your file is there. Alternatively, you can copy files manually to the :guilabel:`input/` directory to include entire directories. @@ -78,7 +87,7 @@ Instructions .. code-block:: console - >> Pipeline(s) added to the project + >> Pipeline docker added to the project - Check the status of the pipeline added to your project: @@ -107,12 +116,13 @@ Instructions .. code-block:: console >> Pipeline docker run in progress... - 2021-07-07 10:39:26.49 Pipeline [docker] starting - 2021-07-07 10:39:26.53 Step [extract_images] starting - 2021-07-07 10:39:26.71 Step [extract_images] completed in 0.18 seconds - 2021-07-07 10:39:26.71 Step [extract_layers] starting + Pipeline [docker] starting + Step [extract_images] starting + Step [extract_images] completed in 0.18 seconds + Step [extract_layers] starting [...] - 2021-07-07 10:39:31.39 Pipeline completed + Pipeline completed + docker successfully executed on project staticbox - Executing the ``show-pipeline`` command again will also confirm the success of the pipeline execution - **"[SUCCESS] docker"** status: @@ -129,14 +139,10 @@ Instructions .. code-block:: console - $ scanpipe output --project staticbox --format json - -.. code-block:: console - - >> projects/staticbox-d4ed9405/output/results-2021-07-07-08-54-02.json + $ scanpipe output --project staticbox --format json --print > staticbox_results.json -- Finally, open the ``output/results-.json`` file in your preferred - text editor/file viewer. +- Finally, open the ``staticbox_results.json`` file in your preferred text + editor/file viewer. .. note:: To understand the output of the pipeline execution, see our :ref:`output_files` @@ -145,7 +151,7 @@ Instructions .. tip:: The ``inputs`` and ``pipelines`` can be provided directly at once when calling the ``create-project`` command. - An ``execute`` option is also available to start the pipeline execution right + The ``--execute`` option is also available to start the pipeline execution right after the project creation. For example, the following command will create a project named ``staticbox2``, download the test Docker image to the project's :guilabel:`input/` @@ -153,7 +159,7 @@ Instructions .. code-block:: bash - $ scanpipe create-project staticbox2 \ - --input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar \ - --pipeline docker \ - --execute + $ scanpipe create-project staticbox2 \ + --input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar \ + --pipeline docker \ + --execute diff --git a/docs/tutorial_vulnerablecode_integration.rst b/docs/tutorial_vulnerablecode_integration.rst index e236b45ab2..e1aa2a9010 100644 --- a/docs/tutorial_vulnerablecode_integration.rst +++ b/docs/tutorial_vulnerablecode_integration.rst @@ -60,7 +60,3 @@ Open any of your existing projects containing a few detected packages. - Click red bug icon to reach the vulnerability details for this package: .. image:: images/tutorial-find-vulnerabilities-extra-data.png - -.. tip:: - Refer to the complementary :ref:`tutorial_web_ui_review_scan_results` page, to - understand this tutorial's scan results/output. diff --git a/docs/tutorial_web_ui_analyze_docker_image.rst b/docs/tutorial_web_ui_analyze_docker_image.rst index 1d04b3f4b0..5374e3226d 100644 --- a/docs/tutorial_web_ui_analyze_docker_image.rst +++ b/docs/tutorial_web_ui_analyze_docker_image.rst @@ -23,7 +23,7 @@ We’ll assume that you have: - Installed **ScanCode.io** locally - Access to the web application from your preferred browser on http://localhost/ or - http://127.0.0.1:8001/ if you run on a local development setup. + http://localhost:8001/ if you run on a local development setup. .. tip:: You can view our :ref:`user_interface` section for general information about diff --git a/docs/tutorial_web_ui_review_scan_results.rst b/docs/tutorial_web_ui_review_scan_results.rst index 17da48480b..508937da1d 100644 --- a/docs/tutorial_web_ui_review_scan_results.rst +++ b/docs/tutorial_web_ui_review_scan_results.rst @@ -83,3 +83,7 @@ new window with some extra pipeline-specific details, such start and end date, launch, execution time and status, run log, etc. .. image:: images/tutorial-web-ui-run-log-modal.png + +.. tip:: + The next tutorial chapter :ref:`tutorial_cli_analyze_docker_image` will explore + the interaction with ScanCode.io through a command line interface. diff --git a/docs/user-interface.rst b/docs/user-interface.rst index 362dfec074..5e9afe0d67 100644 --- a/docs/user-interface.rst +++ b/docs/user-interface.rst @@ -8,7 +8,7 @@ create and manage your projects from a browser. You’ll get access to this visual interface when you successfully install ScanCode.io locally. To access the web application, open your web browser and visit http://localhost/ or -http://127.0.0.1:8001/ if you run on a local development setup. +http://localhost:8001/ if you run on a local development setup. .. note:: All the capabilities offered by the ScanCode.io Web Interface are also available as diff --git a/scanpipe/management/commands/__init__.py b/scanpipe/management/commands/__init__.py index 0b94e66e99..04acd77d47 100644 --- a/scanpipe/management/commands/__init__.py +++ b/scanpipe/management/commands/__init__.py @@ -26,6 +26,7 @@ from django.core.exceptions import ObjectDoesNotExist from django.core.management.base import BaseCommand from django.core.management.base import CommandError +from django.template.defaultfilters import pluralize from scanpipe.models import CodebaseResource from scanpipe.models import DiscoveredPackage @@ -154,7 +155,7 @@ def handle_input_files(self, inputs_files): copied.append(filename) self.project.add_input_source(filename, source="uploaded", save=True) - msg = "File(s) copied to the project inputs directory:" + msg = f"File{pluralize(inputs_files)} copied to the project inputs directory:" self.stdout.write(msg, self.style.SUCCESS) msg = "\n".join(["- " + filename for filename in copied]) self.stdout.write(msg) diff --git a/scanpipe/management/commands/add-pipeline.py b/scanpipe/management/commands/add-pipeline.py index eab782e630..6d0ec3c983 100644 --- a/scanpipe/management/commands/add-pipeline.py +++ b/scanpipe/management/commands/add-pipeline.py @@ -20,6 +20,8 @@ # ScanCode.io is a free software code scanning tool from nexB Inc. and others. # Visit https://github.com/nexB/scancode.io for support and download. +from django.template.defaultfilters import pluralize + from scanpipe.management.commands import ProjectCommand from scanpipe.management.commands import validate_pipelines @@ -43,5 +45,8 @@ def handle(self, *pipeline_names, **options): for pipeline_name in pipeline_names: self.project.add_pipeline(pipeline_name) - msg = "Pipeline(s) added to the project" + msg = ( + f"Pipeline{pluralize(pipeline_names)} {', '.join(pipeline_names)} " + f"added to the project" + ) self.stdout.write(msg, self.style.SUCCESS) diff --git a/scanpipe/management/commands/output.py b/scanpipe/management/commands/output.py index 5f84a698d4..240d8c23b6 100644 --- a/scanpipe/management/commands/output.py +++ b/scanpipe/management/commands/output.py @@ -20,35 +20,51 @@ # ScanCode.io is a free software code scanning tool from nexB Inc. and others. # Visit https://github.com/nexB/scancode.io for support and download. +from django.core.management.base import CommandError + from scanpipe.management.commands import ProjectCommand -from scanpipe.pipes.output import to_csv -from scanpipe.pipes.output import to_json -from scanpipe.pipes.output import to_xlsx +from scanpipe.pipes import output class Command(ProjectCommand): - help = "Output project results as JSON, CSV, or XLSX." + help = "Output project results as JSON, XLSX, SPDX, and CycloneDX." def add_arguments(self, parser): super().add_arguments(parser) parser.add_argument( "--format", default="json", - choices=["json", "csv", "xlsx"], + choices=["json", "csv", "xlsx", "spdx", "cyclonedx"], help="Specifies the output serialization format for the results.", ) + parser.add_argument( + "--print", + action="store_true", + help="Print the output to stdout.", + ) def handle(self, *args, **options): super().handle(*args, **options) + print_to_stdout = options["print"] + format = options["format"] output_function = { - "json": to_json, - "csv": to_csv, - "xlsx": to_xlsx, - }.get(options["format"]) + "json": output.to_json, + "csv": output.to_csv, + "xlsx": output.to_xlsx, + "spdx": output.to_spdx, + "cyclonedx": output.to_cyclonedx, + }.get(format) + + if print_to_stdout and format in ["xlsx", "csv"]: + raise CommandError("--print is not compatible with xlsx and csv formats.") output_file = output_function(self.project) if isinstance(output_file, list): output_file = "\n".join([str(path) for path in output_file]) - self.stdout.write(str(output_file), self.style.SUCCESS) + + if options["print"]: + self.stdout.write(output_file.read_text()) + else: + self.stdout.write(str(output_file), self.style.SUCCESS) diff --git a/scanpipe/tests/test_commands.py b/scanpipe/tests/test_commands.py index 3a77fd0efc..52ff5699ba 100644 --- a/scanpipe/tests/test_commands.py +++ b/scanpipe/tests/test_commands.py @@ -199,7 +199,7 @@ def test_scanpipe_management_command_add_input_file(self): options.extend(["--project", project.name]) call_command("add-input", *options, stdout=out) - self.assertIn("File(s) copied to the project inputs directory", out.getvalue()) + self.assertIn("Files copied to the project inputs directory", out.getvalue()) expected = sorted(["test_commands.py", "test_models.py"]) self.assertEqual(expected, sorted(project.input_files)) @@ -249,7 +249,9 @@ def test_scanpipe_management_command_add_pipeline(self): options.extend(["--project", project.name]) call_command("add-pipeline", *options, stdout=out) - self.assertIn("Pipeline(s) added to the project", out.getvalue()) + self.assertIn( + "Pipelines docker, root_filesystems added to the project", out.getvalue() + ) self.assertEqual(pipelines, [run.pipeline_name for run in project.runs.all()]) options = ["--project", project.name, "non-existing"] @@ -411,22 +413,39 @@ def test_scanpipe_management_command_output(self): self.assertIn(filename, project.output_root) out = StringIO() + options = ["--project", project.name, "--no-color"] options.extend(["--format", "csv"]) call_command("output", *options, stdout=out) out_value = out.getvalue().strip() for output_file in out_value.split("\n"): - filename = out_value.split("/")[-1] + filename = output_file.split("/")[-1] self.assertIn(filename, project.output_root) out = StringIO() + options = ["--project", project.name, "--no-color"] options.extend(["--format", "WRONG"]) message = ( "Error: argument --format: invalid choice: 'WRONG' " - "(choose from 'json', 'csv', 'xlsx')" + "(choose from 'json', 'csv', 'xlsx', 'spdx', 'cyclonedx')" ) with self.assertRaisesMessage(CommandError, message): call_command("output", *options, stdout=out) + out = StringIO() + options = ["--project", project.name, "--no-color"] + options.extend(["--format", "xlsx", "--print"]) + message = "--print is not compatible with xlsx and csv formats." + with self.assertRaisesMessage(CommandError, message): + call_command("output", *options, stdout=out) + + out = StringIO() + options = ["--project", project.name, "--no-color"] + options.extend(["--format", "json", "--print"]) + call_command("output", *options, stdout=out) + out_value = out.getvalue().strip() + self.assertIn('"tool_name": "scanpipe"', out_value) + self.assertIn('"notice": "Generated with ScanCode.io', out_value) + def test_scanpipe_management_command_delete_project(self): project = Project.objects.create(name="my_project") work_path = project.work_path