Improve documentation

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne

CHANGELOG.rst

@@ -43,6 +43,28 @@ Package detection:
  - Add support to track installed files for each Package type.
 
 
+v21.8.1
+---------
+
+This is a minor bug fix release primarily for Windows installation.
+There is no feature change.
+
+Installation:
+~~~~~~~~~~~~~~~~~~
+
+- Application installation on Windows works again. This fixes #2610
+- We now build and test app bundles on all supported Python versions: 3.6 to 3.9
+
+
+Thank you to @gunaztar for reporting the #2610 bug
+
+Documentation:
+~~~~~~~~~~~~~~~~~~
+
+- Documentation is updated to reference supported Python versions 3.6 to 3.9
+
+
+
 v21.7.30
 ---------
 

INSTALL.rst

@@ -16,13 +16,12 @@ Prerequisites
 -------------
 
 Before installing ScanCode make sure you have  installed these prerequisites.
-The main one is a Python interpreter.
-Python 3.6 is required for the standard installation.
+The main one is to have Python installed version 3.6, 3.7, 3.8 or 3.9.
 
 - For Linux(Ubuntu): ``sudo apt install python3.6-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev``
-- For MacOS: Install Python 3.6.8 from https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg
-- For Windows: Install Python 3.6.8 from https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe
-- For FreeBSD: (this ineeds to be documented)
+- For MacOS: Install Python 3.x from https://www.python.org/
+- For Windows: Install Python 3.x from https://www.python.org/  using the 64 bits amd64 variant
+- For FreeBSD: (this needs to be documented)
 
 Refer `Prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on all different operating systems and Python versions.
@@ -46,9 +45,9 @@ Use a release download and install as an application
 Advanced installation: pip install a Python PyPI package
 --------------------------------------------------------
 
-- Create a virtual environment for Python 3.6 (of higher) and activate it::
+- Create a virtual environment for Python 3.x and activate it::
 
-    virtualenv -p /usr/bin/python3.6 venv-scancode && source venv-scancode/bin/activate
+    virtualenv -p python3 venv-scancode && source venv-scancode/bin/activate
 
 - Run ``pip install scancode-toolkit[full]``
 

README.rst

@@ -120,8 +120,7 @@ Installation
 ============
 
 Before installing ScanCode make sure that you have installed the prerequisites
-properly. This means installing Python (Python 3.6 or higher is required.
-When installing the app tarball or zip, only Python 3.6 is supported for now).
+properly. This means installing Python (Python 3.6 or higher is required).
 
 See `prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on the support platforms and Python versions.

docs/source/cli-reference/output-format.rst

@@ -29,8 +29,6 @@ following options.
     others. Scancode Workbench and other applications that use Scancode Result data as input accept
     only the ``json`` format.
 
-    .. include::  /rst_snippets/note_snippets/output_json_notdef.rst
-
     The following code performs a scan on the samples directory, and publishes the results in
     ``json`` format::
 

docs/source/contribute/contrib_dev.rst

@@ -3,131 +3,177 @@
 Contributing to Code Development
 ================================
 
+TL;DR:
+
+- Contributions comes as bugs/questions/issues and as pull requests.
+- Source code and runtime data are in the /src/ directory.
+- Test code and test data are in the /tests/ directory.
+- Datasets (inluding licenses) and test data are in /data/ sub-directories.
+- We use DCO signoff in commit messages, like Linux does. 
+- Porting ScanCode to other OS (FreeBSD is supported, etc.) is possible. Enter an issue for help.
+
 See `CONTRIBUTING.rst <https://github.com/nexB/scancode-toolkit/blob/master/CONTRIBUTING.rst>`_
 for details.
 
+
 .. _contrib_code_conven:
 
 Code layout and conventions
 ---------------------------
 
-Source code is in ``src/`` Tests are in ``tests/``.
+Source code is in the ``src/`` directory, tests are in the ``tests/`` directory.
+Miscellaneous scripts and configuration files are in the ``etc/`` directory.
 
-There is one Python package for each major feature under ``src/`` and a corresponding directory
-with the same name under ``tests`` (but this is not a package by design).
+There is one Python package for each major feature under ``src/`` and a
+corresponding directory with the same name under ``tests`` (but this is not a
+package by design as it would not make sense to have a top level "tests" package
+which is a name that's too common).
 
-Each test script is named ``test_XXXX`` and while we love to use ``py.test`` as a test runner,
-most tests have no dependencies on ``py.test``, only on the ``unittest`` module (with the exception
-of some command line tests that depend on pytest monkeypatching capabilities.
+Each test script is named ``test_XXXX``; we prefer organizing tests in subclasses
+of the standard library ``unittest`` module. But we also use plain functions
+that are discovered nicely by ``pytest``.
 
 When source or tests need data files, we store these in a ``data`` subdirectory.
+This is used extensively in tests and also in source code for the reference
+license texts and data and license detection rules files.
 
-We use PEP8 conventions with a relaxed line length that can be up to 90'ish characters long when
-needed to keep the code clear and readable.
+We use PEP8 conventions with a relaxed line length that can be up to 90'ish
+characters long when needed to keep the code clear and readable.
 
-We store pre-built bundled native binaries in ``bin/`` sub-directories of each ``src/`` packages.
-These binaries are organized by OS and architecture. This ensures that ScanCode works out of the box
-either using a checkout or a download, without needing a compiler and toolchain to be installed.
-The corresponding source code for the pre-built binaries are stored in a separate repository at
-https://github.com/nexB/scancode-thirdparty-src.
+We write tests, a lot of tests, thousands of tests.  When finding bugs or adding
+new features, we add tests. See existing test code for examples which form also
+a good specification for the supported features.
 
-We store bundled thirdparty components and libraries in the ``thirdparty`` directory. Python
-libraries are stored as wheels, eventually pre-built if the corresponding wheel is not available
-in the Pypi repository. Some of these components may be advanced builds with bug fixes or advanced
-patches.
+The tests should pass on Linux 64 bits, Windows 64 bits and on
+macOS 10.14 and up. We maintain multiple CI loops with Azure (all OSes) 
+at https://dev.azure.com/nexB/scancode-toolkit/_build and Appveyor (Windows) at
+https://ci.appveyor.com/project/nexB/scancode-toolkit .
 
-We write tests, a lot of tests, thousands of tests. Several tests are data-driven and use data
-files as test input and sometimes data files as test expectation (in this case using either
-JSON or YAML files). The tests should pass on Linux 64 bits, Windows 32 and 64 bits and on
-MacOS 10.9 and up. We maintain two CI loops with Travis (Linux) at
-https://travis-ci.org/nexB/scancode-toolkit and Appveyor (Windows) at
-https://ci.appveyor.com/project/nexB/scancode-toolkit.
 
-When finding bugs or adding new features, we add tests. See existing test code for examples.
+Several tests are data-driven and use data files as test input and sometimes
+data files as test expectation (in this case using either JSON or YAML files);
+a large number of copyright, license and package manifest parsing tests are such
+data-driven tests. 
 
-More info:
 
-- Source code and license datasets are in the /src/ directory.
-- Test code and test data are in the /tests/ directory.
-- Datasets and test data are in /data/ sub-directories.
-- Third-party components are vendored in the /thirdparty/ directory. ScanCode is self contained
-  and should not require network access for installation or configuration of third-party libraries.
-- Additional pre-compiled vendored binaries are stored in bin/ sub-directories of the /src/
-  directory with their sources in this repo: https://github.com/nexB/scancode-thirdparty-src/
-- Porting ScanCode to other OS (FreeBSD, etc.) is possible. Enter an issue for help.
-- Bugs and pull requests are welcomed.
-- See the wiki and CONTRIBUTING.rst for more info.
 
-.. _scancode_toolkit_developement_running_tests:
+.. _scancode_toolkit_development_running_tests:
+
 
 Running tests
 -------------
 
-ScanCode comes with over 13,000 unit tests to ensure detection accuracy and stability across Linux,
-Windows and macOS OSes: we kinda love tests, do we?
+ScanCode comes with over 29,000 unit tests to ensure detection accuracy and
+stability across Linux, Windows and macOS OSes: we kinda love tests, do we?
 
-We use pytest to run the tests: call the ``py.test`` script to run the whole test suite. This is
-installed by ``pytest``, which is bundled with a ScanCode checkout and installed when you
-run ``./configure``).
+We use pytest to run the tests: call the ``pytest`` script to run the whole
+test suite. This is installed with the ``pytest`` package which is installed
+when you run ``./configure --dev``).
 
 If you are running from a fresh git clone and you run ``./configure`` and then
-``source bin/activate`` the ``py.test`` command will be available in your path.
+``source bin/activate`` the ``pytest`` command will be available in your path.
 
-Alternatively, if you have already configured but are not in an activated "virtualenv" the
-``py.test`` command is available under ``<root of your checkout>/bin/py.test``
+Alternatively, if you have already configured but are not in an activated
+"virtualenv" the ``pytest`` command is available under
+``<root of your checkout>/bin/pytest``
 
 (Note: paths here are for POSIX, but mostly the same applies to Windows)
 
-If you have a multiprocessor machine you might want to run the tests in parallel (and faster)
-For instance: ``py.test -n4`` runs the tests on 4 CPUs. We typically run the tests in
-verbose mode with ``py.test -vvs -n4``.
+If you have a multiprocessor machine you might want to run the tests in parallel
+(and faster). For instance: ``pytest -n4`` runs the tests on 4 CPUs. We
+typically run the tests in verbose mode with ``pytest -vvs -n4``.
 
 You can also run a subset of the test suite as shown in the CI configs
 https://github.com/nexB/scancode-toolkit/blob/develop/appveyor.yml#L6 e,g,
-``py.test -n 2 -vvs tests/scancode`` runs only the test scripts present in the ``tests/scancode``
-directory. (You can pass a path to a specific test script file there too).
-
-See also https://docs.pytest.org for details or use the ``py.test -h`` command to show the many
-other options available.
-
-One useful option is to run a select subset of the test functions matching a pattern with the
-``-k`` option, for instance: ``py.test -vvs -k tcpdump`` would only run test functions that contain
-the string "tcpdump" in their name or their class name or module name .
-
-Another useful option after a test run with some failures is to re-run only the failed tests with
-the ``--lf`` option, for instance: ``py.test -vvs --lf`` would only run only test functions that
-failed in the previous run.
-
-.. _contrib_dev_pip_and_configure:
-
-pip requirements and the configure script
------------------------------------------
-
-ScanCode use the ``configure`` and ``configure.bat`` (and ``etc/configure.py`` behind the scenes)
-scripts to install a `virtualenv <https://virtualenv.pypa.io/en/stable/>`_ , install required
-packaged dependencies as `pip <https://github.com/pypa/pip>`_ requirements and more configure tasks
-such that ScanCode can be installed in a self-contained way with no network connectivity required.
-
-Earlier unreleased versions of ScanCode where using ``buildout`` to install and configure
-eventually complex dependencies. We had some improvements that were merged in the upstream
-``buildout`` to support bootstrapping and installing without a network connection and When we
-migrated to use ``pip`` and ``wheels`` as new, improved and faster way to install and configure
-dependencies we missed some of the features of ``buildout`` like the ``recipes``, being able to
-invoke arbitrary Python or shell scripts after installing packages and have scripts or requirements
-that are operating system-specific.
-
-ScanCode requirements and third-party Python libraries
-------------------------------------------------------
+``pytest -n 2 -vvs tests/scancode`` runs only the test scripts present in the
+``tests/scancode`` directory. (You can give the path to a specific test script
+file there too).
+
+See also https://docs.pytest.org for details or use the ``pytest -h`` command
+to show the many other options available.
+
+One useful option is to run a select subset of the test functions matching a
+pattern with the ``-k`` option, for instance: ``pytest -vvs -k tcpdump`` would
+only run test functions that contain the string "tcpdump" in their name or their
+class name or module name.
+
+Another useful option after a test run with some failures is to re-run only the
+failed tests with the ``--lf`` option, for instance: ``pytest -vvs --lf`` would
+only run only test functions that failed in the previous run.
+
+Because we have a lot of tests (over 29,000), we organized theses in test suites
+using pytest markers that are defined in the ``conftest.py`` pytest plugin.
+These are enabled by adding a ``--test-suite`` option to the pytest command.
+
+- ``--test-suite=standard`` is the default and runs a decent but basic test suite
+- ``--test-suite=all`` runs the ``standard`` test and adds a comprehensive test suite
+- ``--test-suite=validate`` runs the ``standra`` and ``all`` test and adds
+  extensive data-driven and data validations (for package, copyright and license
+  detection)
+
+
+
+.. _scancode_toolkit_development_thirdparty_libraries:
+
+Thirdparty libraries and dependencies management
+-----------------------------------------------------
+
+ScanCode uses the ``configure`` and ``configure.bat`` scripts to install a
+`virtualenv <https://virtualenv.pypa.io/en/stable/>`_ , install required
+packaged dependencies using  `setuptools <https://github.com/pypa/setuptools>`_ 
+and such that ScanCode can be installed in a repeatable and consistent manner on
+all OSes and Python versions.
+
+For this we maintain a ``setup.cfg`` with our direct dependencies with loose
+minimum version constraints; and we keep pinned exact versions of these
+dependencies in the ``requirements.txt`` and ``requirements-dev.txt`` (for
+testing and development).
+
+Note: we also have a ``setup-mini.cfg`` used to create a ScanCode PyPI package
+with minimal dependencies (and limited features). This is mostly duplicated
+from ``setup.cfg``.
+
+And to ensure that we also all use well known version of the core virtualenv,
+pip, setuptools and wheel libraries, we use the ``virtualenv.pyz`` Python
+zipp app from https://github.com/pypa/get-virtualenv/tree/master/public and
+store it in the Git repo in the ``etc/thirdparty`` directory.
+
+We bundle pre-built bundled native binaries as plugins which are installed as
+wheels. These binaries are organized by OS and architecture; they ensure that
+ScanCode works out of the box either using a checkout or a download, without
+needing a compiler and toolchain to be installed. 
+
+The corresponding source code and build scripts for all for the
+pre-built binaries are stored in a separate repository at 
+https://github.com/nexB/scancode-plugins
+
+ScanCode app archives should not require network access for installation or
+configuration of its third-party libraries and dependencies. To enable this, 
+we store bundled thirdparty components and  libraries in the ``thirdparty``
+directory of released app archives; this is done at build time. 
+These dependencies are stored as pre-built wheels. These wheels are sometimes
+built by us when there is no wheel available upstream on PyPI. We store all
+these prebuilt wheels with corresponding .ABOUT and .LICENSE files in
+https://github.com/nexB/thirdparty-packages/tree/main/pypi which is published
+for download at  https://thirdparty.aboutcode.org/pypi/
+
+Because this is used by the configure script, all the thirdparty dependencies
+used in ScanCode MUST be available there first. Therefore adding a new
+dependency means requesting a merge/PR in 
+https://github.com/nexB/thirdparty-packages/ first that contains all the
+recursive dependencies.
+
+There are utility scripts in ``etc/release`` that can help with the dependencies
+management process in particular to build or update wheels with native code for
+multiple OSes (Linux, macOS and Windows) and multiple Python versions (3.6+),
+which is not a completely simple operation (and requires eventually 12 wheels
+and one source distribution to be published as we support 3 OSes and 4 Python
+versions).
 
-In a somewhat unconventional way, all the required libraries are bundled aka. Copied in the repo
-itself in the thirdparty/ directory. If ScanCode were only a library it would not make sense. But
-it is first an application and having a well defined frozen set of dependent packages is important
-for an app. The benefit of this approach (combined with the ``configure`` script) means that a mere
-checkout of the repository contains everything needed to run ScanCode except for a
-Python interpreter.
 
 Using ScanCode as a Python library
 ----------------------------------
 
-ScanCode can be used alright as a Python library and is available as as a Python wheel in Pypi and
-installed with ``pip install scancode-toolkit``.
+ScanCode can be used alright as a Python library and is available as as a
+Python wheel in Pypi and installed with ``pip install scancode-toolkit`` or
+``pip install scancode-toolkit-mini``.

docs/source/contribute/contrib_doc.rst

@@ -18,7 +18,7 @@ That will create an ``/scancode-toolkit`` directory in your working directory.
 Now you can install the dependencies in a virtualenv::
 
     cd scancode-toolkit
-    virtualenv -p /usr/bin/python3.6 docs-venv
+    virtualenv -p /usr/bin/python3 docs-venv
     source docs-venv/bin/activate
 
 Now, the following prerequisites are installed

docs/source/contribute/long_running_issues.rst

@@ -22,8 +22,6 @@ are considering, what the long-term goals are.
 
 - :ref:`roadmap`
 - `Milestones <https://github.com/nexB/scancode-toolkit/milestones>`_
-- `v3.1 <https://github.com/nexB/scancode-toolkit/milestone/11>`_
-- `v3.2 <https://github.com/nexB/scancode-toolkit/milestone/12>`_
 - `Documentation Roadmap <https://github.com/nexB/scancode-toolkit/issues/1824>`_
 
 .. _good_first_issue:

docs/source/explanations/overview.rst

@@ -3,8 +3,8 @@ Overview
 
 .. _explain_how_scancode_works:
 
-How does ScanCode work?
------------------------
+How does ScanCode detect licenses?
+--------------------------------------
 
 For license detection, ScanCode uses a (large) number of license texts and license detection
 'rules' that are compiled in a search index. When scanning, the text of the target file is