Merge pull request #3 from nexB/develop

Updating my fork

Author: akugarg

.bumpversion.cfg

@@ -1,10 +1,35 @@
 Changelog
 =========
 
+
 v21.x (next)
 ------------
 
 
+v21.2.25
+--------
+
+Installation:
+
+ - Resolve reported installation issues on macOS, Windows and Linux
+ - Stop using extras for a default wheel installation
+ - Build new scancode-toolkit-mini package with limited dependencies for use
+   when packaging in distros and similar
+ - The new Dockerfile will be create smaller images and containers
+
+License scanning:
+
+ - Over 150 and and updated licenses
+ - Support the latest SPDX license list v3.11
+ - Improve license detection accuracy with over 740 new and improved license
+   detection rules
+ - Fix license cache handling issues
+
+Misc.:
+
+ - Update extractcode, typecode and their native dependencies for better support
+   of latests versions of macOS.
+
 
 v21.2.9
 -------
@@ -26,13 +51,16 @@ Operating system support:
 
 License scanning:
 
- - Improve license detection accuracy with over 8400 new license detection fules
+ - Improve license detection accuracy with over 8400 new license detection rules
    added or updated
  - Remove the previously deprecated --license-diag option
  - Include pre-built license index in release archives to speed up start #988
  - Use SPDX LicenseRef-scancode namespace for all licenses keys not in SPDX
  - Replace DEJACODE_LICENSE_URL with SCANCODE_LICENSEDB_URL at
    https://scancode-licensedb.aboutcode.org #2165
+ - Add new license flag in license detection results "is_license_intro" that
+   is used to indicate that a license rule is a short license introduction
+   statement (that typically may be reported as some unknown license)
 
 Package scanning:
 
@@ -58,7 +86,6 @@ Misc.:
  - Updated scancode.bat to handle % signs in the arguments #1876
 
 
-
 v3.2.3 (2020-10-27)
 -------------------
 

CHANGELOG.rst

@@ -21,12 +21,13 @@
 # limitations under the License.
 #
 
-FROM python:3.6 
+FROM python:3.6-slim-buster 
 
 # Requirements as per https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html
 RUN apt-get update \
- && apt-get install -y bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev \
- && rm -rf /var/lib/apt/lists/*
+ && apt-get install -y bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libgomp1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
 # Create directory for scancode sources
 RUN mkdir scancode-toolkit

Dockerfile

@@ -41,11 +41,18 @@ Why use ScanCode?
   companion `ScanCode workbench GUI app <https://github.com/nexB/scancode-workbench>`_
   to review and display scan results, statistics and graphics.
 
+- You can also organize and run ScanCode server-side with the
+  companion `ScanCode.io web app <https://github.com/nexB/scancodeio>`_
+  to organize and store multiple scan projects including scripting scanning piplines.
+
+
 - ScanCode is **actively maintained**, has a **growing users and contributors
   community**.
 
 - ScanCode is heavily **tested** with an automated test suite of over **20,000 tests**.
 
+- ScanCode has an extensive and growing documentation.
+
 See our `roadmap <https://scancode-toolkit.readthedocs.io/en/latest/contribute/roadmap.html>`_
 for upcoming features.
 
@@ -91,7 +98,9 @@ Other Important Documentation Pages:
 - An exhaustive list of `all available options <https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/list-options.html>`_
 
 - Documentation on `Contributing to Code Development <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_dev.html>`_
+
 - Documentation on `Plugin Architecture <https://scancode-toolkit.readthedocs.io/en/latest/plugins/plugin_arch.html>`_
+
 - `FAQ <https://scancode-toolkit.readthedocs.io/en/latest/misc/faq.html>`_
 
 See also https://aboutcode.org for related companion projects and tools.
@@ -101,7 +110,7 @@ Installation
 ============
 
 Before installing ScanCode make sure that you have installed the prerequisites
-properly. This means installing Python (Python 3.6 is required in most cases).
+properly. This means installing Python (Python 3.6+ is required).
 See `prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on the support platforms and Python versions.
 
@@ -110,17 +119,16 @@ There are a few common ways to `install ScanCode <https://scancode-toolkit.readt
 - `*Recommended* installation as an application: Download a release archive, extract and run.
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-as-an-application-downloading-releases>`_
 
-- `Development installation from source code: git clone 
+- `Development installation from source code using a git clone
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-from-source-code-git-clone>`_
 
-- `Development installation as a library: via pip install scancode-toolkit[full] 
+- `Development installation as a library with "pip install scancode-toolkit"
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#pip-install>`_
 
-- `Run as a Docker container: git clone and docker run
+- `Run in a Docker container with a git clone and "docker run"
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-via-docker>`_
 
 
-
 Quick Start
 ===========
 
@@ -146,17 +154,17 @@ for more information.
 
 You can also refer to the `command line options synopsis
 <https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/synopsis.html>`_
-and an exhaustive list of `all available options
+and an exhaustive list of `all available command line options
 <https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/list-options.html>`_.
 
 
 Archive extraction
 ==================
 
-ScanCode does not extract by default files from tarballs, zip files, and
+By default ScanCode does not extract files from tarballs, zip files, and
 other archives as part of the scan. The archives that exist in a codebase
-must be extracted before running a scan: the bundled utility `extractcode`
-is a mostly-universal archive extractor. For example, this command will
+must be extracted before running a scan: `extractcode` is a bundled utility
+behaving as a mostly-universal archive extractor. For example, this command will
 recursively extract the mytar.tar.bz2 tarball in the mytar.tar.bz2-extract
 directory::
 
@@ -198,7 +206,7 @@ License
 =======
 
 * Apache-2.0 as the overall license
-* CC-BY-4.0 for reference datasets (initially was in the Public Domain.
+* CC-BY-4.0 for reference datasets (initially was in the Public Domain).
 * Multiple other secondary permissive or copyleft licenses (LGPL, MIT,
   BSD, GPL 2/3, etc.) for third-party components.
 

README.rst

@@ -17,22 +17,22 @@ jobs:
           python_versions: ['3.6']
           test_suites:
               misc: |
-                bin/pytest -n 3 -vvs --reruns=2 --test-suite=all \
+                bin/pytest -n 3 -vvs --test-suite=all \
                   --ignore=tests/scancode \
                   --ignore=tests/cluecode \
                   --ignore=tests/licensedcode
 
               scancode: |
-                bin/pytest --reruns=2 -vvs --test-suite=all \
+                bin/pytest -vvs --test-suite=all \
                   tests/scancode
 
               cluecode: |
-                bin/pytest -n 3 --reruns=2 -vvs --test-suite=all \
+                bin/pytest -n 3 -vvs --test-suite=all \
                   tests/cluecode \
                   --cov=src --cov-report=term
 
               license_base: |
-                bin/pytest -n 3 --reruns=2 -vvs --test-suite=all \
+                bin/pytest -n 3 -vvs --test-suite=all \
                   --ignore=tests/licensedcode/test_zzzz_cache.py \
                   --ignore=tests/licensedcode/test_detection_datadriven_external.py \
                   --ignore=tests/licensedcode/test_detection_datadriven1.py \
@@ -72,39 +72,108 @@ jobs:
                   -k TestValidateLicenseExtended
 
               license_cache: |
-                bin/pytest -n 3 --reruns=2 -vvs --test-suite=all \
+                bin/pytest -n 3 -vvs --test-suite=all \
                   tests/licensedcode/test_zzzz_cache.py
 
     - template: etc/ci/azure-mac.yml
       parameters:
-          job_name: macos1015_py36
+          job_name: macos_1014_py3x
+          image_name: macos-10.14
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
+
+    - template: etc/ci/azure-mac.yml
+      parameters:
+          job_name: macos_1015_py3x
           image_name: macos-10.15
-          python_versions: ['3.6']
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: bin/pytest -n 3 -vvs --reruns=2
+              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
+
+#    - template: etc/ci/azure-mac.yml
+#      parameters:
+#          job_name: macos_110_py3x
+#          image_name: macos-11.0
+#          python_versions: ['3.6', '3.7', '3.8', '3.9']
+#          test_suites:
+#              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
 
     - template: etc/ci/azure-win.yml
       parameters:
-          job_name: Win2016_py36
+          job_name: win_2016_py3x
           image_name: vs2017-win2016
-          python_versions: ['3.6']
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          python_architecture: x64
+          test_suites:
+              all: Scripts\pytest -n 2 -vvs tests\scancode\test_cli.py
+
+
+    - template: etc/ci/azure-win.yml
+      parameters:
+          job_name: win_2019_py3x
+          image_name: windows-2019
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
           python_architecture: x64
           test_suites:
-              all: Scripts\pytest -n 3 -vvs --reruns=2
+              all: Scripts\pytest -n 2 -vvs tests\scancode\test_cli.py
+
+
+    - template: etc/ci/azure-linux.yml
+      parameters:
+          job_name: vm_ubuntu16_py3x
+          image_name: ubuntu-16.04
+          python_versions: ['3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
+
+    - template: etc/ci/azure-linux.yml
+      parameters:
+          job_name: vm_ubuntu18_py3x
+          image_name: ubuntu-18.04
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
+
+    - template: etc/ci/azure-linux.yml
+      parameters:
+          job_name: vm_ubuntu20_py3x
+          image_name: ubuntu-20.04
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
 
 ################################################################################
 # These jobs are using containers and their own Python 3.6
 ################################################################################
 
-    - template: etc/ci/azure-container-deb.yml
-      parameters:
-          job_name: ubuntu18_py36
-          container: ubuntu:bionic
-          python_path: python3.6
-          python_version: '3.6'
-          install_python: sudo apt-get install -y python3.6 python3.6-venv python3.6-dev
-          test_suite_label: all
-          test_suite: bin/pytest -n 3 -vvs --reruns=2
+#    - template: etc/ci/azure-container-deb.yml
+#      parameters:
+#          job_name: ubuntu18_py36
+#          container: ubuntu:bionic
+#          python_path: python3.6
+#          python_version: '3.6'
+#          install_python: sudo apt-get install -y python3.6 python3.6-venv python3.6-dev
+#          test_suite_label: all
+#          test_suite: bin/pytest -n 3 -vvs tests/scancode
+
+
+#    - template: etc/ci/azure-container-deb.yml
+#      parameters:
+#          job_name: ubuntu20_py38
+#          container: ubuntu:focal
+#          python_path: python3.8
+#          python_version: '3.8'
+#          install_python: sudo apt-get install -y python3.8 python3.8-venv python3.8-dev
+#          test_suite_label: all
+#          test_suite: bin/pytest -n 3 -vvs tests/scancode
+
 
 #    - template: etc/ci/azure-container-rpm.yml
 #      parameters:
@@ -117,7 +186,7 @@ jobs:
 #              sudo yum install -y epel-release
 #              sudo yum install -y python36 python36-devel python36-virtualenv
 #          test_suite_label: all
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 #
 #    - template: etc/ci/azure-container-rpm.yml
 #      parameters:
@@ -128,7 +197,7 @@ jobs:
 #          install_packages: echo "No extra packages for now"
 #          install_python: echo "Python is pre-installed"
 #          test_suite_label: all
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 #
 #    - template: etc/ci/azure-container-rpm.yml
 #      parameters:
@@ -139,7 +208,7 @@ jobs:
 #          install_packages: echo "No extra packages for now"
 #          install_python: echo "Python is pre-installed"
 #          test_suite_label: all
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 
 
 ################################################################################
@@ -157,7 +226,7 @@ jobs:
 #              set -e -x
 #              sudo dnf install -y python3 python3-devel python3-virtualenv
 #          test_suite_label: all
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 #
 #    - template: etc/ci/azure-container-rpm.yml
 #      parameters:
@@ -168,7 +237,7 @@ jobs:
 #          install_packages: echo "No extra packages for now"
 #          install_python: echo "Python is pre-installed"
 #          test_suite_label: all
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 
 
 ################################################################################
@@ -199,7 +268,7 @@ jobs:
 #            make -j8
 #            sudo make altinstall
 #          test_suite_label: basic tests
-#          test_suite: bin/pytest -n 2 -vvs --reruns=3
+#          test_suite: bin/pytest -n 2 -vvs
 
 
 ########################################################################

azure-pipelines.yml

@@ -51,8 +51,8 @@ export THIRDPARTY_LINKS="https://thirdparty.aboutcode.org/pypi"
 
 # requirements used by default or with --dev.
 # note the use of constraints with -c 
-REQUIREMENTS="--editable .[full] --constraint requirements.txt"
-DEV_REQUIREMENTS="--editable .[full] --constraint requirements.txt --editable .[dev] --constraint requirements-dev.txt"
+REQUIREMENTS="--editable . --constraint requirements.txt"
+DEV_REQUIREMENTS="--editable .[dev] --constraint requirements.txt --constraint requirements-dev.txt"
 
 # default supported Python version
 if [[ "$CONFIGURE_SUPPORTED_PYTHON" == "" ]]; then