Merge latest develop

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne

.github/ISSUE_TEMPLATE/--lic-det-bug-report.md

@@ -0,0 +1,94 @@
+---
+name: "\U0001F41B License Detection Bug report"
+about: Create a Report for a License Detection Bug
+title: ''
+labels: 'bug', 'license scan', 'new and improved data'
+assignees: ''
+
+---
+
+<!-- 
+Please fill out as much of the below template and delete unnecessary text.
+Sample License Detection Bug Reports 
+	- https://github.com/nexB/scancode-toolkit/issues/2126 
+	- https://github.com/nexB/scancode-toolkit/issues/2266
+Markdown Styling - https://commonmark.org/help/
+-->
+
+### Description
+
+> Please leave a brief description of the License Detection Bug:
+
+
+### Where to get the File/Package
+
+> Link to Public Repository/Package Index, if there are multiple files that has license detection errors
+> Also list some example files. If the bug is in a single file then link to the file. 
+
+
+
+## Scan Reports 
+
+> What scancode command was used?
+
+
+> Scan Outputs (Preferably of a whole file in [json-pretty-print](https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/output-format.html#json-pp-file))
+> Note:- 
+> If possible please use the `--license-text` and `--license-text-diagonostics` [command line options](https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/basic-options.html#license-text-diagnostics-options) 
+> as it helps understand which parts were wrongly matched and gives us more insight into the bug
+> Upload your results.json file on this issue [like this](https://docs.github.com/en/enterprise/2.16/user/github/managing-your-work-on-github/file-attachments-on-issues-and-pull-requests)
+
+
+
+### System/Scancode Configuration
+
+> For bug reports, it really helps us to know:
+
+* What OS are you running on? 
+> (Windows/MacOS/Linux)
+
+
+* What version of scancode-toolkit was used to generate the scan file?
+> `./scancode --version` 
+
+
+* What installation method was used to install/run scancode? 
+> (pip/source download/other)
+
+
+## Links to Rule or excerpts of Rule Texts (If Possible)
+
+> Which Licenses/Rules were wrongly matched (if Applicable)
+> 
+> [Rule Directory](https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules) and [License Directory](https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses)
+
+
+
+## Select the Relevant Issue Category(s)
+
+> [ ] - Unselected, [x] - Selected
+
+-  [ ] Unknown License 
+-  [ ] False Positive
+-  [ ] Multiple Detections, Some Wrong
+-  [ ] License Version Mismatch
+-  [ ] New License
+-  [ ] Change in License/Rule Attributes
+-  [ ] Edit existing Rule
+-  [ ] From [`scancode-results-analyzer`](https://github.com/nexB/scancode-results-analyzer) 
+
+
+## Comments on the License Detection Bug
+
+> What should have been detected instead, why is it a bug.
+
+
+
+## Comments (if any) on How to Add a New Rule/Modify an existing Rule for solving this Bug
+
+> Refer the documentation page at [Add new license](https://scancode-toolkit.readthedocs.io/en/latest/how-to-guides/add_new_license.html) and [Add new rule](https://scancode-toolkit.readthedocs.io/en/latest/how-to-guides/add_new_license_detection_rule.html) 
+
+
+<!-- 
+Your help makes ScanCode Toolkit better! We *deeply* appreciate your help in improving ScanCode Toolkit.
+-->

.gitignore

@@ -79,3 +79,5 @@ pyvenv.cfg
 lib64
 tcl
 /.env
+
+*.orig

CHANGELOG.rst

@@ -5,6 +5,12 @@ v20.x (next)
 ------------
 
  - Adopt calendar versioning for scancode version numbers
+ - Updated scancode.bat to handle % signs in the arguments #1876
+ - Support officially Python 64 bits on Windows 64 bits #335
+ - Drop support for Python 32 bits on Windows #335
+ - Update configuration and boostrap scripts
+ - Drop support for Python 2  #295
+ 
 
 
 v3.2.3 (2020-10-27)

INSTALL.rst

@@ -2,9 +2,10 @@
 Installation
 ============
 
-There are 3 main ways you can `install ScanCode <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html>`_.
+There are 4 main ways you can `install ScanCode <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html>`_.
 
 - Installation as an Application: Downloading Releases (Recommended)
+- Docker Installation
 - Installation as a library: via pip
 - Installation from Source Code: Git Clone
 
@@ -32,14 +33,30 @@ Installation as an Application : Downloading Releases
     - Linux/Mac : ``./scancode --help``
     - Windows : ``scancode --help``
 
+Docker Installation
+-------------------
+
+#. Download the Source Code as an archive from the `GitHub releases <https://github.com/nexB/scancode-toolkit/releases>`_ and unzip it, or via `git clone`.
+
+#. Build the docker image from the `scancode-toolkit` directory.::
+
+	docker build -t scancode-toolkit .
+
+#. Mount current working directory and run scan on mounted folder::
+
+    docker run -v $PWD/:/project scancode-toolkit -clpeui --json-pp /project/result.json /project
+
+Note that the parameters *before* ``scancode-toolkit`` are used for docker,
+those after will be forwarded to scancode.
+
 Installation as a library: via pip
 ----------------------------------
 
 #. Create a Python 3.6 Virtual Environment and activate the same::
 
     virtualenv -p /usr/bin/python3.6 venv-scancode && source venv-scancode/bin/activate
 
-#. Run ``pip install scancode-toolkit`` 
+#. Run ``pip install scancode-toolkit[full]``
 
 Installation from Source Code: Git Clone
 ----------------------------------------

docs/source/contribute/cut_new_release.rst

@@ -42,4 +42,4 @@ How to cut a new release:
     - Run twine with ``twine upload dist/<path to the built wheel>``
     - Once uploaded check the published release at https://pypi.python.org/pypi/scancode-toolkit/
     - Then create a new fresh local virtualenv and test the wheel installation with:
-      ``pip install scancode-toolkit``
+      ``pip install scancode-toolkit[full]``

docs/source/getting-started/install.rst

@@ -246,6 +246,7 @@ Installation via Docker:
 ------------------------
 
 You can install Scancode Toolkit by building a Docker image from the included Dockerfile.
+The prerequisite is a working `docker installation <https://docs.docker.com/engine/install/>`_.
 
 Download the ScanCode-Toolkit Source Code
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -258,7 +259,7 @@ Build the Docker image
 ^^^^^^^^^^^^^^^^^^^^^^
 
 The ``docker build`` command needs to run in the directory of the source code,
-make sure to ``cd`` into the correct directory.
+make sure to ``cd`` into the correct directory.::
 
     cd scancode-toolkit
     docker build -t scancode-toolkit .
@@ -268,17 +269,17 @@ Run using Docker
 
 The docker image will forward all arguments it receives directly to the ``scancode`` command.
 
-Display help:
+Display help::
 
     docker run scancode-toolkit --help
 
-Mount current working directory and run scan on mounted folder:
+Mount current working directory and run scan on mounted folder::
 
     docker run -v $PWD/:/project scancode-toolkit -clpeui --json-pp /project/result.json /project
 
 This will mount your current working from the host into ``/project`` in the container
 and then scan the contents. The output ``result.json`` will be written back to your
-corrent working directory on the host.
+current working directory on the host.
 
 Note that the parameters *before* ``scancode-toolkit`` are used for docker,
 those after will be forwarded to scancode.
@@ -361,7 +362,7 @@ For more information on Python virtualenv, visit this `page <https://docs.python
 
     source venv-scancode/bin/activate
 
-#. Run ``pip install scancode-toolkit`` to install the latest version of Scancode.
+#. Run ``pip install scancode-toolkit[full]`` to install the latest version of Scancode.
 
 .. NOTE::
 

docs/source/getting-started/newcomer.rst

@@ -82,12 +82,13 @@ Before you start using Scancode
    is created.
 
 :ref:`app_install`
+:ref:`docker_install`
 :ref:`pip_install`
 :ref:`source_code_install`
 
 
 #. Now you can either follow the instructions for the recommended :ref:`app_install` method ,
-   or run ``pip install scancode-toolkit`` like that in the :ref:`pip_install` documentation.
+   or run ``pip install scancode-toolkit[full]`` like that in the :ref:`pip_install` documentation.
    Alternatively, you can also :ref:`source_code_install`.
 
 #. Run ``scancode -h`` to make sure Scancode was installed properly.

docs/source/tutorials/how_to_run_a_scan.rst

@@ -36,7 +36,7 @@ Setting up Scancode Toolkit
 
 Get ScanCode Toolkit from ``pip``::
 
-    pip install scancode-toolkit
+    pip install scancode-toolkit[full]
 
 .. Note::
 

setup.py

@@ -15,14 +15,13 @@
 from setuptools import find_packages
 from setuptools import setup
 
-version = '20.10.27'
+version = '20.12.03'
 
 
 def read(*names, **kwargs):
     import os
     return open(
         os.path.join(os.path.dirname(__file__), *names),
-        # encoding=kwargs.get('encoding', 'utf8')
     ).read()
 
 
@@ -123,14 +122,17 @@ def read(*names, **kwargs):
         'colorama >= 0.3.9',
         'pluggy >= 0.4.0, < 1.0',
         'attrs >= 18.1, !=20.1.0',
-        'typing >=3.6, < 3.7',
+        # Importing typing causes errors after python 3.6.
+        # See https://github.com/python/typing/issues/573
+        'typing >=3.6, < 3.7; python_version < "3.7"',
 
         # scancode outputs
         'jinja2 >= 2.7.0, < 3.0.0',
         'MarkupSafe >= 0.23',
         'simplejson',
         'spdx_tools >= 0.6.0',
         'unicodecsv',
+        'jsonstreams >= 0.5.0',
 
         # ScanCode caching and locking
         'yg.lockfile >= 2.3, < 3.0.0',

src/formattedcode/output_json.py

@@ -27,7 +27,7 @@
 
 from collections import OrderedDict
 
-import simplejson
+import jsonstreams
 from six import string_types
 
 from commoncode.system import py2
@@ -95,8 +95,7 @@ def is_enabled(self, output_json, **kwargs):
         return output_json
 
     def process_codebase(self, codebase, output_json, **kwargs):
-        results = get_results(codebase, as_list=False, **kwargs)
-        write_json(results, output_file=output_json, pretty=False)
+        write_results(codebase, output_file=output_json, pretty=False, **kwargs)
 
 
 @output_impl
@@ -115,33 +114,48 @@ def is_enabled(self, output_json_pp, **kwargs):
         return output_json_pp
 
     def process_codebase(self, codebase, output_json_pp, **kwargs):
-        results = get_results(codebase, as_list=False, **kwargs)
-        write_json(results, output_file=output_json_pp, pretty=True)
+        write_results(codebase, output_file=output_json_pp, pretty=True, **kwargs)
 
 
-def write_json(results, output_file, pretty=False, **kwargs):
+def write_results(codebase, output_file, pretty=False, **kwargs):
     """
-    Write `results` to the `output_file` opened file-like object.
+    Write headers, files, and other attributes from `codebase` to `output_file`
+
+    Enable JSON indentation if `pretty` is True
     """
-    # NOTE: we write as encoded, binary bytes, not as unicode, decoded text on py2
-    kwargs = dict(iterable_as_array=True, encoding='utf-8')
+    # Set indentation for JSON output if `pretty` is True
+    # We use a separate dict for jsonstream kwargs since we are passing
+    # this function's kwargs as arguments to OutputPlugin.get_files()
     if pretty:
-        kwargs.update(dict(indent=2 * space))
+        jsonstreams_kwargs = dict(indent=2, pretty=True)
     else:
-        kwargs.update(dict(separators=(comma, colon,)))
-
-    close_of = False
-
-
-    try:
-        if isinstance(output_file, string_types):
-            output_file = open(output_file, mode)
-            close_of = True
-        output_file.write(simplejson.dumps(results, **kwargs))
-        output_file.write(eol)
-    finally:
-        if close_of:
-            output_file.close()
+        jsonstreams_kwargs = dict(indent=None, pretty=False)
+
+    # If `output_file` is a path string, open the file at path `output_file` and use it as `output_file`
+    close_fd = False
+    if isinstance(output_file, string_types):
+        output_file = open(output_file, mode)
+        close_fd = True
+
+    # Begin writing JSON to `output_file`
+    with jsonstreams.Stream(jsonstreams.Type.object, fd=output_file, close_fd=close_fd, **jsonstreams_kwargs) as s:
+        # Write headers
+        codebase.add_files_count_to_current_header()
+        codebase_headers = codebase.get_headers()
+        s.write('headers', codebase_headers)
+
+        # Write attributes
+        if codebase.attributes:
+            for attribute_key, attribute_value in codebase.attributes.to_dict().items():
+                s.write(attribute_key, attribute_value)
+
+        # Write files
+        codebase_files = OutputPlugin.get_files(codebase, **kwargs)
+        if py3:
+            # OutputPlugin.get_files() returns a `map()`, which isn's JSON
+            # serializable in Python 3
+            codebase_files = list(codebase_files)
+        s.write('files', codebase_files)
 
 
 def get_results(codebase, as_list=False, **kwargs):
@@ -169,4 +183,3 @@ def get_results(codebase, as_list=False, **kwargs):
         logger_debug(pformat(files))
 
     return results
-