Update Docs with Changes in Wiki

docs/source/deltacode/deltacode_scoring.rst

@@ -30,6 +30,10 @@ In descending order of importance, the categories are:
 #. ``removed``: A file has been removed from the ``old`` codebase.
 #. ``unmodified``: The file is contained in both the ``new`` and ``old`` codebase and has not been modified or moved.
 
+.. note:: 
+
+    Files are determined to be Moved by looping thru the `added` and `removed` Delta objects and checking their sha1 values.
+
 The score of a Delta object characterized as ``added`` or ``modified`` may be increased based on the detection of license- and/or copyright-related changes. See :ref:`license_additions_and_changes` and :ref:`copyright_holder_additions_and_changes` below.
 
 Delta Object Fields and Values

docs/source/deltacode/gsoc19_final_report.rst

@@ -0,0 +1,69 @@
+Google Summer of Code 2019 - Final report
+=========================================
+
+**Project: Approximately similar file detection in DeltaCode**
+--------------------------------------------------------------
+
+**Arnav Mandal <arnav.mandal1234@gmail.com>**
+
+Project Overview
+----------------
+
+DeltaCode is a tool to compare and report scan differences. It takes JSON files as an input which
+is the output of ScanCode-toolkit as well. When comparing files, it only uses the exact comparison.
+By exact comparison, I mean it compares the hash value of the files. The output of DeltaCode is a
+JSON/CSV file which includes the details of the scan such as delta score, delta count, etc. The
+goal of this project is to improve the usefulness of the delta by also finding files that are
+mostly the same (e.g. quasi or near duplicates) vs. files that are completely different. After this
+project, DeltaCode would be able to detect similar files in a directory approximately.
+
+Requirements of the project
+---------------------------
+
+- Provided two files using ScanCode-toolkit, the new near-duplicate detection should return the
+  distance between the two files.
+- The code should be seamlessly integrated with ScanCode-toolkit. It should be highly configurable
+  by the maintainers.
+- The strictness of near-duplicates should be noted and adjusted by a threshold variable.
+
+The Project
+-----------
+
+- Addition of new fingerprint plugin in the ScanCode Toolkit.
+- Implementation and integration of the fingerprint generation algorithm in the ScanCode Toolkit
+  codebase.
+- Implementation of distance finding algorithm between the files and process them further in the
+  DeltaCode codebase.
+- Integration of fingerprint field in the JSON file to compare the deltas and provide them with
+  appropriate scores.
+- Make changes to old unit tests and addition of new unit tests in ScanCode Toolkit as well as
+  DeltaCode.
+
+I have completed all the tasks that were in the scope of this GSoC project.
+
+Pull Requests
+-------------
+
+- https://github.com/nexB/scancode-toolkit/pull/1576 [Closed] (something went wrong while rebasing)
+- https://github.com/nexB/scancode-toolkit/pull/1651 [Merged]
+- https://github.com/nexB/deltacode/pull/128 [Merged]
+
+Links
+-----
+
+..
+    [Org Link] https://summerofcode.withgoogle.com/organizations/6118953540124672/
+    [Project Link] https://summerofcode.withgoogle.com/projects/#6422961651712000
+
+- `Project Details <https://summerofcode.withgoogle.com/organizations/6118953540124672/>`_
+- `Proposal <https://docs.google.com/document/d/1XGUH4Ogovc8ZTt_bjZc7ML75E1NrhTlAtelckESpSGA/edit?usp=sharing>`_
+- `ScanCode Toolkit <https://github.com/nexB/scancode-toolkit>`_
+- `DeltaCode <https://github.com/nexB/deltacode>`_
+
+------------
+
+I’ve had a wonderful time during these three months and have learned plenty of things. I would
+really like to thank `@pombredanne <https://github.com/pombredanne>`_,
+`@majurg <https://github.com/majurg>`_, and `@JonoYang <https://github.com/JonoYang>`_ for their
+constant support throughout the journey. From good job claps to nit-picky constructive
+code-reviews, I enjoyed every bit of this GSoC project.

docs/source/deltacode/index.rst

@@ -11,4 +11,5 @@ Welcome to Deltacode Documentation.
    deltacode_scoring
    development
    json_to_csv
-   release_process
+   release_process
+   gsoc19_final_report

docs/source/scancode-toolkit/cli-reference/output-format.rst

@@ -12,6 +12,12 @@ following options.
 
 ----
 
+.. _output_to_stdout:
+
+.. include::  /scancode-toolkit/rst_snippets/stdout.rst
+
+----
+
 ``--json FILE``
 ---------------
 

docs/source/scancode-toolkit/cli-reference/synopsis.rst

@@ -70,6 +70,8 @@ is JSON, the entire file being in one line, without whitespace characters.
 The following example scans will show you how to run a scan with each of the result formats. For
 the scans, we will use the ``samples`` directory provided with the ScanCode Toolkit.
 
+.. include::  /scancode-toolkit/rst_snippets/tip_snippets/stdout.rst
+
 JSON file output
 ^^^^^^^^^^^^^^^^
 

docs/source/scancode-toolkit/contribute/contrib_doc.rst

@@ -26,7 +26,7 @@ In the project root, run the following command::
 
 .. note::
 
-   Only the scancode-toolkit documentation style standards are enforced presently. 
+   Only the scancode-toolkit documentation style standards are enforced presently.
 
 A sample output is::
 
@@ -50,8 +50,8 @@ A sample output is::
 
 Now fix the errors and run again till there isn't any style error in the documentation.
 
-What is Checked
-^^^^^^^^^^^^^^^
+What is Checked?
+^^^^^^^^^^^^^^^^
 
 PyCQA is an Organization for code quality tools (and plugins) for the Python programming language.
 Doc8 is a sub-project of the same Organization. Refer this `README <https://github.com/PyCQA/doc8/blob/master/README.rst>`_ for more details.
@@ -143,3 +143,18 @@ Extra Style Checks
     temporary ``.bak`` files generated. Refer the `manual <http://aspell.net/man-html/>`_ for more
     information on how to use.
 
+7. Notes and Warning Snippets
+
+    Every ``Note`` and ``Warning`` sections are to be kept in ``rst_snippets/note_snippets/`` and
+    ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are
+    frequently used in multiple files.
+
+Converting from Markdown
+------------------------
+
+If you want to convert a ``.md`` file to a ``.rst`` file, this `tool <https://github.com/chrissimpkins/md2rst>`_
+does it pretty well. You'd still have to clean up and check for errors as this contains a lot of
+bugs. But this is definitely better than converting everything by yourself.
+
+This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for
+Sphinx/ReadTheDocs hosting.

docs/source/scancode-toolkit/contribute/gsoc19_final_report.rst

@@ -0,0 +1,102 @@
+Google Summer of Code 2017 - Final report
+=========================================
+
+Project: **scancode-toolkit** to Python 3
+-----------------------------------------
+
+**Owner:** `Abhishek Kumar <https://github.com/Abhishek-Dev09>`_
+
+**Mentor:** `Philippe Ombredanne <https://github.com/pombredanne>`_
+
+Overview
+--------
+
+**Problem:** Since Python 2.7 will retire in few months and will not be maintained any longer.
+
+**Solution:** `Scancode <https://github.com/nexB/scancode-toolkit/>`__ needs to be ported to
+python 3 and all test suites must pass on both version of Python. The main difference that
+makes Python 3 better than Python 2.x is that the support for unicode is greatly improved in
+Python 3. This will also be useful for scancode as scancode has users in more than 100 languages
+and it's easy to translate strings from unicode to other languages.
+
+**Objective**: To make scancode-toolkit installable on on Python 3.6 and higher, as presently it
+installs with Python 2.7 only.
+
+Implementation
+--------------
+
+- It was started in development mode(editable mode) and then it was moved to work in virtual
+  environments.
+- I have worked module by module according to the order of hierarchy of modules. For example :All
+  module is dependent on commoncode, so it must be ported first. In this way we have created the
+  Porting order:
+
+   1.  commoncode
+   2.  plugincode
+   3.  typecode
+   4.  extractcode
+   5.  textcode
+   6.  scancode basics (some tests are integration tests and will have to wait to be ported)
+   7.  formattedcode, starting with JSON (some tests are integration tests and will have to wait
+       to be ported)
+   8.  cluecode
+   9.  licensedcode
+   10. packagedcode (depends on licensecode)
+   11. summarycode
+   12. fixup the remaining bits and tests
+
+After porting each module, I have marked these modules as ported ``scanpy3`` with help of
+**conffest** plugin (created by `@pombredanne <https://github.com/pombredanne>`_). **Conffest**
+plugin is heart of this project. Without this, it was very difficult to do. Dependencies was fixed
+at the time of porting the module where it was used.
+
+Challenging part of Project
+---------------------------
+
+It is very difficult to deal with paths on different operating systems.The issue is around
+macOS/Windows/Linux. The first two OS handle unicode paths comfortably on Python 2 and 3 but not
+completely on macOS Mojave because its filesystem encoding is APFS. Linux paths are bytes and
+os.listdir is broken on Python 2. As a result you can only sanely handle Linux paths as bytes
+on Python 2. But on Python 3 path seems to be corrected as ``unicode`` on Linux.
+
+For more details visit here :
+
+- https://vstinner.github.io/painful-history-python-filesystem-encoding.html
+- `jaraco/path.py#130 <https://github.com/jaraco/path.py/issues/130>`__
+
+We came with various Solution:
+
+- To use pathlib which generally handle paths correctly across platforms. And for backports we use
+  pathlib 2. But this solution also fails because pathlib 2 does not work as expected wrt unicode
+  vs bytes. And os.listdir also doesn't work properly.
+
+- To use `path.py <https://pypi.org/project/path.py/>`__ which handles the paths across all the
+  platforms even on macOS Mojave .
+
+- Use ``bytes`` on linux and python 3 and ``unicode`` everywhere.
+
+We choose the third solution because it is most fundamental and simple and easy to use.
+
+Project was tracked in this ticket `nexB/scancode-toolkit#295 <https://github.com/nexB/scancode-toolkit/issues/295>`__
+
+**Project link :** `Port Scancode to Python 3 <https://summerofcode.withgoogle.com/organizations/6118953540124672/>`__
+
+..
+    [Org Link] https://summerofcode.withgoogle.com/organizations/6118953540124672/
+    [Project Link] https://summerofcode.withgoogle.com/projects/#5969926387400704
+
+**My contribution :** `List of Commits <https://github.com/nexB/scancode-toolkit/commits?author=Abhishek-Dev09>`__
+
+**Note :** Please give your feedback `here <https://github.com/nexB/scancode-toolkit/issues/295>`_
+
+Outcome
+-------
+
+Now we have liftoff on Python 3 . We are able to run basic scans without errors on develop branch.
+You check it by running ``scancode -clipeu samples/ --json-pp - -n4`` .
+
+At last I would like to thanks my Mentor **@pombredanne** aka
+`Philippe Ombredanne <https://github.com/pombredanne>`__ . He has helped lot in completing this
+project. He is very supportive and responsive. I have learned a lot from him. By his encouragement
+and motivation, I am very improving day by day, building and developing my skills. I have completed
+all the tasks that were in the scope of this GSoC project.

docs/source/scancode-toolkit/contribute/index.rst

@@ -8,4 +8,5 @@
    cut_new_release
    contrib_doc
    roadmap
-   gsoc_17_final_report
+   gsoc17_final_report
+   gsoc19_final_report

docs/source/scancode-toolkit/contribute/roadmap.rst

@@ -23,9 +23,9 @@ Packages manifests and dependencies parsers
 - |clock1030| Windows PE #652
 - |white_large_square| RPMs dependencies #649
 - |white_large_square| Windows Nuget dependencies #648
-- |white_large_square| Bower packages #654
-- |white_large_square| Python dependencies #653
-- |white_large_square| CRAN
+- |white_check_mark| Bower packages #654
+- |clock1030| Python dependencies #653
+- |white_check_mark| CRAN
 - |white_check_mark| Plain packages
 - |white_large_square| other Java-related meta files (SBT, Ivy, Gradle, etc.)
 - |white_large_square| Debian debs
@@ -57,22 +57,22 @@ Core features
 - |white_check_mark| pre scan filtering (ignore binaries, etc)
 - |white_check_mark| pre/post/ouput plugins! (worked as part of the GSoC by @yadsharaf )
 - |white_check_mark| scan plugins (e.g. plugins that run a scan to collect data)
-- |clock1030| support Python 3 #295
+- |white_check_mark| support Python 3 #295
 - |clock1030| transparent archive extraction (as opposed to on-demand with extractcode)
 - |clock1030| scancode.yml configuration file for exclusions, defaults, scan failure conditions,
   etc.
 - |white_large_square| support scan pipelines and rules to organize more complex scans
-- |white_large_square| scan baselining, delta scan and failure conditions (such as license change,
-  etc) (will be spawned as its own DeltaCode project)
+- |white_check_mark| scan baselining, delta scan and failure conditions (such as license change,
+  etc) ( spawned as its the `DeltaCode <https://github.com/nexB/deltacode/>`_ project)
 - |white_large_square| dedupe and similarities to avoid re-scanning. For now only identical files
   are scanned only once.
-- |white_large_square| Improved logging, tracing and error diagnostics
-- |clock1030| native support for ABC Data (See :ref:`aboutcode_data` )
+- |clock1030| Improved logging, tracing and error diagnostics
+- |white_check_mark| native support for ABC Data (See :ref:`aboutcode_data` )
 
 Classification, summarization and deduction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- |white_check_mark| File classification #426
+- |clock1030| File classification #426
 - |white_check_mark| summarize and aggregate data #377 at the top level
 
 Source code support (some will be spawned as their own tool)
@@ -100,7 +100,7 @@ Packaging
 
 - |white_large_square| simpler installation, automated installer
 - |white_check_mark| distro-friendly packaging
-- |white_large_square| unbundle and package as multiple libaries (commoncode, extractcode, etc)
+- |clock1030| unbundle and package as multiple libaries (commoncode, extractcode, etc)
 
 Documentation
 ^^^^^^^^^^^^^
@@ -120,12 +120,13 @@ CI integration
 Other work in progress
 ----------------------
 
-- ScanCode server: Spawned as its own project: https://github.com/nexB/scancode-server. Will
-  include Integration / webhooks for Github, Bitbucket.
-- VulnerableCode: NVD and CVE lookups: Spawned as its own project: https://github.com/nexB/vulnerablecode
-- ScanCode Workbench: desktop app for scan review: Spawned as its own project:
+- |clock1030| ScanCode server: Spawned as its own project: https://github.com/nexB/scancode-server.
+  Will include Integration / webhooks for Github, Bitbucket.
+- |clock1030| VulnerableCode: NVD and CVE lookups: Spawned as its own project:
+  https://github.com/nexB/vulnerablecode
+- |white_check_mark| ScanCode Workbench: desktop app for scan review: Spawned as its own project:
   https://github.com/nexB/scancode-workbench
-- DependentCode: dynamic dependencies resolutions: Spawned as its own project:
+- |white_large_square| DependentCode: dynamic dependencies resolutions: Spawned as its own project:
   https://github.com/nexB/dependentcode
 
 Package mining and matching

docs/source/scancode-toolkit/index.rst

@@ -6,6 +6,7 @@
 
    getting-started/index
    cli-reference/index
+   plugins/index
    tutorials/index
    how-to-guides/index
    explanations/index

docs/source/scancode-toolkit/misc/index.rst

@@ -7,5 +7,3 @@
    faq
    support
    perf_report
-   plugin_arch
-   licence_policy_plugin

docs/source/scancode-toolkit/plugins/index.rst

@@ -0,0 +1,9 @@
+**Plugins**
+=================
+
+.. toctree::
+   :maxdepth: 2
+
+   plugin_arch
+   licence_policy_plugin
+   plugin_tutorials

docs/source/scancode-toolkit/misc/licence_policy_plugin.rst → docs/source/scancode-toolkit/plugins/licence_policy_plugin.rst

@@ -10,7 +10,7 @@ attribute: ``license_policy``.
 
 Policy File Specification
 -------------------------
-The Policy file is a YAML (``.yml``) document with the following struture::
+The Policy file is a YAML (``.yml``) document with the following structure::
 
   license_policies:
   -   license_key: mit
@@ -26,7 +26,7 @@ The Policy file is a YAML (``.yml``) document with the following struture::
       color_code: '#FFcc33'
       icon: icon-warning-sign
 
-The only required key is ``license_key``, which represents the scancode license key to match
+The only required key is ``license_key``, which represents the ScanCode license key to match
 against the detected licenses in the scan results.
 
 In the above example, a descriptive label is added along with a color code and CSS ``id`` name

docs/source/scancode-toolkit/misc/plugin_arch.rst → docs/source/scancode-toolkit/plugins/plugin_arch.rst

@@ -3,6 +3,9 @@
 Plugin Architecture
 ===================
 
+Notes: this is the initial design for ScanCode plugins. The actual architecture evolved and is
+different.
+
 Abstract:
 ---------
 

docs/source/scancode-toolkit/plugins/plugin_tutorials.rst

@@ -0,0 +1,4 @@
+Plugin Tutorials
+================
+
+- :ref:`how_to_add_post_scan_plugin`

docs/source/scancode-toolkit/rst_snippets/stdout.rst

@@ -0,0 +1,9 @@
+Print to ``stdout`` (Terminal)
+------------------------------
+
+If you want to format the output in JSON and print it at stdout, you can replace the JSON filename
+with a "-", like ``--json-pp -`` instead of ``--json-pp output.json``.
+
+The following command will output the scan results in JSON format to ``stdout`` (In the Terminal)::
+
+    ./scancode -clpieu --json-pp - samples/