From 0876acaf465ba72095ac3516eda097ea2082814e Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Tue, 14 Mar 2023 22:43:23 +0530 Subject: [PATCH 01/13] Fix unordered lists bug Looks like there was an issue in docutils/sphinx-rtd-theme so we needed to downgrade to docutils < 0.17 for the unordered lists to be rendered again properly. Reference: https://github.com/readthedocs/sphinx_rtd_theme/issues/1115 Signed-off-by: Ayan Sinha Mahapatra --- setup.cfg | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/setup.cfg b/setup.cfg index afa42855642..ff738af4568 100644 --- a/setup.cfg +++ b/setup.cfg @@ -136,9 +136,10 @@ testing = pytest-rerunfailures docs = - Sphinx >= 3.3.1 - sphinx-rtd-theme >= 0.5.0 - doc8 >= 0.8.1 + Sphinx >= 4.3.0 + sphinx_rtd_theme >= 0.5.1 + docutils < 0.17 + doc8 # linux-only package handling packages = From 1fd2a7fb045e85bea4010ed3f5cf1c99dbea987a Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Wed, 15 Mar 2023 13:55:31 +0530 Subject: [PATCH 02/13] Add docs about M1 support #3277 Signed-off-by: Ayan Sinha Mahapatra --- docs/source/getting-started/install.rst | 28 ++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/source/getting-started/install.rst b/docs/source/getting-started/install.rst index 4e5c14da80d..71aaef2056f 100644 --- a/docs/source/getting-started/install.rst +++ b/docs/source/getting-started/install.rst @@ -2,7 +2,7 @@ Comprehensive Installation ========================== -There in one way to install ScanCode. +The recommended way to install ScanCode is using app archives: - :ref:`app_install` @@ -40,6 +40,7 @@ Before Installing - ScanCode requires a Python version 3.7, 3.8, 3.9 or 3.10 and is tested on Linux, macOS, and Windows. It should work fine on FreeBSD. +.. _system_requirements: System Requirements ^^^^^^^^^^^^^^^^^^^ @@ -53,6 +54,7 @@ System Requirements #. Linux: on recent 64-bit Linux distributions, #. Mac: on recent x86 64-bit macOS (10.15 and up, including 11 and 12), Use the X86 emulation mode on Apple ARM M1 CPUs. + (Note that `pip install` does not work on ARM CPUs) #. Windows: on Windows 10 and up, #. FreeBSD. @@ -116,6 +118,13 @@ ScanCode needs a Python 3.7+ interpreter; We support all Python versions from Installation as an Application: Downloading Releases ----------------------------------------------------- +Get the Scancode Toolkit tarball archive of a specific version and your +operating system by going to the `project releases page `_ + +For example, Version 30.0.1 archive can be obtained from +`Toolkit release 30.0.1 `_ +under assets options. + .. Note:: ScanCode app archives come with packaged with all required dependencies except @@ -134,12 +143,7 @@ Installation as an Application: Downloading Releases Installation on Linux and Mac ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Get the Scancode Toolkit tarball archive of a specific version and your -operating system by going to the `project releases page `_ - -For example, Version 30.0.1 archive can be obtained from -`Toolkit release 30.0.1 `_ -under assets options. Download the archive for your operating systen and extract +Download the archive for your operating systen and extract the archive from command line:: tar -xvf scancode-toolkit-30.0.1_py38-linux.tar.gz @@ -337,6 +341,16 @@ Installation as a library: via ``pip`` ScanCode can be installed from the public PyPI repository using ``pip`` which the standard Python package management tool. +.. NOTE:: + + Note that `pip` installation method does work on ARM chips, i.e. Linux/MacOS on + Apple M1 chips, as some non-native dependencies does not have pre-built wheels + for ARM (like py-ahocorasick, intbitset). See :ref:`system_requirements` for + more information. See related issues for more info: + + - `Fallback pure-python deps `_ + - `pip install failing on M1 `_ + The steps are: #. Create a Python virtual environment:: From 637ad9bc38ee00ef9671f92421c079ce0b5399b7 Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Wed, 15 Mar 2023 21:23:50 +0530 Subject: [PATCH 03/13] Update How-to-Guides with license updates Signed-off-by: Ayan Sinha Mahapatra --- docs/source/how-to-guides/add_new_license.rst | 81 ++++++++++++++----- .../add_new_license_detection_rule.rst | 76 ++++++++++------- .../install_new_license_plugin.rst | 13 +-- 3 files changed, 114 insertions(+), 56 deletions(-) diff --git a/docs/source/how-to-guides/add_new_license.rst b/docs/source/how-to-guides/add_new_license.rst index afa30bd7c31..5c441cdfa4c 100644 --- a/docs/source/how-to-guides/add_new_license.rst +++ b/docs/source/how-to-guides/add_new_license.rst @@ -6,30 +6,75 @@ How To Add a New License for Detection How to add a new license for detection? --------------------------------------- -To add new license, you first need to select a new and unique license key (mit -and gpl-2.0 are some of the existing license keys). All licenses are stored as -plain text files in the src/licensedcode/data/licenses directory using their key -as base for the file name(s). +To add new license, you first need to select a new and unique license `key` (mit +and gpl-2.0 are some of the existing license keys). -You need to create a pair of files: +The key name can contain only these symbols: -- a file with the text of the license saved in a plain text file named - key.LICENSE +- lowercase letters from a to z, +- numbers from 0 to 9,and +- dash - and . period signs. No spaces or underscore. -- a small text data file (in YAML format) named key.yml that contains license - information such as:: +The license key also has to be less than 50 characters (same for `short_name`). - key: my-license - name: My License +We also have to add a `spdx_license_key` which is either a valid SPDX license key at +` The SPDX license list `_, or a `Licenseref-scancode-`. -The key name can contain only these symbols: +All licenses are stored as a plain text file in the `src/licensedcode/data/licenses` +directory using their key as base for the file name. I.e. the filename for a license +with `key: mit` would be `mit.LICENSE`. -- lowercase letters from a to z, -- numbers from 0 to 9,and -- dash - and . period signs. No spaces. +You need to create a file with: + +- the text of the license saved in plain text. We usually get rid of HTML tags or + other special characters, copyrights and only keep the original text as is, + with the original formatting intact. + +- the data attributes for the license in YAML format as YAML frontmatter. + +See an example license: `apache-2.0.LICENSE `_ + +There are a couple of mandatory attributes: + +- `key` +- `spdx_license_key` +- `short_name` +- `name` +- `category` (Use "Unstated License" if not known) +- `owner` (Use "Unspecified" if not known) + +And more attributes which are not mandatory but nice to have always (if applicable): + +- `other_spdx_license_keys` +- `osi_license_key` +- `minimum_coverage` +- `standard_notice` +- `notes` + +We want to use `minimum_coverage` when there are other licenses which are very similar +and we want to make sure we match correctly these licenses, and `notes` for interesting +cases of licenses with descriptions to help identify origin, similarities to other licenses, +notes about the SPDX keys and others. + +Some URLs: + +- `homepage_url` +- `text_urls` +- `osi_url` +- `faq_url` +- `other_urls` + +Also attributes having ignorables in the license text: + +- `ignorable_urls` +- `ignorable_copyrights` +- `ignorable_authors` +- `ignorable_holders` +- `ignorable_emails` -Save these two files in the ``src/licensedcode/data/licenses/`` directory. +See the ``src/licensedcode/data/licenses/`` directory for many more examples. -Done! +.. note:: -See the ``src/licensedcode/data/licenses/`` directory for many examples. + Add licenses in a local developement installation and run `scancode-reindex-licenses` + to make sure we reindex the licenses and this validates the new licenses. diff --git a/docs/source/how-to-guides/add_new_license_detection_rule.rst b/docs/source/how-to-guides/add_new_license_detection_rule.rst index ddc06ca9a0b..b24970ecc97 100644 --- a/docs/source/how-to-guides/add_new_license_detection_rule.rst +++ b/docs/source/how-to-guides/add_new_license_detection_rule.rst @@ -4,65 +4,83 @@ How to Add New License Rules for Enhanced Detection =================================================== ScanCode relies on license rules to detect licenses. A rule is a simple text -file containing a license text or notice or mention; And a small companion YAML -text file that tells ScanCode which license expression to report when the text -is detected. +file containing a license text or notice or mention with YAML frontmatter with data +attributes that tells ScanCode which license expression to report when the text +is detected, and other properties. -See the :ref:`faq` for a high level description of :ref:`add_new_license_det_rule`. +See the :ref:`faq` for a high level description of adding license detection rules. How to add a new license detection rule? ---------------------------------------- -A license detection rule is a pair of files: +A license detection rule is a file with: -- a plain text rule file that is typically a variant of a license text, notice - or license mention. +- a plain text that is typically a variant of a license text, notice or license + mention. -- a small text data file (in YAML format) documenting which license expression - should be detected when the rule text is found in a codebase. +- data as YAML frontmatter documenting license expression and other + rule attributes. To add a new rule, you need to pick a unique base file name. As a convention, we like to include the license expression that should be detected in that name to make it more descriptive. For example: mit_and_gpl-2.0 is a good base name for a rule that would detect an MIT and GPL-2.0 license combination at once. Add a -suffix to make it unique if there is already a rule with this base name. Do not -use spaces or special characters in that name. +suffix (usually numeric) to make it unique if there is already a rule with +this base name. Do not use spaces or special characters in that name. -Then create the rule file in the src/licensedcode/data/rules/ directory using -this name, replacing selected_base_name with the base name you selected:: +Then create the rule file in the `src/licensedcode/data/rules/` directory using +this name, for example a rule with `license_expression` as `mit AND apache-2.0` +might have a filename: `mit_and_apache-2.0_10.RULE`. - selected_base_name.RULE +Save your rule text in this file, if there are specific words like company names, +projects or other, it is better to have rules with and without these so we have +better detection. -Save your rule text in this file. +For a simple `mit AND apache-2.0` license expression detection, here is an example +rule file:: -Then create the YAML data file in the src/licensedcode/data/rules/ directory -using this name:: - selected_base_name.yml - -For a simple mit and gpl-2.0 license expression detection, the content of -this file can be this YAML snippet:: - - license_expression: mit AND gpl-2.0 + --- + license_expression: mit AND apache-2.0 is_license_notice: yes + relevance: 100 + referenced_filenames: + - LICENSE + --- -Save these two files in the ``src/licensedcode/data/licenses/`` directory and -you are done! + ## License + The MIT License (MIT) + Apache 2.0. Read [LICENSE](LICENSE). See the ``src/licensedcode/data/rules/`` directory for many examples. More (advanced) rules options: -- you can use a notes: text field to document this rule and explain where you +- you can use a `notes` text field to document this rule and explain where you found it first. - if no license should be detected for your .RULE text, do not add a license expression, just add a ``notes`` field. -- Each rules needs have one flag such as is_license_notice. See the - ``src/licensedcode/models.py`` directory for a list of all possible values and - other options. +- Each rules needs have one flag to describe the type of license rule, the options are: + + - `is_license_notice` + - `is_license_text` + - `is_license_tag` + - `is_license_reference` + - `is_license_intro` + +- There can also be false positive rules, which if detected in the file scanned, will not + be present in the result license detections. These just have the license text and a + `is_false_positive` flag set to True. - you can specify key phrases by surrounding one or more words between the `{{` and `}}` tags. Key phrases are words that **must** be matched/present in order for a RULE to be considered a match. + +See the ``src/licensedcode/models.py`` directory for a list of all possible values +and other options. + +.. note:: + + Add rules in a local developement installation and run `scancode-reindex-licenses` + to make sure we reindex the rules and this validates the new licenses. diff --git a/docs/source/how-to-guides/install_new_license_plugin.rst b/docs/source/how-to-guides/install_new_license_plugin.rst index 28092e3e883..d527c95d068 100644 --- a/docs/source/how-to-guides/install_new_license_plugin.rst +++ b/docs/source/how-to-guides/install_new_license_plugin.rst @@ -28,10 +28,8 @@ This is the basic structure of the example plugin:: │ └── licenses_to_install1/ │ ├── licenses/ │ │ ├── example-installed-1.LICENSE - │ │ └── example-installed-1.yaml | ├── rules/ │ │ ├── example-installed-1.RULE - │ │ └── example-installed-1.yaml │ └── __init__.py ├── apache-2.0.LICENSE ├── MANIFEST.in @@ -104,17 +102,15 @@ an example of a plugin with tests. The tests are contained in the ``tests`` dire │ └── licenses_to_install1/ │ ├── licenses/ │ │ ├── example-installed-1.LICENSE - │ │ └── example-installed-1.yaml │ ├── rules/ │ │ ├── example-installed-1.RULE - │ │ └── example-installed-1.yaml │ └── __init__.py/ ├── tests/ │ ├── data/ │ │ ├── example-installed-1.txt │ │ └── example-installed-1.txt.yml │ └── test_detection_datadriven.py - ├── gpl-1.0.LICENSE + ├── apache-2.0.LICENSE ├── MANIFEST.in ├── setup.cfg └── setup.py @@ -142,8 +138,9 @@ Then you can define a test class and call the ``build_tests`` method defined in TEST_DIR, clazz=TestLicenseDataDriven1, regen=scancode_config.REGEN_TEST_FIXTURES) -The ``tests/data`` directory contains a pair of files for each license: -a license text file and a YAML file specifying the expected license expressions from the test. +The ``tests/data`` directory contains a file for each license: +a license text file with a YAML frontmatter specifying the expected license expression +from the test. Finally, install the plugin and run the test: @@ -163,10 +160,8 @@ This is the basic structure of the example license directory:: additional_license_directory/ ├── licenses/ │ ├── example-installed-1.LICENSE - │ └── example-installed-1.yaml ├── rules/ │ ├── example-installed-1.RULE - │ └── example-installed-1.yaml Adding the licenses to the index ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From afa2d4c77dbca5acb4381f7633637ca94f069baa Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Sun, 26 Mar 2023 15:18:52 +0530 Subject: [PATCH 04/13] Update license detection reference documentation Refernce: https://github.com/nexB/scancode-toolkit/issues/3278 Signed-off-by: Ayan Sinha Mahapatra --- .../license-detection-reference.rst | 550 ++++++++++-------- 1 file changed, 296 insertions(+), 254 deletions(-) diff --git a/docs/source/explanations/license-detection-reference.rst b/docs/source/explanations/license-detection-reference.rst index 2ec1313f442..25e439e53d7 100644 --- a/docs/source/explanations/license-detection-reference.rst +++ b/docs/source/explanations/license-detection-reference.rst @@ -11,9 +11,10 @@ References: The Problem: ------------ -There was a lot of false-positives in scancode license results, specially -`unknown-license-reference` detections and to tackle this the following -solution elements were discussed: +The goal was to reduce false-positives in scancode license detection results, specially +`unknown-license-reference` detections and approximate detections reporting best-guess +license_expressions. To tackle this the following solution elements were discussed and +implemented: 1. Reporting the primary, declared license in a scan summary record 2. tagging mandatory portions in rules `#2773 `_ @@ -27,6 +28,8 @@ solution elements were discussed: 7. also apply LicenseDetection to package license detections `#2961 `_ 8. rename resource and package license fields `#2961 `_ +Some more elements are still WIP, see `issue #3300 `_ for more details on this. + What is a LicenseDetection? --------------------------- @@ -37,14 +40,23 @@ Properties: - A file can have multiple LicenseDetections (seperated by non-legalese lines) - This can be from a file directly or a package. -- We should be mostly certain of a proper detection to create a LicenseDetection. +- We should be mostly certain of a proper detection to report a LicenseDetection. - One LicenseDetection can have matches from different files, in case of local license references. +- We don't remove any detection matches, but we only add more matches to rectify and + correct the license_expression. + +Also there are two levels of reporting license detections: + +- File/package level License Detections +- Codebase level unique License Detections (summarized from the file/package level detections) Examples ^^^^^^^^ -A License Intro example:: +A License Intro example: + +Consider the following text:: /********************************************************************* * Copyright (c) 2019 Red Hat, Inc. @@ -57,8 +69,11 @@ A License Intro example:: **********************************************************************/ -The text ``"This program and the accompanying materials are made\n* available under the terms -of the",`` is detected as ``unknown-license-reference`` with ``is_license_intro`` as True, +The text:: + + "This program and the accompanying materials are made\n* available under the terms of the" + +is detected as ``unknown-license-reference`` with ``is_license_intro`` as True, and has several ``epl-2.0`` detections after that. This can be considered as a single License Detection with it's detected license-expression as @@ -86,12 +101,64 @@ files, and a concluded license_expression ``apache-2.0`` instead of the ``unknown-license-reference``. +Chnagelog Summary +----------------- + +- There is a new ``license_detections`` codebase level attribute with all the + unique license detections in the whole scan, both in resources and packages. + +- The data structure of the JSON output has changed for licenses at resource + level, also with new attribute names, ``licenses`` -> ``license_detections`` + and ``license_expressions`` -> ``detected_license_expression`` also with a + SPDX version of the same. As license detection attributes we have: + ``license_expression``, ``identifier`` and ``matches``. We also have a + ``detection_log`` (present optionally if the ``--license-diagnostics`` + option is enabled). + +- There are ``license_detections`` now reported at packages, and the data + structure of license attributes in ``package_data`` and the codebase level + ``packages`` has been also updated: ``license_expression`` -> + ``declared_license_expression``, also with it's SPDX version, + ``declared_license`` -> ``extracted_license_statement``, and also secondary + license detections data in: ``other_license_expression`` and + ``other_license_detections``. + +- Instead of reporting one match for each license ``key`` of a matched + license expression, we now report instead one single match for each matched + license expression, avoiding data duplication. Inside each match, we also + list each match and matched rule attributred directly to avoiding nesting. + +- License and Rule reference data is not reported at match level in license + detections and instead is reported at codebase-level with a new CLI option + ``--license-references`` as new attributes: ``license_references`` and + ``license_rule_references`` that list unique detected license and + license rules with their details. + + Change in License Data format: Resource --------------------------------------- -To move into the new LicenseDetection concept, the license data in scancode outputs has -undergone a major change. See the before/after results for a file to compare the -changes. +The data structure of the JSON output has changed for licenses at file level: + +- The ``licenses`` attribute is deleted. + +- A new ``license_detections`` attribute contains license detections in that file. + This object has three attributes: ``license_expression``, ``detection_log`` + and ``matches``. ``matches`` is a list of license matches and is roughly + the same as ``licenses`` in the previous version with additional structure + changes detailed below. + +- A new attribute ``license_clues`` contains license matches with the + same data structure as the ``matches`` attribute in ``license_detections``. + This contains license matches that are mere clues and where not considered + to be a proper conclusive license detection. + +- The ``license_expressions`` list of license expressions is deleted and + replaced by a ``detected_license_expression`` single expression. + Similarly ``spdx_license_expressions`` was removed and replaced by + ``detected_license_expression_spdx``. + +See the before/after results for a file to compare the changes. Before:: @@ -146,58 +213,31 @@ Before:: After:: - { - "detected_license_expression": "apache-2.0", - "detected_license_expression_spdx": "Apache-2.0", - "license_detections": [ - { - "license_expression": "apache-2.0", - "detection_log": [ - "not-combined" - ], - "matches": [ - { - "score": 100.0, - "start_line": 1, - "end_line": 1, - "matched_length": 4, - "match_coverage": 100.0, - "matcher": "1-hash", - "license_expression": "apache-2.0", - "rule_identifier": "apache-2.0_65.RULE", - "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE", - "referenced_filenames": [], - "is_license_text": false, - "is_license_notice": false, - "is_license_reference": false, - "is_license_tag": true, - "is_license_intro": false, - "rule_length": 4, - "rule_relevance": 100, - "matched_text": "License: Apache-2.0", - "licenses": [ - { - "key": "apache-2.0", - "name": "Apache License 2.0", - "short_name": "Apache 2.0", - "category": "Permissive", - "is_exception": false, - "is_unknown": false, - "owner": "Apache Software Foundation", - "homepage_url": "http://www.apache.org/licenses/", - "text_url": "http://www.apache.org/licenses/LICENSE-2.0", - "reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0", - "scancode_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE", - "spdx_license_key": "Apache-2.0", - "spdx_url": "https://spdx.org/licenses/Apache-2.0" - } - ] - } - ] - } - ], - "license_clues": [], - } + "detected_license_expression": "apache-2.0", + "detected_license_expression_spdx": "Apache-2.0", + "license_detections": [ + { + "license_expression": "apache-2.0", + "matches": [ + { + "score": 100.0, + "start_line": 1, + "end_line": 1, + "matched_length": 4, + "match_coverage": 100.0, + "matcher": "1-hash", + "license_expression": "apache-2.0", + "rule_identifier": "apache-2.0_65.RULE", + "rule_relevance": 100, + "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE", + "matched_text": "license: apache 2.0" + } + ], + "detection_log": [], + "identifier": "apache_2_0-ec759ae0-ea5a-f138-793e-388520e080c0" + } + ], + "license_clues": [], Change in License Data format: Package -------------------------------------- @@ -219,63 +259,33 @@ Before:: After:: - { - "declared_license_expression": "mit", - "declared_license_expression_spdx": "MIT", - "license_detections": [ - { - "license_expression": "mit", - "detection_log": [ - "not-combined" - ], - "matches": [ - { - "score": 100.0, - "start_line": 1, - "end_line": 1, - "matched_length": 4, - "match_coverage": 100.0, - "matcher": "1-hash", - "license_expression": "mit", - "rule_identifier": "mit_in_manifest.RULE", - "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_in_manifest.RULE", - "referenced_filenames": [ - "LICENSE" - ], - "is_license_text": false, - "is_license_notice": false, - "is_license_reference": true, - "is_license_tag": false, - "is_license_intro": false, - "rule_length": 4, - "rule_relevance": 100, - "matched_text": ":type = MIT, :file = LICENSE", - "licenses": [ - { - "key": "mit", - "name": "MIT License", - "short_name": "MIT License", - "category": "Permissive", - "is_exception": false, - "is_unknown": false, - "owner": "MIT", - "homepage_url": "http://opensource.org/licenses/mit-license.php", - "text_url": "http://opensource.org/licenses/mit-license.php", - "reference_url": "https://scancode-licensedb.aboutcode.org/mit", - "scancode_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/mit.LICENSE", - "spdx_license_key": "MIT", - "spdx_url": "https://spdx.org/licenses/MIT" - } - ] - } - ] - } - ], - "other_license_expression": null, - "other_license_expression_spdx": null, - "other_license_detections": [], - "extracted_license_statement": ":type = MIT, :file = LICENSE", - } + "declared_license_expression": "mit", + "declared_license_expression_spdx": "MIT", + "license_detections": [ + { + "license_expression": "mit", + "matches": [ + { + "score": 100.0, + "start_line": 1, + "end_line": 1, + "matched_length": 4, + "match_coverage": 100.0, + "matcher": "1-hash", + "license_expression": "mit", + "rule_identifier": "mit_in_manifest.RULE", + "rule_relevance": 100, + "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_in_manifest.RULE", + "matched_text": ":type = MIT, :file = LICENSE" + } + ], + "identifier": "mit-74f1df5b-f94d-2423-6bb8-3e4d809c26a5" + } + ], + "other_license_expression": null, + "other_license_expression_spdx": null, + "other_license_detections": [], + "extracted_license_statement": ":type = MIT, :file = LICENSE", Previously in package data only the license_expression was present and it was very hard to debug license detections. Now there's a ``license_detections`` field with the detections, same as @@ -283,67 +293,67 @@ the resource ``license_detections``, with additional ``declared_license_expressi ``other_license_expression`` with their SPDX counterparts. The ``declared_license`` field also has been renamed to ``extracted_license_statement``. - +.. _license_detections_unique: New codebase level Unique License Detection ------------------------------------------- We now have a new codebase level attribute ``license_detections`` which has Unique -license detection across the codebase, in both packages and resources. There is also -a new resource level attribute to reference to the codebase level unique license -detections, which is ``for_license_detections``. +license detection across the codebase, in both packages and resources. They are +linked by a common attribute ``identifier`` containing the ``license_expression`` +and a UUID generated from the match content. The match level data is only present +at the resource level if needed, to look at details. New codebase level attribute:: { "license_detections": [ { - "identifier": "epl_1_0-1867eafe-a258-cbb4-408f-2bd33d02ee23", + "identifier": "epl_1_0-583490fb-0b3a-f445-a1b9-1b96423b9ec3", "license_expression": "epl-1.0", - "count": 2, - "detection_log": [ - "not-combined" - ], - "matches": [ - { - "score": 99.34, - "start_line": 12, - "end_line": 25, - "matched_length": 150, - "match_coverage": 99.34, - "matcher": "3-seq", - "license_expression": "epl-1.0", - "rule_identifier": "epl-1.0_3.RULE", - "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/epl-1.0_3.RULE" - }, - { - "score": 100.0, - "start_line": 17, - "end_line": 17, - "matched_length": 8, - "match_coverage": 100.0, - "matcher": "2-aho", - "license_expression": "epl-1.0", - "rule_identifier": "epl-1.0_7.RULE", - "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/epl-1.0_7.RULE" - } - ] + "detection_count": 2, + "detection_log": [] } ] } -New resource level attribute:: +For the corresponding resource level license detection:: + + "license_detections": [ + { + "license_expression": "epl-1.0", + "matches": [ + { + "score": 99.34, + "start_line": 12, + "end_line": 25, + "matched_length": 150, + "match_coverage": 99.34, + "matcher": "3-seq", + "license_expression": "epl-1.0", + "rule_identifier": "epl-1.0_3.RULE", + "rule_relevance": 100, + "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/epl-1.0_3.RULE", + }, + { + "score": 100.0, + "start_line": 17, + "end_line": 17, + "matched_length": 8, + "match_coverage": 100.0, + "matcher": "2-aho", + "license_expression": "epl-1.0", + "rule_identifier": "epl-1.0_7.RULE", + "rule_relevance": 100, + "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/epl-1.0_7.RULE", + } + ], + "detection_log": [], + "identifier": "epl_1_0-583490fb-0b3a-f445-a1b9-1b96423b9ec3" + } + ] - { - "files": [ - { - "for_license_detections": [ - "epl_1_0-1867eafe-a258-cbb4-408f-2bd33d02ee23" - ] - } - ] - } LicenseMatch Result Data @@ -352,108 +362,136 @@ LicenseMatch Result Data LicenseMatch data was based on a ``license key`` instead of being based on an ``license-expression``. -So if there is a ``mit and apache-2.0`` license expression detected from a single -LicenseMatch, there was two entries in the ``licenses`` list for that -resource, one for each license key, (here ``mit`` and ``apache-2.0`` respectively). -This repeats the match details as these two entries have the same details except the -license key. And this is wrong. +So if there is a ``gpl-2.0 AND patent-disclaimer`` license expression detected +from a single LicenseMatch, there was two entries in the ``licenses`` list +for that resource, one for each license key, (here ``gpl-2.0`` and +``patent-disclaimer`` respectively). This repeats the match details as these +two entries have the same details except the license key. -We should only add one entry per match (and therefore per ``rule``) and here the -primary attribute should be the ``license-expression``, rather than the ``license-key``. +We should only add one entry per match (and therefore per ``rule``) and here +the primary attribute should be the ``license-expression``, rather than the +``license-key``. -We also create a mapping inside a mapping in these license details to refer to the -license rule (and there are other incosistencies in how we report here). We should -just report a flat mapping here, (with a list at last for each of the license keys). +We also used to create a mapping inside a mapping in these license details +to refer to the license rule (and there are other incosistencies in how we +report here). We are now just reporting a flat mapping here, and all the +rule details are also not present in the license match, and only available +as a reference optionally. See this before/after comparision to see how the license data in results has eveolved. Before:: - { - "key": "apache-2.0", - "score": 100.0, - "name": "Apache License 2.0", - "short_name": "Apache 2.0", - "category": "Permissive", - "is_exception": false, - "is_unknown": false, - "owner": "Apache Software Foundation", - "homepage_url": "http://www.apache.org/licenses/", - "text_url": "http://www.apache.org/licenses/LICENSE-2.0", - "reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0", - "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE", - "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml", - "spdx_license_key": "Apache-2.0", - "spdx_url": "https://spdx.org/licenses/Apache-2.0", - "start_line": 1, - "end_line": 1, - "matched_rule": { - "identifier": "apache-2.0_65.RULE", - "license_expression": "apache-2.0", - "licenses": [ - "apache-2.0" - ], - "referenced_filenames": [], - "is_license_text": false, - "is_license_notice": false, - "is_license_reference": false, - "is_license_tag": true, - "is_license_intro": false, - "has_unknown": false, - "matcher": "1-hash", - "rule_length": 4, - "matched_length": 4, - "match_coverage": 100.0, - "rule_relevance": 100, - "is_builtin": true + "licenses": [ + { + "key": "gpl-2.0", + "score": 100.0, + "name": "GNU General Public License 2.0", + "short_name": "GPL 2.0", + "category": "Copyleft", + "is_exception": false, + "is_unknown": false, + "owner": "Free Software Foundation (FSF)", + "homepage_url": "http://www.gnu.org/licenses/gpl-2.0.html", + "text_url": "http://www.gnu.org/licenses/gpl-2.0.txt", + "reference_url": "https://scancode-licensedb.aboutcode.org/gpl-2.0", + "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.LICENSE", + "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.yml", + "spdx_license_key": "GPL-2.0-only", + "spdx_url": "https://spdx.org/licenses/GPL-2.0-only", + "start_line": 4, + "end_line": 30, + "matched_rule": { + "identifier": "gpl-2.0_and_patent-disclaimer_3.RULE", + "license_expression": "gpl-2.0 AND patent-disclaimer", + "licenses": [ + "gpl-2.0", + "patent-disclaimer" + ], + "referenced_filenames": [], + "is_license_text": false, + "is_license_notice": true, + "is_license_reference": false, + "is_license_tag": false, + "is_license_intro": false, + "has_unknown": false, + "matcher": "2-aho", + "rule_length": 185, + "matched_length": 185, + "match_coverage": 100.0, + "rule_relevance": 100 + } }, - "matched_text": "License: Apache-2.0" - } + { + "key": "patent-disclaimer", + "score": 100.0, + "name": "Generic patent disclaimer", + "short_name": "Generic patent disclaimer", + "category": "Permissive", + "is_exception": false, + "is_unknown": false, + "owner": "Unspecified", + "homepage_url": null, + "text_url": "", + "reference_url": "https://scancode-licensedb.aboutcode.org/patent-disclaimer", + "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/patent-disclaimer.LICENSE", + "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/patent-disclaimer.yml", + "spdx_license_key": "LicenseRef-scancode-patent-disclaimer", + "spdx_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/patent-disclaimer.LICENSE", + "start_line": 4, + "end_line": 30, + "matched_rule": { + "identifier": "gpl-2.0_and_patent-disclaimer_3.RULE", + "license_expression": "gpl-2.0 AND patent-disclaimer", + "licenses": [ + "gpl-2.0", + "patent-disclaimer" + ], + "referenced_filenames": [], + "is_license_text": false, + "is_license_notice": true, + "is_license_reference": false, + "is_license_tag": false, + "is_license_intro": false, + "has_unknown": false, + "matcher": "2-aho", + "rule_length": 185, + "matched_length": 185, + "match_coverage": 100.0, + "rule_relevance": 100 + } + } + ], + "license_expressions": [ + "gpl-2.0 AND patent-disclaimer" + ], After:: - { - "score": 100.0, - "start_line": 1, - "end_line": 1, - "matched_length": 4, - "match_coverage": 100.0, - "matcher": "1-hash", - "license_expression": "apache-2.0", - "rule_identifier": "apache-2.0_65.RULE", - "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE", - "referenced_filenames": [], - "is_license_text": false, - "is_license_notice": false, - "is_license_reference": false, - "is_license_tag": true, - "is_license_intro": false, - "rule_length": 4, - "rule_relevance": 100, - "matched_text": "License: Apache-2.0", - "licenses": [ - { - "key": "apache-2.0", - "name": "Apache License 2.0", - "short_name": "Apache 2.0", - "category": "Permissive", - "is_exception": false, - "is_unknown": false, - "owner": "Apache Software Foundation", - "homepage_url": "http://www.apache.org/licenses/", - "text_url": "http://www.apache.org/licenses/LICENSE-2.0", - "reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0", - "scancode_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE", - "spdx_license_key": "Apache-2.0", - "spdx_url": "https://spdx.org/licenses/Apache-2.0" - } - ] - } - + "license_detections": [ + { + "license_expression": "gpl-2.0 AND patent-disclaimer", + "matches": [ + { + "score": 100.0, + "start_line": 4, + "end_line": 30, + "matched_length": 185, + "match_coverage": 100.0, + "matcher": "2-aho", + "license_expression": "gpl-2.0 AND patent-disclaimer", + "rule_identifier": "gpl-2.0_and_patent-disclaimer_3.RULE", + "rule_relevance": 100, + "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/gpl-2.0_and_patent-disclaimer_3.RULE" + } + ], + "identifier": "gpl_2_0_and_patent_disclaimer-3bb2602f-86f5-b9da-9bf5-b52e6920c8d1" + } + ], Only reference License related Data @@ -464,7 +502,7 @@ a lot of information. This repeatation exists in three levels: - License-level Data (a license-key) - Rule-level Data (a license rule) -- LicenseDetection Data +- LicenseDetection Data (a license detection) License Data ^^^^^^^^^^^^ @@ -643,7 +681,11 @@ After:: LicenseDetection Data ^^^^^^^^^^^^^^^^^^^^^ -This is referencing by LicenseDetections, and has one or multiple license Matches. -This is not referenced to a top-level list, but there could be a list of ambiguous -detections as a summary to review. This is WIP, see -`scancode-toolkit#3122 `_. +This is referencing by LicenseDetections objcts, and has one or multiple +license matches. This is linked to the resource level detections through +an ``identifier`` attribute present in both resource and codebase level +detections. See the `unique license detections section `_ +above for more details on this. + +There could be a list of ambiguous detections as a summary to review. +This is WIP, see `scancode-toolkit#3122 `_. From 3522153024bb8c496fa1634cc1816a8f0cc95bf6 Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Sun, 26 Mar 2023 15:20:25 +0530 Subject: [PATCH 05/13] Misc doc changes and test fixes Signed-off-by: Ayan Sinha Mahapatra --- docs/source/contribute/index.rst | 1 - .../source/contribute/long_running_issues.rst | 181 ------------------ .../license-detection-reference.rst | 4 +- docs/source/getting-started/install.rst | 4 +- docs/source/getting-started/newcomer.rst | 18 +- docs/source/how-to-guides/add_new_license.rst | 2 +- .../add_new_license_detection_rule.rst | 2 +- .../install_new_license_plugin.rst | 2 +- 8 files changed, 10 insertions(+), 204 deletions(-) delete mode 100644 docs/source/contribute/long_running_issues.rst diff --git a/docs/source/contribute/index.rst b/docs/source/contribute/index.rst index 69d83fe7ecc..142bd90f77f 100644 --- a/docs/source/contribute/index.rst +++ b/docs/source/contribute/index.rst @@ -11,4 +11,3 @@ gsoc17_final_report gsoc19_final_report gsoc21_final_report - long_running_issues diff --git a/docs/source/contribute/long_running_issues.rst b/docs/source/contribute/long_running_issues.rst deleted file mode 100644 index aaec386a1db..00000000000 --- a/docs/source/contribute/long_running_issues.rst +++ /dev/null @@ -1,181 +0,0 @@ -Long Running Issues - Help Needed -================================= - -Long Running Issues - Scancode Toolkit --------------------------------------- - -- `Good First Issues List `_ -- `First Timers Only Issues List `_ -- `Add improved documentation for V3.0 `_ -- `Documentation Roadmap `_ -- `Documentation Inconsistencies Tracker `_ - -.. - [Suggestions for LRIs][ToDo] - - People Interested in GSoC (And PRs Opened by Them) - -Roadmaps --------- - -This section provides a overview of ScanCode developement roadmap, in the sense what maintainers -are considering, what the long-term goals are. - -- :ref:`roadmap` -- `Milestones `_ -- `Documentation Roadmap `_ - -.. _good_first_issue: - -Good First Issue ----------------- - -A `good first issue `_ -means it's recommended for people who haven't contributed to our codebase before. - -A ``Solving Good First Issue`` is basically an issue based guide for new contributors to -solve a ``Good First Issue``. - -Manitainers of ScanCode Toolkit label small bugs as good first issues for new contributors. - -.. _good_1st_issue_links: - -Important Links -^^^^^^^^^^^^^^^ - -- Issues labeled `good first issue `_ -- The status `list of good first issue `_ -- opening a `Solving Good First Issue `_ - -.. _good_1st_issue_understand_b4_solving: - -What you should understand before Solving a Good First Issue -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- ScanCode Toolkit tracks different kinds of `issues using GitHub `_. -- Some of these issues are labeled ``Good First Issue`` by the Maintainers, as in they - recommend them. -- A list of ``Good First Issue`` is also maintained `here `_. -- The list of ``Good First Issue`` contains a list of Issues, people assigned to them - (or waiting to be assigned) and their status (active/completed). -- Issues here are marked Solved using a `Task List `_. -- Contributors come at this issue (and not at individual issues) and ask the maintainers to - be assigned. -- After being assigned, contributors open another Issue labeled ``Solving Good First Issue`` - so maintainers can Track their Progress on that Issue. -- The task list in the new issue opened (labeled ``Solving Good First Issue``) has to be updated by - the Contributor. - -.. _good_1st_issue_workflow: - -Step by Step Workflow : Contributor's Tasks -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -#. Look at the Issues in `Good First Issues List `_ - These are basically all the `issues labeled Good First Issue `_ - with information on people assigned to the issues, and their status. - -#. Select a ``Good First Issue`` which isn't assigned and comment in the `Good First Issues List `_ - asking the maintainers to assign you. - -#. After you are assigned to that Issue by a maintainer, make sure you are assigned to the one you - requested. It'll look like ``[ ] #1724 @AyanSinhaMahapatra`` where ``#1724`` is the Issue number - and ``AyanSinhaMahapatra`` is your GitHub user ID. - -#. Open an Issue labeled `Solving Good First Issue here `_ . - -#. In this issue you opened, reference the following: - (include ``#[number]`` in comments, like ``#1724``) - - - The Original issue labeled ``Good First Issue`` that you are solving. - - The Pull Request you open later. - - The `Good First Issues List `_ - -#. Follow the Instruction on that Issue (It's a Template you'll just have to mark the Task List - as you complete tasks) - - - **Claim this issue** - - **Reproduce** - - **Create a Unit test** - - **Come up with a Solution** - - **Code your Solution** - - **Run the tests locally** - - **Commit and Push** - - **Start a Pull Request** - - **Make Sure all the Tests Pass** - - **Done** Ask in comments for a review :) - -#. Remind maintainers to close both Issues, after your Pull Request is Merged. - - - The Original issue labeled ``Good First Issue`` - - The issue opened by you labeled ``Solving Good First Issue`` - -.. _first_timers_only: - -First Timers Only ------------------ - -A `first timers only `_. -issue means we've worked to make it more legible to folks who either **haven't contributed to our -codebase before, or even folks who haven't contributed to open source before**. - -Small documenation inconsistencies are first timers only issues for new contributors. - -Important Links -^^^^^^^^^^^^^^^ - -- Issues labeled `first timers only `_ -- The status list of `all first timers only `_ Issues -- All `Documentation Inconsistencies Issue `_. -- opening a `First Timers Only Issue `_ - -What you should understand before Solving a First Timers Only Issue -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- ScanCode Toolkit tracks different kinds of `issues using GitHub `_. -- Some of these issues are documentation inconsistencies `listed here `_. -- A list of ``First Timers Only`` Issues is also `maintained here `_. -- The list of ``First Timers Only`` contains a list of Issues, people assigned to them - (or waiting to be assigned) and their status (active/completed). -- Issues here are marked Solved using a `Task List `_. -- Contributors come at this issue and ask the maintainers to be assigned. -- After being assigned, contributors open another Issue labeled ``First Timers Only`` - so maintainers can Track their Progress. -- The task list has to be updated by the Contributor. - -Step by Step Workflow : Contributor's Tasks -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -#. Look at the Issues in ``First Timers Only`` `List `_ - These are basically all the `issues `_ - labeled ``First Timers Only`` issues, with information on people assigned to the issues, - and their status. - -#. Select a ``First Timers Only`` issue which isn't assigned to anyone yet, and comment on the - `List `_ asking the maintainers to assign - you. - -#. After you are assigned to that Issue by a maintainer, make sure you are assigned to the one you - requested. It'll look like ``[ ] #1724 @AyanSinhaMahapatra`` where ``#1724`` is the Issue number - and ``AyanSinhaMahapatra`` is your GitHub user ID. - -#. Open an Issue labeled `First Timers Only here `_ . - -#. In this issue you opened, reference the following: - (include ``#[number]`` in comments, like ``#1724``) - - - The Original issue labeled ``Good First Issue`` that you are solving. - - The Pull Request you open later. - - The `Good First Issues List `_ - -#. Follow the Instruction on that Issue (It's a Template you'll just have to mark the Task List - as you complete tasks) - - - **Issue Claimed** - - **Review Guidelines** - - **Refer Documentation** - - **Build Docs Locally** - - **Update** - - **Commit and Push** - - **Start a Pull Request** - - **Make Sure all the Tests Pass** - - **Done** Ask in comments for a review :) diff --git a/docs/source/explanations/license-detection-reference.rst b/docs/source/explanations/license-detection-reference.rst index 25e439e53d7..a8d50f51fba 100644 --- a/docs/source/explanations/license-detection-reference.rst +++ b/docs/source/explanations/license-detection-reference.rst @@ -303,7 +303,7 @@ We now have a new codebase level attribute ``license_detections`` which has Uniq license detection across the codebase, in both packages and resources. They are linked by a common attribute ``identifier`` containing the ``license_expression`` and a UUID generated from the match content. The match level data is only present -at the resource level if needed, to look at details. +at the resource level if needed, to look at details. New codebase level attribute:: @@ -684,7 +684,7 @@ LicenseDetection Data This is referencing by LicenseDetections objcts, and has one or multiple license matches. This is linked to the resource level detections through an ``identifier`` attribute present in both resource and codebase level -detections. See the `unique license detections section `_ +detections. See the `unique license detections section `_ above for more details on this. There could be a list of ambiguous detections as a summary to review. diff --git a/docs/source/getting-started/install.rst b/docs/source/getting-started/install.rst index 71aaef2056f..c4dbc8c72f1 100644 --- a/docs/source/getting-started/install.rst +++ b/docs/source/getting-started/install.rst @@ -123,7 +123,7 @@ operating system by going to the `project releases page `_ -under assets options. +under assets options. .. Note:: @@ -347,7 +347,7 @@ the standard Python package management tool. Apple M1 chips, as some non-native dependencies does not have pre-built wheels for ARM (like py-ahocorasick, intbitset). See :ref:`system_requirements` for more information. See related issues for more info: - + - `Fallback pure-python deps `_ - `pip install failing on M1 `_ diff --git a/docs/source/getting-started/newcomer.rst b/docs/source/getting-started/newcomer.rst index de02397c96f..3fd1f43dcfc 100644 --- a/docs/source/getting-started/newcomer.rst +++ b/docs/source/getting-started/newcomer.rst @@ -252,17 +252,6 @@ Good First Issues A `good first issue `_ means it's recommended for people who haven't contributed to Scancode Toolkit before. -#. Refer the detailed documentation for :ref:`good_first_issue`. -#. :ref:`good_1st_issue_links` for Good First issues are also compiled. -#. :ref:`good_1st_issue_understand_b4_solving` -#. :ref:`good_1st_issue_workflow` - -A `first timers only `_ -issue means we've worked to make it more legible to folks who either **haven't contributed to our -codebase before, or even folks who haven't contributed to open source before**. - -Refer the detailed documentation for :ref:`first_timers_only`. - .. _newcomer_add_functionalirty: Add new Functionality/Enhancement to ScanCode @@ -324,14 +313,13 @@ If you want to participate in any of the two programs: Then: #. Keep an eye out for Application Timelines. -#. Solve multiple of these :ref:`good_first_issue` to demonstrate your skills, +#. Solve multiple of these :ref:`newcomer_good_first_issue` to demonstrate your skills, and improve your chances of selection. #. Refer the Projects Ideas list for details on tentative projects. - - :ref:`GSoC2019` - - :ref:`GSoD2019` + - `GSoC2023 `_ -#. Remain active in Gitter and talk with the organization mentors well ahead of the deadlines. +#. Remain active in Element and talk with the organization mentors well ahead of the deadlines. #. Select projects according to your skills and finalize project proposals. #. Discuss your proposals extensively with corresponding mentors. #. Apply for the Programs well before the Deadline. diff --git a/docs/source/how-to-guides/add_new_license.rst b/docs/source/how-to-guides/add_new_license.rst index 5c441cdfa4c..cd671056394 100644 --- a/docs/source/how-to-guides/add_new_license.rst +++ b/docs/source/how-to-guides/add_new_license.rst @@ -54,7 +54,7 @@ And more attributes which are not mandatory but nice to have always (if applicab We want to use `minimum_coverage` when there are other licenses which are very similar and we want to make sure we match correctly these licenses, and `notes` for interesting cases of licenses with descriptions to help identify origin, similarities to other licenses, -notes about the SPDX keys and others. +notes about the SPDX keys and others. Some URLs: diff --git a/docs/source/how-to-guides/add_new_license_detection_rule.rst b/docs/source/how-to-guides/add_new_license_detection_rule.rst index b24970ecc97..9893aca9a17 100644 --- a/docs/source/how-to-guides/add_new_license_detection_rule.rst +++ b/docs/source/how-to-guides/add_new_license_detection_rule.rst @@ -34,7 +34,7 @@ might have a filename: `mit_and_apache-2.0_10.RULE`. Save your rule text in this file, if there are specific words like company names, projects or other, it is better to have rules with and without these so we have -better detection. +better detection. For a simple `mit AND apache-2.0` license expression detection, here is an example rule file:: diff --git a/docs/source/how-to-guides/install_new_license_plugin.rst b/docs/source/how-to-guides/install_new_license_plugin.rst index d527c95d068..5d6416e0404 100644 --- a/docs/source/how-to-guides/install_new_license_plugin.rst +++ b/docs/source/how-to-guides/install_new_license_plugin.rst @@ -139,7 +139,7 @@ Then you can define a test class and call the ``build_tests`` method defined in clazz=TestLicenseDataDriven1, regen=scancode_config.REGEN_TEST_FIXTURES) The ``tests/data`` directory contains a file for each license: -a license text file with a YAML frontmatter specifying the expected license expression +a license text file with a YAML frontmatter specifying the expected license expression from the test. Finally, install the plugin and run the test: From 8eba8ce43dab9487ab7a83dcb71d6159d6e7ba0e Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Mon, 27 Mar 2023 19:06:40 +0530 Subject: [PATCH 06/13] Update contribution and other misc docs Signed-off-by: Ayan Sinha Mahapatra --- docs/source/contribute/contrib_dev.rst | 6 ++ docs/source/contribute/cut_new_release.rst | 77 ++++++++++++---------- docs/source/contribute/roadmap.rst | 16 +++-- docs/source/misc/support.rst | 15 +++-- docs/source/misc/versioning.rst | 2 +- 5 files changed, 72 insertions(+), 44 deletions(-) diff --git a/docs/source/contribute/contrib_dev.rst b/docs/source/contribute/contrib_dev.rst index 6241175aa18..8f8ad892f7c 100644 --- a/docs/source/contribute/contrib_dev.rst +++ b/docs/source/contribute/contrib_dev.rst @@ -198,3 +198,9 @@ Using ScanCode as a Python library ScanCode can be used alright as a Python library and is available as as a Python wheel in Pypi and installed with ``pip install scancode-toolkit`` or ``pip install scancode-toolkit-mini``. + +.. _note: + + Since we do not pin dependencies to avoid dependency resolution conflicts + for downstream users, there are possibilities of issues arising from + dependencies silently changing API/functions which scancode uses. diff --git a/docs/source/contribute/cut_new_release.rst b/docs/source/contribute/cut_new_release.rst index 8dea6a47d40..2147011cb71 100644 --- a/docs/source/contribute/cut_new_release.rst +++ b/docs/source/contribute/cut_new_release.rst @@ -1,59 +1,68 @@ +.. _release_process: + How to cut a new release: ========================= Update version -------------- -- Run bumpversion with major, minor or patch to bump the version in - ``setup.cfg`` and ``src/scancode_config.py``. Note that this is CalVer. +- Bump version to update major, minor or patch version in ``setup.cfg`` + ``setup-mini.cfg`` and ``src/scancode_config.py``. Note that this is SemVer, + though we used CalVer previously, we have switched back to SemVer. + +- If scancode output data format is changed, increment manually the major, + minor or patch version to bump the version in ``src/scancode_config.py``. + Note that this is SemVer. -- If scancode output data format is changed, increment manually the major, minor or patch - to bump the version in ``src/scancode_config.py``. Note that this is SemVer. +See our :ref:``versioning`` for more details. Tag and publish --------------- -- Update the CHANGELOG.rst +- Changes for a release should also be pushed to a branch and a pull + Request should be created for it, for review. + +- Update the CHANGELOG.rst with detailed documentation of updates + and API/CLI option changes, or any significant changes. -- Commit these changes and push changes to develop: +- Commit these changes and push changes to develop (here we use an + example tag ``v1.6.1``): - - ``git commit -m "commit message"`` - - ``git push --set-upstream origin develop`` + - ``git commit -s`` + - ``git push --set-upstream origin release-prep-v1.6.1`` -- Merge develop branch in master and tag the release. +- Merge this ``release-prep-v1.6.1`` branch in master after review approval + and tag the release: - - ``git checkout master`` - - ``git merge develop`` - ``git tag -a v1.6.1 -m "Release v1.6.1"`` - - ``git push --set-upstream origin master`` + - ``git push --set-upstream origin release-prep-v1.6.1`` - ``git push --set-upstream origin v1.6.1`` -- Draft a new release in GitHub, using the previous release blurb as a base. Highlight new and - noteworthy changes from the CHANGELOG.rst. - -Create Release on GitHub ------------------------- - -- Run ``etc/release/scancode_release.sh`` locally. +Automated Release Process +------------------------- -- Upload the release archives created in the ``dist/`` directory to the GitHub release page. +- We have a `automated release script `_ + triggered by a pushed tag, where jobs run to: -- Save the release as a draft. Use the previous release notes to create notes in the same style. - Ensure that the link to third-party source code is present. + - Build pypi wheels and sdist archives + - Build app release archives for linux/mac/windows + - This happens for all supported python versions + - Test these wheels and app archives in linux/mac/windows for all supported + versions of python + - Create a GitHub release (draft by default) with all wheels, sdists and app arcives + (for all os/python combinations) + - Upload sdists and wheels (all python versions) and publish a release + (This won't be a stable release for beta/release-candidate tags) -- test the downloads. +- Populate the draft GitHub release by clicking the ``Generate Release Notes`` button + and this pre-populates the release notes with PRs and contributors. -- publish the release on GitHub +- Add more details to the release notes talking about the key features and changes in the + release. -Upload wheels to pypi and Test ------------------------------- +- Publish the release on GitHub + (Note the ``Set as a pre-release`` vs ``Set as the latest release`` checkboxes) -- then build and publish the released wheel on Pypi. For this you need your own Pypi credentials - (and get authorized to publish Pypi release: ask @pombredanne) and you need to have the ``twine`` - package installed and configured. +- Announce in public channels and chats about the release - - Build a ``.whl`` and source distribution with ``python setup.py release`` - - Run twine with ``twine upload dist/`` - - Once uploaded check the published release at https://pypi.python.org/pypi/scancode-toolkit/ - - Then create a new fresh local virtualenv and test the wheel installation with: - ``pip install scancode-toolkit[full]`` +- Do test the release archives yourself. diff --git a/docs/source/contribute/roadmap.rst b/docs/source/contribute/roadmap.rst index 540a6ce9cdd..3167335c316 100644 --- a/docs/source/contribute/roadmap.rst +++ b/docs/source/contribute/roadmap.rst @@ -5,6 +5,14 @@ Roadmap This is a high level list of what we are working on and what is completed. +.. ToDo: Move this to Discussions or wikis as this is not used or updated + regularly. + +.. _note: + + This is not updated regularly, see the `milstones `_ + instead for updated shorter and longer term roadmaps. + Legend ------ @@ -26,7 +34,7 @@ Package manifest and dependency parsers - |white_large_square| RPM dependencies #649 - |white_large_square| Windows Nuget dependencies #648 - |white_check_mark| Bower packages #654 -- |clock1030| Python dependencies #653 +- |white_check_mark| Python dependencies #653 - |white_check_mark| CRAN - |white_check_mark| Plain packages - |white_large_square| other Java-related meta files (SBT, Ivy, Gradle, etc.) @@ -39,7 +47,7 @@ License Detection - |white_check_mark| support and detect license expressions (code in https://github.com/nexB/license-expression) - |clock1030| support and detect composite licenses -- |white_large_square| support custom licenses +- |white_check_mark| support custom licenses - |white_large_square| move licenses data set to external separate repository - |white_check_mark| Improved unknown license detection - |white_check_mark| sync with external sources (DejaCode, SPDX, etc.) @@ -61,8 +69,8 @@ Core features - |white_check_mark| scan plugins (e.g. plugins that run a scan to collect data) - |white_check_mark| support Python 3 #295 - |clock1030| transparent archive extraction (as opposed to on-demand with extractcode) -- |clock1030| scancode.yml configuration file for exclusions, defaults, scan failure conditions, - etc. +- |white_large_square| scancode.yml configuration file for exclusions, defaults, scan + failure conditions, etc. - |white_large_square| support scan pipelines and rules to organize more complex scans - |white_check_mark| scan baselining, delta scan and failure conditions (such as license change, etc) ( spawned as its the `DeltaCode `_ project) diff --git a/docs/source/misc/support.rst b/docs/source/misc/support.rst index 48064e2f8b0..9c0a0cef628 100644 --- a/docs/source/misc/support.rst +++ b/docs/source/misc/support.rst @@ -6,18 +6,23 @@ Support Documentation ------------- -The ScanCode toolkit documentation lives at aboutcode.readthedocs.io/en/latest/scancode-toolkit/. - +The ScanCode toolkit documentation lives at https://scancode-toolkit.readthedocs.io/. Issue Tracker ------------- -Post questions and bugs as GitHub tickets at: https://github.com/nexB/scancode-toolkit/issues +Post issues you are having and bugs as `GitHub tickets `_ + +Discussions +----------- +If you want to ask questions or anything else that you think are not bugs/new +features open a `discussion `_ Join the conversation ---------------------- -Join our `Gitter Channel `_ to chat -with users and developers of ScanCode Toolkit. +Join our `general chatroom `_ to chat +with aboutcode community members and if you want to talk to users and developers of +ScanCode Toolkit, use `scancode room `_ diff --git a/docs/source/misc/versioning.rst b/docs/source/misc/versioning.rst index b87451a3f41..f2bc3619851 100644 --- a/docs/source/misc/versioning.rst +++ b/docs/source/misc/versioning.rst @@ -1,4 +1,4 @@ -.. versioning: +.. _versioning: Versioning approach From 67256a1d5312d2b9a09e99ca3f53f7bd3c658ccc Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Tue, 28 Mar 2023 15:33:21 +0530 Subject: [PATCH 07/13] Add auto generated doc page on available package parsers - Add new console script `regen-package-docs` - Add new shell script `regen_package_docs.sh` - Add new jinja template to generate docs using data - Add this script on makefile to run automatically on doc generation - Move explanations -> reference section Reference: https://github.com/nexB/scancode-toolkit/issues/3298 Signed-off-by: Ayan Sinha Mahapatra --- docs/Makefile | 1 + docs/scripts/regen_package_docs.sh | 12 + docs/source/contribute/contrib_doc.rst | 12 + docs/source/explanations/index.rst | 24 - docs/source/index.rst | 9 +- .../reference/available_package_parsers.rst | 769 ++++++++++++++++++ docs/source/reference/index.rst | 11 + .../license-detection-reference.rst | 0 .../{explanations => reference}/overview.rst | 0 setup.cfg | 1 + src/packagedcode/plugin_package.py | 47 +- src/packagedcode/regen_package_docs.py | 78 ++ .../templates/available_package_parsers.rst | 33 + 13 files changed, 960 insertions(+), 37 deletions(-) create mode 100755 docs/scripts/regen_package_docs.sh delete mode 100644 docs/source/explanations/index.rst create mode 100644 docs/source/reference/available_package_parsers.rst create mode 100644 docs/source/reference/index.rst rename docs/source/{explanations => reference}/license-detection-reference.rst (100%) rename docs/source/{explanations => reference}/overview.rst (100%) create mode 100644 src/packagedcode/regen_package_docs.py create mode 100644 src/packagedcode/templates/available_package_parsers.rst diff --git a/docs/Makefile b/docs/Makefile index d0c3cbf1020..649b9e371a3 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -18,3 +18,4 @@ help: # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + ./scripts/regen_package_docs.sh diff --git a/docs/scripts/regen_package_docs.sh b/docs/scripts/regen_package_docs.sh new file mode 100755 index 00000000000..d5e0ff8fad0 --- /dev/null +++ b/docs/scripts/regen_package_docs.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# halt script on error +set -e + +# Path to the available packages doc +DOC_PATH="source/reference/available_package_parsers.rst" + +# Regen docs for available package parsers +regen-package-docs --path "$DOC_PATH" + +# Delete whitespace from last line to not fail doc8 tests +sed -i '$ d' "$DOC_PATH" \ No newline at end of file diff --git a/docs/source/contribute/contrib_doc.rst b/docs/source/contribute/contrib_doc.rst index a8c787cbd3e..72d460ea6f9 100644 --- a/docs/source/contribute/contrib_doc.rst +++ b/docs/source/contribute/contrib_doc.rst @@ -312,3 +312,15 @@ bugs. But this is definitely better than converting everything by yourself. This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for Sphinx/ReadTheDocs hosting. + +Automatic Docs Generation +------------------------- + +It's possible to generate docs automatically from data by using a combination of: + +- `shell scripts: example `_ +- `python scripts: example `_ +- `jinja templates: example `_ + +And we do this currently to keep a documentation page for all the supported package formats. +See :ref:`supported_packages` for details. diff --git a/docs/source/explanations/index.rst b/docs/source/explanations/index.rst deleted file mode 100644 index e96c2519c44..00000000000 --- a/docs/source/explanations/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. _how_it_works: - -**How it all Works** -==================== - -.. toctree:: - :maxdepth: 2 - - overview - license-detection-reference - -.. - [ToAdd] - file_collect_classify - extract - rules_engine - parser - package - report - integration - -.. - [ToDo] - Each Section to Be Elaborated diff --git a/docs/source/index.rst b/docs/source/index.rst index cffcec81f65..a23a774e0a8 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -96,16 +96,17 @@ Miscellaneous Documents misc/index -Discussion Documents +Reference Documents -------------------- -Discussion documents provide insights into how scancode works, and integration of the application -into your own software development life cycle. +Reference documents provide reference pages for technical reference information +about scancode-toolkit, how it works or supported features. .. toctree:: :maxdepth: 2 - explanations/index + reference/index + ---- diff --git a/docs/source/reference/available_package_parsers.rst b/docs/source/reference/available_package_parsers.rst new file mode 100644 index 00000000000..7baeb20a93e --- /dev/null +++ b/docs/source/reference/available_package_parsers.rst @@ -0,0 +1,769 @@ + + +.. _supported_packages: + +Supported package manifests and package datafiles +------------------------------------------------- + +Scancode supports a wide variety of package manifests, lockfiles +and other package datafiles containing package and dependency +information. + +This documentation page is generated automatically from available package +parsers in scancode-toolkit during documentation builds. + + +.. list-table:: Supported Package Parsers + :widths: 10 10 20 10 10 2 + :header-rows: 1 + + * - Package type + - Datasource ID + - Path Patterns + - Primary Language + - Documentation URL + - Description + * - None + - ``java_jar`` + - ``*.jar`` + - None + - https://en.wikipedia.org/wiki/JAR_(file_format) + - JAR Java Archive + * - ``about`` + - ``about_file`` + - ``*.ABOUT`` + - None + - https://aboutcode-toolkit.readthedocs.io/en/latest/specification.html + - AboutCode ABOUT file + * - ``alpine`` + - ``alpine_apk_archive`` + - ``*.apk`` + - None + - https://wiki.alpinelinux.org/wiki/Alpine_package_format + - Alpine Linux .apk package archive + * - ``alpine`` + - ``alpine_apkbuild`` + - ``*APKBUILD`` + - None + - https://wiki.alpinelinux.org/wiki/APKBUILD_Reference + - Alpine Linux APKBUILD package script + * - ``alpine`` + - ``alpine_installed_db`` + - ``*lib/apk/db/installed`` + - None + - None + - Alpine Linux installed package database + * - ``android`` + - ``android_apk`` + - ``*.apk`` + - Java + - https://en.wikipedia.org/wiki/Apk_(file_format) + - Android application package + * - ``android_lib`` + - ``android_aar_library`` + - ``*.aar`` + - Java + - https://developer.android.com/studio/projects/android-library + - Android library archive + * - ``autotools`` + - ``autotools_configure`` + - ``*/configure`` + ``*/configure.ac`` + - None + - https://www.gnu.org/software/automake/ + - Autotools configure script + * - ``axis2`` + - ``axis2_mar`` + - ``*.mar`` + - Java + - https://axis.apache.org/axis2/java/core/docs/modules.html + - Apache Axis2 module archive + * - ``axis2`` + - ``axis2_module_xml`` + - ``*/meta-inf/module.xml`` + - Java + - https://axis.apache.org/axis2/java/core/docs/modules.html + - Apache Axis2 module.xml + * - ``bazel`` + - ``bazel_build`` + - ``*/BUILD`` + - None + - https://bazel.build/ + - Bazel BUILD + * - ``bower`` + - ``bower_json`` + - ``*/bower.json`` + ``*/.bower.json`` + - JavaScript + - https://bower.io + - Bower package + * - ``buck`` + - ``buck_file`` + - ``*/BUCK`` + - None + - https://buck.build/ + - Buck file + * - ``buck`` + - ``buck_metadata`` + - ``*/METADATA.bzl`` + - None + - https://buck.build/ + - Buck metadata file + * - ``cab`` + - ``microsoft_cabinet`` + - ``*.cab`` + - C + - https://docs.microsoft.com/en-us/windows/win32/msi/cabinet-files + - Microsoft cabinet archive + * - ``cargo`` + - ``cargo_lock`` + - ``*/Cargo.lock`` + ``*/cargo.lock`` + - Rust + - https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html + - Rust Cargo.lock dependencies lockfile + * - ``cargo`` + - ``cargo_toml`` + - ``*/Cargo.toml`` + ``*/cargo.toml`` + - Rust + - https://doc.rust-lang.org/cargo/reference/manifest.html + - Rust Cargo.toml package manifest + * - ``chef`` + - ``chef_cookbook_metadata_json`` + - ``*/metadata.json`` + - Ruby + - https://docs.chef.io/config_rb_metadata/ + - Chef cookbook metadata.json + * - ``chef`` + - ``chef_cookbook_metadata_rb`` + - ``*/metadata.rb`` + - Ruby + - https://docs.chef.io/config_rb_metadata/ + - Chef cookbook metadata.rb + * - ``chrome`` + - ``chrome_crx`` + - ``*.crx`` + - JavaScript + - https://chrome.google.com/extensions + - Chrome extension + * - ``cocoapods`` + - ``cocoapods_podfile`` + - ``*Podfile`` + - Objective-C + - https://guides.cocoapods.org/using/the-podfile.html + - Cocoapods Podfile + * - ``cocoapods`` + - ``cocoapods_podfile_lock`` + - ``*Podfile.lock`` + - Objective-C + - https://guides.cocoapods.org/using/the-podfile.html + - Cocoapods Podfile.lock + * - ``cocoapods`` + - ``cocoapods_podspec`` + - ``*.podspec`` + - Objective-C + - https://guides.cocoapods.org/syntax/podspec.html + - Cocoapods .podspec + * - ``cocoapods`` + - ``cocoapods_podspec_json`` + - ``*.podspec.json`` + - Objective-C + - https://guides.cocoapods.org/syntax/podspec.html + - Cocoapods .podspec.json + * - ``composer`` + - ``php_composer_json`` + - ``*composer.json`` + - PHP + - https://getcomposer.org/doc/04-schema.md + - PHP composer manifest + * - ``composer`` + - ``php_composer_lock`` + - ``*composer.lock`` + - PHP + - https://getcomposer.org/doc/01-basic-usage.md#commit-your-composer-lock-file-to-version-control + - PHP composer lockfile + * - ``conda`` + - ``conda_meta_yaml`` + - ``*/meta.yaml`` + - None + - https://docs.conda.io/ + - Conda meta.yml manifest + * - ``cpan`` + - ``cpan_dist_ini`` + - ``*/dist.ini`` + - Perl + - https://metacpan.org/pod/Dist::Zilla::Tutorial + - CPAN Perl dist.ini + * - ``cpan`` + - ``cpan_makefile`` + - ``*/Makefile.PL`` + - Perl + - https://www.perlmonks.org/?node_id=128077 + - CPAN Perl Makefile.PL + * - ``cpan`` + - ``cpan_manifest`` + - ``*/MANIFEST`` + - Perl + - https://metacpan.org/pod/Module::Manifest + - CPAN Perl module MANIFEST + * - ``cpan`` + - ``cpan_meta_json`` + - ``*/META.json`` + - Perl + - https://metacpan.org/pod/Parse::CPAN::Meta + - CPAN Perl META.json + * - ``cpan`` + - ``cpan_meta_yml`` + - ``*/META.yml`` + - Perl + - https://metacpan.org/pod/CPAN::Meta::YAML + - CPAN Perl META.yml + * - ``cran`` + - ``cran_description`` + - ``*/DESCRIPTION`` + - R + - https://r-pkgs.org/description.html + - CRAN package DESCRIPTION + * - ``deb`` + - ``debian_control_extracted_deb`` + - ``*/control.tar.gz-extract/control`` + - None + - https://www.debian.org/doc/debian-policy/ch-controlfields.html + - Debian control file - extracted layout + * - ``deb`` + - ``debian_control_in_source`` + - ``*/debian/control`` + - None + - https://www.debian.org/doc/debian-policy/ch-controlfields.html + - Debian control file - source layout + * - ``deb`` + - ``debian_copyright_in_package`` + - ``*usr/share/doc/*/copyright`` + - None + - https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ + - Debian machine readable file in source + * - ``deb`` + - ``debian_copyright_in_source`` + - ``*/debian/copyright`` + - None + - https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ + - Debian machine readable file in source + * - ``deb`` + - ``debian_deb`` + - ``*.deb`` + - None + - https://manpages.debian.org/unstable/dpkg-dev/deb.5.en.html + - Debian binary package archive + * - ``deb`` + - ``debian_distroless_installed_db`` + - ``*var/lib/dpkg/status.d/*`` + - None + - https://www.debian.org/doc/debian-policy/ch-controlfields.html + - Debian distroless installed database + * - ``deb`` + - ``debian_installed_files_list`` + - ``*var/lib/dpkg/info/*.list`` + - None + - None + - Debian installed file paths list + * - ``deb`` + - ``debian_installed_md5sums`` + - ``*var/lib/dpkg/info/*.md5sums`` + - None + - https://www.debian.org/doc/manuals/debian-handbook/sect.package-meta-information.en.html#sect.configuration-scripts + - Debian installed file MD5 and paths list + * - ``deb`` + - ``debian_installed_status_db`` + - ``*var/lib/dpkg/status`` + - None + - https://www.debian.org/doc/debian-policy/ch-controlfields.html + - Debian installed packages database + * - ``deb`` + - ``debian_md5sums_in_extracted_deb`` + - ``*/control.tar.gz-extract/md5sums`` + ``*/control.tar.xz-extract/md5sums`` + - None + - https://www.debian.org/doc/manuals/debian-handbook/sect.package-meta-information.en.html#sect.configuration-scripts + - Debian file MD5 and paths list in .deb archive + * - ``deb`` + - ``debian_original_source_tarball`` + - ``*.orig.tar.xz`` + ``*.orig.tar.gz`` + - None + - https://manpages.debian.org/unstable/dpkg-dev/deb.5.en.html + - Debian package original source archive + * - ``deb`` + - ``debian_source_control_dsc`` + - ``*.dsc`` + - None + - https://wiki.debian.org/dsc + - Debian source control file + * - ``deb`` + - ``debian_source_metadata_tarball`` + - ``*.debian.tar.xz`` + ``*.debian.tar.gz`` + - None + - https://manpages.debian.org/unstable/dpkg-dev/deb.5.en.html + - Debian source package metadata archive + * - ``dmg`` + - ``apple_dmg`` + - ``*.dmg`` + ``*.sparseimage`` + - None + - None + - None + * - ``ear`` + - ``java_ear_application_xml`` + - ``*/META-INF/application.xml`` + - Java + - https://en.wikipedia.org/wiki/EAR_(file_format) + - Java EAR application.xml + * - ``ear`` + - ``java_ear_archive`` + - ``*.ear`` + - Java + - https://en.wikipedia.org/wiki/EAR_(file_format) + - Java EAR Enterprise application archive + * - ``freebsd`` + - ``freebsd_compact_manifest`` + - ``*/+COMPACT_MANIFEST`` + - None + - https://www.freebsd.org/cgi/man.cgi?pkg-create(8)#MANIFEST_FILE_DETAILS + - FreeBSD compact package manifest + * - ``gem`` + - ``gem_archive`` + - ``*.gem`` + - Ruby + - https://web.archive.org/web/20220326093616/https://piotrmurach.com/articles/looking-inside-a-ruby-gem/ + - RubyGems gem package archive + * - ``gem`` + - ``gem_archive_extracted`` + - ``*/metadata.gz-extract`` + - Ruby + - https://web.archive.org/web/20220326093616/https://piotrmurach.com/articles/looking-inside-a-ruby-gem/ + - RubyGems gem package extracted archive + * - ``gem`` + - ``gem_gemspec_installed_specifications`` + - ``*/specifications/*.gemspec`` + - Ruby + - https://guides.rubygems.org/specification-reference/ + - RubyGems gemspec manifest - installed vendor/bundle/specifications layout + * - ``gem`` + - ``gemfile`` + - ``*/Gemfile`` + ``*/*.gemfile`` + ``*/Gemfile-*`` + - Ruby + - https://bundler.io/man/gemfile.5.html + - RubyGems Bundler Gemfile + * - ``gem`` + - ``gemfile_extracted`` + - ``*/data.gz-extract/Gemfile`` + - Ruby + - https://bundler.io/man/gemfile.5.html + - RubyGems Bundler Gemfile - extracted layout + * - ``gem`` + - ``gemfile_lock`` + - ``*/Gemfile.lock`` + - Ruby + - https://bundler.io/man/gemfile.5.html + - RubyGems Bundler Gemfile.lock + * - ``gem`` + - ``gemfile_lock_extracted`` + - ``*/data.gz-extract/Gemfile.lock`` + - Ruby + - https://bundler.io/man/gemfile.5.html + - RubyGems Bundler Gemfile.lock - extracted layout + * - ``gem`` + - ``gemspec`` + - ``*.gemspec`` + - Ruby + - https://guides.rubygems.org/specification-reference/ + - RubyGems gemspec manifest + * - ``gem`` + - ``gemspec_extracted`` + - ``*/data.gz-extract/*.gemspec`` + - Ruby + - https://guides.rubygems.org/specification-reference/ + - RubyGems gemspec manifest - extracted data layout + * - ``golang`` + - ``go_mod`` + - ``*/go.mod`` + - Go + - https://go.dev/ref/mod + - Go modules file + * - ``golang`` + - ``go_sum`` + - ``*/go.sum`` + - Go + - https://go.dev/ref/mod#go-sum-files + - Go module cheksums file + * - ``golang`` + - ``godeps`` + - ``*/Godeps.json`` + - Go + - https://github.com/tools/godep + - Go Godeps + * - ``haxe`` + - ``haxelib_json`` + - ``*/haxelib.json`` + - Haxe + - https://lib.haxe.org/documentation/creating-a-haxelib-package/ + - Haxe haxelib.json metadata file + * - ``installshield`` + - ``installshield_installer`` + - ``*.exe`` + - None + - https://www.revenera.com/install/products/installshield + - InstallShield installer + * - ``ios`` + - ``ios_ipa`` + - ``*.ipa`` + - Objective-C + - https://en.wikipedia.org/wiki/.ipa + - iOS package archive + * - ``iso`` + - ``iso_disk_image`` + - ``*.iso`` + ``*.udf`` + ``*.img`` + - None + - https://en.wikipedia.org/wiki/ISO_9660 + - ISO disk image + * - ``ivy`` + - ``ant_ivy_xml`` + - ``*/ivy.xml`` + - Java + - https://ant.apache.org/ivy/history/latest-milestone/ivyfile.html + - Ant IVY dependency file + * - ``jar`` + - ``java_jar_manifest`` + - ``*/META-INF/MANIFEST.MF`` + - Java + - https://docs.oracle.com/javase/tutorial/deployment/jar/manifestindex.html + - Java JAR MANIFEST.MF + * - ``jboss-service`` + - ``jboss_sar`` + - ``*.sar`` + - Java + - https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/ch02s01.html + - JBOSS service archive + * - ``jboss-service`` + - ``jboss_service_xml`` + - ``*/meta-inf/jboss-service.xml`` + - Java + - https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/ch02s01.html + - JBOSS service.xml + * - ``linux-distro`` + - ``etc_os_release`` + - ``*etc/os-release`` + ``*usr/lib/os-release`` + - None + - https://www.freedesktop.org/software/systemd/man/os-release.html + - Linux OS release metadata file + * - ``maven`` + - ``build_gradle`` + - ``*/build.gradle`` + ``*/build.gradle.kts`` + - None + - None + - Gradle build script + * - ``maven`` + - ``maven_pom`` + - ``*.pom`` + ``*pom.xml`` + - Java + - https://maven.apache.org/pom.html + - Apache Maven pom + * - ``maven`` + - ``maven_pom_properties`` + - ``*/pom.properties`` + - Java + - https://maven.apache.org/pom.html + - Apache Maven pom properties file + * - ``meteor`` + - ``meteor_package`` + - ``*/package.js`` + - JavaScript + - https://docs.meteor.com/api/packagejs.html + - Meteor package.js + * - ``mozilla`` + - ``mozilla_xpi`` + - ``*.xpi`` + - JavaScript + - https://en.wikipedia.org/wiki/XPInstall + - Mozilla XPI extension + * - ``msi`` + - ``msi_installer`` + - ``*.msi`` + - None + - https://docs.microsoft.com/en-us/windows/win32/msi/windows-installer-portal + - Microsoft MSI installer + * - ``npm`` + - ``npm_package_json`` + - ``*/package.json`` + - JavaScript + - https://docs.npmjs.com/cli/v8/configuring-npm/package-json + - npm package.json + * - ``npm`` + - ``npm_package_lock_json`` + - ``*/package-lock.json`` + ``*/.package-lock.json`` + - JavaScript + - https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json + - npm package-lock.json lockfile + * - ``npm`` + - ``npm_shrinkwrap_json`` + - ``*/npm-shrinkwrap.json`` + - JavaScript + - https://docs.npmjs.com/cli/v8/configuring-npm/npm-shrinkwrap-json + - npm shrinkwrap.json lockfile + * - ``npm`` + - ``yarn_lock_v1`` + - ``*/yarn.lock`` + - JavaScript + - https://classic.yarnpkg.com/lang/en/docs/yarn-lock/ + - yarn.lock lockfile v1 format + * - ``npm`` + - ``yarn_lock_v2`` + - ``*/yarn.lock`` + - JavaScript + - https://classic.yarnpkg.com/lang/en/docs/yarn-lock/ + - yarn.lock lockfile v2 format + * - ``nsis`` + - ``nsis_installer`` + - ``*.exe`` + - None + - https://nsis.sourceforge.io/Main_Page + - NSIS installer + * - ``nuget`` + - ``nuget_nupkg`` + - ``*.nupkg`` + - None + - https://en.wikipedia.org/wiki/Open_Packaging_Conventions + - NuGet nupkg package archive + * - ``nuget`` + - ``nuget_nupsec`` + - ``*.nuspec`` + - None + - https://docs.microsoft.com/en-us/nuget/reference/nuspec + - NuGet nuspec package manifest + * - ``opam`` + - ``opam_file`` + - ``*opam`` + - Ocaml + - https://opam.ocaml.org/doc/Manual.html#Common-file-format + - Ocaml Opam file + * - ``pubspec`` + - ``pubspec_lock`` + - ``*pubspec.lock`` + - dart + - https://web.archive.org/web/20220330081004/https://gpalma.pt/blog/what-is-the-pubspec-lock/ + - Dart pubspec lockfile + * - ``pubspec`` + - ``pubspec_yaml`` + - ``*pubspec.yaml`` + - dart + - https://dart.dev/tools/pub/pubspec + - Dart pubspec manifest + * - ``pypi`` + - ``conda_yaml`` + - ``*conda.yaml`` + ``*conda.yml`` + - Python + - https://docs.conda.io/ + - Conda yaml manifest + * - ``pypi`` + - ``pip_requirements`` + - ``*requirement*.txt`` + ``*requirement*.pip`` + ``*requirement*.in`` + ``*requires.txt`` + ``*requirements/*.txt`` + ``*requirements/*.pip`` + ``*requirements/*.in`` + ``*reqs.txt`` + - Python + - https://pip.pypa.io/en/latest/reference/requirements-file-format/ + - pip requirements file + * - ``pypi`` + - ``pipfile`` + - ``*Pipfile`` + - Python + - https://github.com/pypa/pipfile + - Pipfile + * - ``pypi`` + - ``pipfile_lock`` + - ``*Pipfile.lock`` + - Python + - https://github.com/pypa/pipfile + - Pipfile.lock + * - ``pypi`` + - ``pypi_editable_egg_pkginfo`` + - ``*.egg-info/PKG-INFO`` + - Python + - https://peps.python.org/pep-0376/ + - PyPI editable local installation PKG-INFO + * - ``pypi`` + - ``pypi_egg`` + - ``*.egg`` + - Python + - https://web.archive.org/web/20210604075235/http://peak.telecommunity.com/DevCenter/PythonEggs + - PyPI egg + * - ``pypi`` + - ``pypi_egg_pkginfo`` + - ``*/EGG-INFO/PKG-INFO`` + - Python + - https://peps.python.org/pep-0376/ + - PyPI extracted egg PKG-INFO + * - ``pypi`` + - ``pypi_pyproject_toml`` + - ``*pyproject.toml`` + - Python + - https://peps.python.org/pep-0621/ + - Python pyproject.toml + * - ``pypi`` + - ``pypi_sdist_pkginfo`` + - ``*/PKG-INFO`` + - Python + - https://peps.python.org/pep-0314/ + - PyPI extracted sdist PKG-INFO + * - ``pypi`` + - ``pypi_setup_cfg`` + - ``*setup.cfg`` + - Python + - https://peps.python.org/pep-0390/ + - Python setup.cfg + * - ``pypi`` + - ``pypi_setup_py`` + - ``*setup.py`` + - Python + - https://docs.python.org/3/distutils/setupscript.html + - Python setup.py + * - ``pypi`` + - ``pypi_wheel`` + - ``*.whl`` + - Python + - https://peps.python.org/pep-0427/ + - PyPI wheel + * - ``pypi`` + - ``pypi_wheel_metadata`` + - ``*.dist-info/METADATA`` + - Python + - https://packaging.python.org/en/latest/specifications/core-metadata/ + - PyPI installed wheel METADATA + * - ``readme`` + - ``readme`` + - ``*/README.android`` + ``*/README.chromium`` + ``*/README.facebook`` + ``*/README.google`` + ``*/README.thirdparty`` + - None + - None + - None + * - ``rpm`` + - ``rpm_archive`` + - ``*.rpm`` + ``*.src.rpm`` + ``*.srpm`` + ``*.mvl`` + ``*.vip`` + - None + - https://en.wikipedia.org/wiki/RPM_Package_Manager + - RPM package archive + * - ``rpm`` + - ``rpm_installed_database_bdb`` + - ``*var/lib/rpm/Packages`` + - None + - https://man7.org/linux/man-pages/man8/rpmdb.8.html + - RPM installed package BDB database + * - ``rpm`` + - ``rpm_installed_database_ndb`` + - ``*usr/lib/sysimage/rpm/Packages.db`` + - None + - https://fedoraproject.org/wiki/Changes/NewRpmDBFormat + - RPM installed package NDB database + * - ``rpm`` + - ``rpm_installed_database_sqlite`` + - ``*var/lib/rpm/rpmdb.sqlite`` + - None + - https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb + - RPM installed package SQLite database + * - ``rpm`` + - ``rpm_spefile`` + - ``*.spec`` + - None + - https://en.wikipedia.org/wiki/RPM_Package_Manager + - RPM specfile + * - ``shar`` + - ``shar_shell_archive`` + - ``*.shar`` + - None + - https://en.wikipedia.org/wiki/Shar + - shell archive + * - ``squashfs`` + - ``squashfs_disk_image`` + - None + - None + - https://en.wikipedia.org/wiki/SquashFS + - Squashfs disk image + * - ``war`` + - ``java_war_archive`` + - ``*.war`` + - Java + - https://en.wikipedia.org/wiki/WAR_(file_format) + - Java Web Application Archive + * - ``war`` + - ``java_war_web_xml`` + - ``*/WEB-INF/web.xml`` + - Java + - https://en.wikipedia.org/wiki/WAR_(file_format) + - Java WAR web/xml + * - ``windows-program`` + - ``win_reg_installed_programs_docker_file_software`` + - ``*/Files/Windows/System32/config/SOFTWARE`` + - None + - https://en.wikipedia.org/wiki/Windows_Registry + - Windows Registry Installed Program - Docker SOFTWARE + * - ``windows-program`` + - ``win_reg_installed_programs_docker_software_delta`` + - ``*/Hives/Software_Delta`` + - None + - https://en.wikipedia.org/wiki/Windows_Registry + - Windows Registry Installed Program - Docker Software Delta + * - ``windows-program`` + - ``win_reg_installed_programs_docker_utility_software`` + - ``*/UtilityVM/Files/Windows/System32/config/SOFTWARE`` + - None + - https://en.wikipedia.org/wiki/Windows_Registry + - Windows Registry Installed Program - Docker UtilityVM SOFTWARE + * - ``windows-update`` + - ``microsoft_update_manifest_mum`` + - ``*.mum`` + - None + - None + - Microsoft Update Manifest .mum file + * - ``winexe`` + - ``windows_executable`` + - ``*.exe`` + ``*.dll`` + ``*.mui`` + ``*.mun`` + ``*.com`` + ``*.winmd`` + ``*.sys`` + ``*.tlb`` + ``*.exe_*`` + ``*.dll_*`` + ``*.mui_*`` + ``*.mun_*`` + ``*.com_*`` + ``*.winmd_*`` + ``*.sys_*`` + ``*.tlb_*`` + ``*.ocx`` + - None + - https://en.wikipedia.org/wiki/Portable_Executable + - Windows Portable Executable metadata diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst new file mode 100644 index 00000000000..e3792ea74d6 --- /dev/null +++ b/docs/source/reference/index.rst @@ -0,0 +1,11 @@ +.. _how_it_works: + +**Reference Docs** +================== + +.. toctree:: + :maxdepth: 2 + + overview + license-detection-reference + available_package_parsers diff --git a/docs/source/explanations/license-detection-reference.rst b/docs/source/reference/license-detection-reference.rst similarity index 100% rename from docs/source/explanations/license-detection-reference.rst rename to docs/source/reference/license-detection-reference.rst diff --git a/docs/source/explanations/overview.rst b/docs/source/reference/overview.rst similarity index 100% rename from docs/source/explanations/overview.rst rename to docs/source/reference/overview.rst diff --git a/setup.cfg b/setup.cfg index ff738af4568..04db84bf5c5 100644 --- a/setup.cfg +++ b/setup.cfg @@ -153,6 +153,7 @@ console_scripts = scancode = scancode.cli:scancode scancode-reindex-licenses = licensedcode.reindex:reindex_licenses scancode-license-data = licensedcode.license_db:dump_scancode_license_data + regen-package-docs = packagedcode.regen_package_docs:regen_package_docs # These are configurations for ScanCode plugins as setuptools entry points. # Each plugin entry hast this form: diff --git a/src/packagedcode/plugin_package.py b/src/packagedcode/plugin_package.py index 88bc868111f..3115467bbd2 100644 --- a/src/packagedcode/plugin_package.py +++ b/src/packagedcode/plugin_package.py @@ -63,23 +63,52 @@ def print_packages(ctx, param, value): if not value or ctx.resilient_parsing: return + for package_data in get_available_package_parsers(): + click.echo('--------------------------------------------') + click.echo(f'Package type: {package_data["package_type"]}') + click.echo(f' datasource_id: {package_data["datasource_id"]}') + click.echo(f' documentation URL: {package_data["documentation_url"]}') + click.echo(f' primary language: {package_data["default_primary_language"]}') + click.echo(f' description: {package_data["description"]}') + click.echo(f' path_patterns: {package_data["path_patterns"]}') + ctx.exit() + + +def get_available_package_parsers(docs=False): + from packagedcode import ALL_DATAFILE_HANDLERS + all_data_packages = [] + for cls in sorted( ALL_DATAFILE_HANDLERS, key=lambda pc: (pc.default_package_type or '', pc.datasource_id), ): - pp = ', '.join(repr(p) for p in cls.path_patterns) - click.echo('--------------------------------------------') - click.echo(f'Package type: {cls.default_package_type}') if cls.datasource_id is None: raise Exception(cls) - click.echo(f' datasource_id: {cls.datasource_id}') - click.echo(f' documentation URL: {cls.documentation_url}') - click.echo(f' primary language: {cls.default_primary_language}') - click.echo(f' description: {cls.description}') - click.echo(f' path_patterns: {pp}') - ctx.exit() + + data_packages = {} + if docs: + path_patterns = '\n '.join(f"``{p}``" for p in cls.path_patterns) + if cls.default_package_type: + data_packages['package_type'] = f"``{cls.default_package_type}``" + else: + data_packages['package_type'] = cls.default_package_type + data_packages['datasource_id'] = f"``{cls.datasource_id}``" + else: + path_patterns = ', '.join(repr(p) for p in cls.path_patterns) + data_packages['package_type'] = cls.default_package_type + data_packages['datasource_id'] = cls.datasource_id + + data_packages['documentation_url'] = cls.documentation_url + data_packages['default_primary_language'] = cls.default_primary_language + data_packages['description'] = cls.description + data_packages['path_patterns'] = path_patterns + + all_data_packages.append(data_packages) + + return all_data_packages + @scan_impl diff --git a/src/packagedcode/regen_package_docs.py b/src/packagedcode/regen_package_docs.py new file mode 100644 index 00000000000..e6b0110c94e --- /dev/null +++ b/src/packagedcode/regen_package_docs.py @@ -0,0 +1,78 @@ +# +# Copyright (c) nexB Inc. and others. All rights reserved. +# ScanCode is a trademark of nexB Inc. +# SPDX-License-Identifier: Apache-2.0 +# See http://www.apache.org/licenses/LICENSE-2.0 for the license text. +# See https://github.com/nexB/scancode-toolkit for support or download. +# See https://aboutcode.org for more information about nexB OSS projects. +# + +import os +import pathlib +from os.path import dirname + +import click + +from commoncode.cliutils import MISC_GROUP +from commoncode.cliutils import PluggableCommandLineOption +from packagedcode.plugin_package import get_available_package_parsers +from jinja2 import Environment, FileSystemLoader + + +TEMPLATES_DIR = os.path.join(dirname(__file__), 'templates') + +def write_file(file_path, content): + file_path.open("w").write(content) + + +def regenerate( + doc_location, + template_dir=TEMPLATES_DIR, +): + """ + Generate a licenseDB static website and dump license data at + ``build_location`` given a license directory ``licenses_data_dir`` using + templates from ``template_dir``. ``test`` is to generate a stable output for + testing only + """ + environment = Environment( + loader=FileSystemLoader(template_dir), + autoescape=True, + ) + doc_path = pathlib.Path(doc_location) + + all_available_packages = get_available_package_parsers(docs=True) + + license_list_template = environment.get_template("available_package_parsers.rst") + packages_doc = license_list_template.render( + all_available_packages=all_available_packages, + ) + write_file(file_path=doc_path, content=packages_doc) + + + +@click.command(name='regen-package-docs') +@click.option( + '--path', + type=click.Path(exists=True, writable=True, file_okay=True, dir_okay=False, path_type=str), + metavar='FILE', + help='Regenerate a reStructuredText documentation page from scancode available package parsers data.', + help_group=MISC_GROUP, + cls=PluggableCommandLineOption, +) +@click.help_option('-h', '--help') +def regen_package_docs( + path, + *args, + **kwargs, +): + """ + Regenerate the scancode available packages documentation at `path`. + """ + click.secho(f'Regenerating package docs at: {path}', err=True) + regenerate(doc_location=path) + click.secho(f'Documentation regeneration done.', err=True) + + +if __name__ == '__main__': + regen_package_docs() diff --git a/src/packagedcode/templates/available_package_parsers.rst b/src/packagedcode/templates/available_package_parsers.rst new file mode 100644 index 00000000000..1e7cbe1aa44 --- /dev/null +++ b/src/packagedcode/templates/available_package_parsers.rst @@ -0,0 +1,33 @@ +{% block content %} + +.. _supported_packages: + +Supported package manifests and package datafiles +------------------------------------------------- + +Scancode supports a wide variety of package manifests, lockfiles +and other package datafiles containing package and dependency +information. + +This documentation page is generated automatically from available package +parsers in scancode-toolkit during documentation builds. + + +.. list-table:: Supported Package Parsers + :widths: 10 10 20 10 10 2 + :header-rows: 1 + + * - Package type + - Datasource ID + - Path Patterns + - Primary Language + - Documentation URL + - Description + {% for package_data in all_available_packages -%} + * - {{ package_data.package_type }} + - {{ package_data.datasource_id }} + - {% if package_data.path_patterns %}{{ package_data.path_patterns }}{% else %}{{"None"}}{% endif %} + - {{ package_data.default_primary_language }} + - {% if package_data.documentation_url %}{{ package_data.documentation_url }}{% else %}{{"None"}}{% endif %} + - {% if package_data.description %}{{ package_data.description }}{% else %}{{"None"}}{% endif %} + {% endfor %}{% endblock %} \ No newline at end of file From bbd98baab4877c29777c93747d1e7c04b05da4be Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Tue, 28 Mar 2023 15:36:09 +0530 Subject: [PATCH 08/13] Misc doc updates - Add doc check reminder in PR template - Delete stray file - Misc license related updates to align docs Signed-off-by: Ayan Sinha Mahapatra --- .github/PULL_REQUEST_TEMPLATE.md | 3 +- .../scancode-non-spdx-licenses-2017-08-01.csv | 854 ------------------ docs/source/plugins/licence_policy_plugin.rst | 6 +- 3 files changed, 6 insertions(+), 857 deletions(-) delete mode 100644 docs/source/misc/data/scancode-non-spdx-licenses-2017-08-01.csv diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 27828992d11..5716601d8eb 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -17,7 +17,8 @@ Make sure these boxes are checked before your pull request (PR) is ready to be r * [ ] Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR Run [tests](https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_dev.html#running-tests) locally to check for errors. * [ ] Commits are in uniquely-named feature branch and has no merge conflicts 📁 - +* [ ] Looked for possible updates in documentation and added updates if applicable +* [ ] Updated CHANGELOG.rst