Update How-to-Guides with license updates

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Author: AyanSinhaMahapatra

docs/source/how-to-guides/add_new_license.rst

@@ -6,30 +6,75 @@ How To Add a New License for Detection
 How to add a new license for detection?
 ---------------------------------------
 
-To add new license, you first need to select a new and unique license key (mit
-and gpl-2.0 are some of the existing license keys). All licenses are stored as
-plain text files in the src/licensedcode/data/licenses directory using their key
-as base for the file name(s).
+To add new license, you first need to select a new and unique license `key` (mit
+and gpl-2.0 are some of the existing license keys).
 
-You need to create a pair of files:
+The key name can contain only these symbols:
 
-- a file with the text of the license saved in a plain text file named
-  key.LICENSE
+- lowercase letters from a to z,
+- numbers from 0 to 9,and
+- dash - and . period signs. No spaces or underscore.
 
-- a small text data file (in YAML format) named key.yml that contains license
-  information such as::
+The license key also has to be less than 50 characters (same for `short_name`).
 
-    key: my-license
-    name: My License
+We also have to add a `spdx_license_key` which is either a valid SPDX license key at
+` The SPDX license list <https://spdx.org/licenses/>`_, or a `Licenseref-scancode-<key>`.
 
-The key name can contain only these symbols:
+All licenses are stored as a plain text file in the `src/licensedcode/data/licenses`
+directory using their key as base for the file name. I.e. the filename for a license
+with `key: mit` would be `mit.LICENSE`.
 
-- lowercase letters from a to z,
-- numbers from 0 to 9,and
-- dash - and . period signs. No spaces.
+You need to create a file with:
+
+- the text of the license saved in plain text. We usually get rid of HTML tags or
+  other special characters, copyrights and only keep the original text as is,
+  with the original formatting intact.
+
+- the data attributes for the license in YAML format as YAML frontmatter.
+
+See an example license: `apache-2.0.LICENSE <https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE>`_
+
+There are a couple of mandatory attributes:
+
+- `key`
+- `spdx_license_key`
+- `short_name`
+- `name`
+- `category` (Use "Unstated License" if not known)
+- `owner` (Use "Unspecified" if not known)
+
+And more attributes which are not mandatory but nice to have always (if applicable):
+
+- `other_spdx_license_keys`
+- `osi_license_key`
+- `minimum_coverage`
+- `standard_notice`
+- `notes`
+
+We want to use `minimum_coverage` when there are other licenses which are very similar
+and we want to make sure we match correctly these licenses, and `notes` for interesting
+cases of licenses with descriptions to help identify origin, similarities to other licenses,
+notes about the SPDX keys and others. 
+
+Some URLs:
+
+- `homepage_url`
+- `text_urls`
+- `osi_url`
+- `faq_url`
+- `other_urls`
+
+Also attributes having ignorables in the license text:
+
+- `ignorable_urls`
+- `ignorable_copyrights`
+- `ignorable_authors`
+- `ignorable_holders`
+- `ignorable_emails`
 
-Save these two files in the ``src/licensedcode/data/licenses/`` directory.
+See the ``src/licensedcode/data/licenses/`` directory for many more examples.
 
-Done!
+.. note::
 
-See the ``src/licensedcode/data/licenses/`` directory for many examples.
+    Add licenses in a local developement installation and run `scancode-reindex-licenses`
+    to make sure we reindex the licenses and this validates the new licenses.

docs/source/how-to-guides/add_new_license_detection_rule.rst

@@ -4,65 +4,83 @@ How to Add New License Rules for Enhanced Detection
 ===================================================
 
 ScanCode relies on license rules to detect licenses. A rule is a simple text
-file containing a license text or notice or mention; And a small companion YAML
-text file that tells ScanCode which license expression to report when the text
-is detected.
+file containing a license text or notice or mention with YAML frontmatter with data
+attributes that tells ScanCode which license expression to report when the text
+is detected, and other properties.
 
-See the :ref:`faq` for a high level description of :ref:`add_new_license_det_rule`.
+See the :ref:`faq` for a high level description of adding license detection rules.
 
 How to add a new license detection rule?
 ----------------------------------------
 
-A license detection rule is a pair of files:
+A license detection rule is a file with:
 
-- a plain text rule file that is typically a variant of a license text, notice
-  or license mention.
+- a plain text that is typically a variant of a license text, notice or license
+  mention.
 
-- a small text data file (in YAML format) documenting which license expression
-  should be detected when the rule text is found in a codebase.
+- data as YAML frontmatter documenting license expression and other
+  rule attributes.
 
 To add a new rule, you need to pick a unique base file name. As a convention, we
 like to include the license expression that should be detected in that name to
 make it more descriptive. For example: mit_and_gpl-2.0 is a good base name for a
 rule that would detect an MIT and GPL-2.0 license combination at once. Add a
-suffix to make it unique if there is already a rule with this base name. Do not
-use spaces or special characters in that name.
+suffix (usually numeric) to make it unique if there is already a rule with
+this base name. Do not use spaces or special characters in that name.
 
-Then create the rule file in the src/licensedcode/data/rules/ directory using
-this name, replacing selected_base_name with the base name you selected::
+Then create the rule file in the `src/licensedcode/data/rules/` directory using
+this name, for example a rule with `license_expression` as `mit AND apache-2.0`
+might have a filename: `mit_and_apache-2.0_10.RULE`.
 
-    selected_base_name.RULE
+Save your rule text in this file, if there are specific words like company names,
+projects or other, it is better to have rules with and without these so we have
+better detection. 
 
-Save your rule text in this file.
+For a simple `mit AND apache-2.0` license expression detection, here is an example
+rule file::
 
-Then create the YAML data file in the src/licensedcode/data/rules/ directory
-using this name::
 
-    selected_base_name.yml
-
-For a simple mit and gpl-2.0 license expression detection, the content of
-this file can be this YAML snippet::
-
-    license_expression: mit AND gpl-2.0
+    ---
+    license_expression: mit AND apache-2.0
     is_license_notice: yes
+    relevance: 100
+    referenced_filenames:
+        - LICENSE
+    ---
 
-Save these two files in the ``src/licensedcode/data/licenses/`` directory and
-you are done!
+    ## License
+    The MIT License (MIT) + Apache 2.0. Read [LICENSE](LICENSE).
 
 See the ``src/licensedcode/data/rules/`` directory for many examples.
 
 More (advanced) rules options:
 
-- you can use a notes: text field to document this rule and explain where you
+- you can use a `notes` text field to document this rule and explain where you
   found it first.
 
 - if no license should be detected for your .RULE text, do not add a license expression,
   just add a ``notes`` field.
 
-- Each rules needs have one flag such as is_license_notice. See the
-  ``src/licensedcode/models.py`` directory for a list of all possible values and
-  other options.
+- Each rules needs have one flag to describe the type of license rule, the options are:
+
+  - `is_license_notice`
+  - `is_license_text`
+  - `is_license_tag`
+  - `is_license_reference`
+  - `is_license_intro`
+
+- There can also be false positive rules, which if detected in the file scanned, will not
+  be present in the result license detections. These just have the license text and a
+  `is_false_positive` flag set to True.
 
 - you can specify key phrases by surrounding one or more words between the `{{`
   and `}}` tags. Key phrases are words that **must** be matched/present in order
   for a RULE to be considered a match.
+
+See the ``src/licensedcode/models.py`` directory for a list of all possible values
+and other options.
+
+.. note::
+
+    Add rules in a local developement installation and run `scancode-reindex-licenses`
+    to make sure we reindex the rules and this validates the new licenses.

docs/source/how-to-guides/install_new_license_plugin.rst

@@ -28,10 +28,8 @@ This is the basic structure of the example plugin::
     │   └── licenses_to_install1/
     │       ├── licenses/
     │       │   ├── example-installed-1.LICENSE
-    │       │   └── example-installed-1.yaml
     |       ├── rules/
     │       │   ├── example-installed-1.RULE
-    │       │   └── example-installed-1.yaml
     │       └── __init__.py
     ├── apache-2.0.LICENSE
     ├── MANIFEST.in
@@ -104,17 +102,15 @@ an example of a plugin with tests. The tests are contained in the ``tests`` dire
     │   └── licenses_to_install1/
     │       ├── licenses/
     │       │   ├── example-installed-1.LICENSE
-    │       │   └── example-installed-1.yaml
     │       ├── rules/
     │       │   ├── example-installed-1.RULE
-    │       │   └── example-installed-1.yaml
     │       └── __init__.py/
     ├── tests/
     │    ├── data/
     │    │   ├── example-installed-1.txt
     │    │   └── example-installed-1.txt.yml
     │    └── test_detection_datadriven.py
-    ├── gpl-1.0.LICENSE
+    ├── apache-2.0.LICENSE
     ├── MANIFEST.in
     ├── setup.cfg
     └── setup.py
@@ -142,8 +138,9 @@ Then you can define a test class and call the ``build_tests`` method defined in
         TEST_DIR,
         clazz=TestLicenseDataDriven1, regen=scancode_config.REGEN_TEST_FIXTURES)
 
-The ``tests/data`` directory contains a pair of files for each license:
-a license text file and a YAML file specifying the expected license expressions from the test.
+The ``tests/data`` directory contains a file for each license:
+a license text file with a YAML frontmatter specifying the expected license expression 
+from the test.
 
 Finally, install the plugin and run the test:
 
@@ -163,10 +160,8 @@ This is the basic structure of the example license directory::
     additional_license_directory/
     ├── licenses/
     │   ├── example-installed-1.LICENSE
-    │   └── example-installed-1.yaml
     ├── rules/
     │   ├── example-installed-1.RULE
-    │   └── example-installed-1.yaml
 
 Adding the licenses to the index
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^