Don't detect license in other package manifest information (author/dependencies)

[chinyeungli]

The following 3 sample files have an "extra" unknown_license_reference detected which seems to be noise:
1/
subset of a package.json

  "author": "Selwyn <talk@selwyn.cc> (https://selwyn.cc/)",
  "license": "ISC",

subset of the license scan result:

      "license_detections": [
        {
          "license_expression": "unknown-license-reference AND isc",
          "license_expression_spdx": "LicenseRef-scancode-unknown-license-reference AND ISC",
          "matches": [
            {
              "license_expression": "unknown-license-reference",
              "license_expression_spdx": "LicenseRef-scancode-unknown-license-reference",
              "from_file": "code/css-declaration-sorter-7.2.0.tgz-extract/package/package.json",
              "start_line": 50,
              "end_line": 51,
              "matcher": "2-aho",
              "score": 60,
              "matched_length": 2,
              "match_coverage": 100,
              "rule_relevance": 60,
              "rule_identifier": "lead-in_unknown_43.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_43.RULE",
              "matched_text": "  \"author\": \"Selwyn \u003Ctalk@selwyn.cc\u003E (https://selwyn.cc/)\",\n  \"license\": \"ISC\",",
              "matched_text_diagnostics": "cc/)\",\n  \"license\": \""
            },
            {
              "license_expression": "isc",
              "license_expression_spdx": "ISC",
              "from_file": "code/css-declaration-sorter-7.2.0.tgz-extract/package/package.json",
              "start_line": 51,
              "end_line": 51,
              "matcher": "2-aho",
              "score": 100,
              "matched_length": 2,
              "match_coverage": 100,
              "rule_relevance": 100,
              "rule_identifier": "isc_38.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/isc_38.RULE",
              "matched_text": "  \"license\": \"ISC\",",
              "matched_text_diagnostics": "license\": \"ISC\","
            }
          ],
          "detection_log": [
            "unknown-match"
          ],
          "identifier": "unknown_license_reference_and_isc-83b73c8e-af2d-adc5-1659-64c84479d7ae"
        }
      ],

2/
subset of the package.json

  "author": "David Bonnet <david@bonnet.cc>",
  "license": "MIT",

subset of the license scan result

      "license_detections": [
        {
          "license_expression": "unknown-license-reference AND mit",
          "license_expression_spdx": "LicenseRef-scancode-unknown-license-reference AND MIT",
          "matches": [
            {
              "license_expression": "unknown-license-reference",
              "license_expression_spdx": "LicenseRef-scancode-unknown-license-reference",
              "from_file": "code/astring-1.8.6.tgz-extract/package/package.json",
              "start_line": 47,
              "end_line": 48,
              "matcher": "2-aho",
              "score": 60,
              "matched_length": 2,
              "match_coverage": 100,
              "rule_relevance": 60,
              "rule_identifier": "lead-in_unknown_43.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_43.RULE",
              "matched_text": "  \"author\": \"David Bonnet \u003Cdavid@bonnet.cc\u003E\",\n  \"license\": \"MIT\",",
              "matched_text_diagnostics": "cc\u003E\",\n  \"license\": \""
            },
            {
              "license_expression": "mit",
              "license_expression_spdx": "MIT",
              "from_file": "code/astring-1.8.6.tgz-extract/package/package.json",
              "start_line": 48,
              "end_line": 48,
              "matcher": "2-aho",
              "score": 100,
              "matched_length": 2,
              "match_coverage": 100,
              "rule_relevance": 100,
              "rule_identifier": "mit_30.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_30.RULE",
              "matched_text": "  \"license\": \"MIT\",",
              "matched_text_diagnostics": "license\": \"MIT\","
            }
          ],
          "detection_log": [
            "unknown-match"
          ],
          "identifier": "unknown_license_reference_and_mit-ec62877c-d215-c922-51a5-b6822ee740cc"
        }
      ],

3/
Not sure if this can be forgivien

          "matches": [
            {
              "license_expression": "unknown-license-reference",
              "license_expression_spdx": "LicenseRef-scancode-unknown-license-reference",
              "from_file": "code/ordered-map-4.4.2.tgz-extract/package/package.json",
              "start_line": 87,
              "end_line": 87,
              "matcher": "2-aho",
              "score": 80,
              "matched_length": 3,
              "match_coverage": 100,
              "rule_relevance": 80,
              "rule_identifier": "unknown-license-reference_334.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/unknown-license-reference_334.RULE",
              "matched_text": "    \"rollup-plugin-license\": \"^3.0.0\",",
              "matched_text_diagnostics": "license\": \"^3.0."
            }
          ],

In summary, in the first two cases, SCTK is misled by the author value, while the last case is influenced by the presence of the "license" wording

[armijnhemel]

My guess is that the .cc from the e-mail address is seen as a reference to creative commons, especially because it is followed by the word license. My guess is that with the proliferation of top level domains this will happen more often.

If I understand correctly package.json is basically treated as a text blob. When encountering a package.json file I think it can be assumed that there is some structure and that fields should not be concatenated.

[mjherzog]

For the 3rd example note that the PURL is: pkg:npm/@js-sdsl:ordered-map@4.4.2 and the matched text is from the devDependencies data which should be ignored for license scanning.

[AyanSinhaMahapatra]

If I understand correctly package.json is basically treated as a text blob. When encountering a package.json file I think it can be assumed that there is some structure and that fields should not be concatenated.

We do a package level license detection using the specific structure which is used for the package and another file level license detection to detect other license declarations which are usually present sometimes. See #3024 for more context. Note that the file level license detections are not used in the package license/summary.

The side effect is false-positives like this where we detect licenses falsely in author detections and dev-dependencies. We can handle these by checking for URL/author detections in those matched lines and remove them in post-processing. We could also add false-positives for specific cases as I've done in #4405 but this would be too much to do for all occurrences like this.

This is not detected for all .cc domains, but only when this is present right next to the license statement so our query heuristics think they are part of the same license declaration.

[AyanSinhaMahapatra]

added with #4405, closing