Add detected, normalized license expression to package manifests

[pombredanne]

Today we only report the "declared_license". The field "license_expression" has been designed to contain the normalized, detected version of the "declared_license" and should be populated alright.

The process for this is to run the license detection on the "declared_license" strings and use the reported expression. Some minimum threshold and score is likely to be needed.

This is a long pending requirement and has also been reported by @tomeks666 #1090 (comment).

[kestewart]

If "SPDX-License-Identifier:" is detected in the file, recommendation these days is to consider that the concluded license for the file, as well as detected.

[pombredanne]

@kestewart you wrote:

If "SPDX-License-Identifier:" is detected in the file, recommendation these days is to consider that the concluded license for the file, as well as detected.

Yes! ScanCode has no notion of concluded licenses though, so SPDX-License-Identifier are reported as a detected license together with extra attribute such that you know that this was detected from an identifier (so you could decide to make it a conclusion alright using that flag). Note that valid SPDX-License-Identifier are reported cleanly as well as older UBoot early versions (that contain a list of ids rather than a proper license expression)

Here this ticket is specifically to detect and report licenses that exists in package manifests (such as a Maven POM, an npm package.json, etc) that are "structured" (in the sense that we have fields in the manifest file and some structured file format such as JSON, YAML, XML, etc) and the data in these fields are NOT prefixed with any "SPDX-License-Identifier:".
The approach for these is to apply multiple detection techniques in sequence, and with the sequence being specific to a package type. For instance npm should be using SPDX license expressions and Rubygems and Composer should be using SPDX license ids, so it make sense to attempt to parse the license licenses first as an expression, then fall back to list of ids on error, then fallback to plain, regular license detection otherwise. Pypi uses its own "trove" classifiers, so there is a mapping for these to corresponding license ids, etc.

[pombredanne]

So the current Package attribute that would hold this data is for now called license_expression which is a terribly bad name IMHO, especially when compared to the raw unprocessed declared_license which is not structured at all (e.g. it can be anything that was there originally).

I think a better name would be normalized_license and this would hold a license expression that will be the normalization of whatever is in declared_license.

[pombredanne]

I am also renaming declared_licensing to declared_license for consistency.
@DennisClark do these names make sense to you?