Add general support to detect "structured" and tagged license notices

[pombredanne]

We should support specialized license detection for some cases where a project provides a structured or semi-structured licensing information: these are becoming common.

✔️ First there are all the package manifest metadata. We collect these alright but only capture the asserted license when present and then also run a plain license detection on the manifest text. We should instead only run a detection on the structured data that was collected in most cases (and sometimes also run license detection some part the text for good measure: Maven pom often contain XML comments with a license notice in addition to or ass a replacement for a POM licenses tags.) -> done. See #1523 for follow up

• There are also some specialized manifest files, such as the Debian copyright files. Not all are "Dep5" or "machine readable", but even these that are not yet machine readable tend to have some structure to them.

• There is the special "legal-like" files such as files named LICENSE, COPYING and similar which are detectable with https://github.com/nexB/scancode-toolkit/blob/f1f6e2cc29516455ac1732787658ede2b927f8d0/src/licensedcode/legal.py ... but whose detection is not yet plugged in the scan pipeline. They likely should always be returned in full as matches even when the matching may not be exact or present.

• There are cases where a project or author uses specific conventions to structure and tag a license notice. For instance Mozilla and several adopters of the MPL licenses family use BEGIN/END LICENSE BLOCK markers such as

/***** BEGIN LICENSE BLOCK *****
 *
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
 *
 ***** END LICENSE BLOCK *****/

• Another example is Broadcom as in gpl-2.0-broadcom-linking not detected. #700 with an XML comment `<:label-BRCM:2011:DUAL/GPL:standard .....[license notice or text].... :>

• VMWare and related organizations often use long notices with clear separators for each license texts or notices. The same can be seen in Android and some other Google projects.

✔️ There is also the SPDX-License-Identifier: convention as presented here https://spdx.org/sites/cpstandard/files/pages/files/using_spdx_license_list_short_identifiers.pdf and some prelimnary version used by earlier versions of uboot. --> Done

• In the Linux Kernel there is the MODULE_LICENSE macro found in source code and corresponding tags found in compiled binaries.

For each of these, the common theme is that processing should be special:

1. identify either a file type or pattern in a file
2. based on this, parse and extract the subset of the license-related text, notice or mention.
3. either run the general license detection on this text subset or run a specialized processing including some simpler mappings and lookups (for instance on the SPDX license identifiers or the NPM package.json license using a license expression parsing and lookup in the SPDX license list.)
4. For the remainder of the text that is not structured, we could perform a simple set/inverted index approximate matching first before deciding to go in any deeper license detection as an optimization

This would have some challenges overall as this may tie package and license detection together in some cases... But in the end that is not a problem... rather an opportunity for more relevant and more accurate results.