LicenseRef not always generated

[sschuberth]

As of #1307 I would have expected any license finding in a ScanCode result file to have the spdx_license_key field populated: If it is a license known to the SPDX standard that field should contain the SPDX ID, and if the license is not known to the SPDX standard that field should contain the value of the key field prefixed by LicenseRef-scancode-.

Is this understanding wrong?

Because running

scancode --license src\licensedcode\data\licenses\indiana-extreme.LICENSE --json-pp -

(on Windows with ScanCode 2.9.9.post2.a7b8c4fa0) results in

      "key": "indiana-extreme",
      "score": 100.0,
      "name": "Indiana University Extreme! Lab Software License Version 1.1.1",
      "short_name": "Indiana Extreme License 1.1.1",
      "category": "Permissive",
      "is_exception": false,
      "owner": "Indiana University",
      "homepage_url": "",
      "text_url": "http://www.bearcave.com/software/java/xml/xmlpull_license.html",
      "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:indiana-extreme",
      "spdx_license_key": "",
      "spdx_url": "",

i.e. the spdx_license_key field is empty. I wonder whether that is because the check at

https://github.com/nexB/scancode-toolkit/blob/a7b8c4fa001b95dd215cd6d95313f5c96d8dafdd/src/formattedcode/output_spdx.py#L248-L252

should be enhanced to also handle empty strings.

[pombredanne]

@sschuberth the spdx_license_key field is empty in the JSON output unless this is a known SPDX license so far. The "LicenseRef" #1307 change you mention in output_spdx.py is a contribution of @furuholm and is only when you output an SPDX document: it does not affect the JSON output and any other output.
We could extend it everywhere within reason. There are a few places to consider:
1.- the license YAML data files: not sure it makes sense to store it there
2.- the API that returns license matches in https://github.com/nexB/scancode-toolkit/blob/a7b8c4fa001b95dd215cd6d95313f5c96d8dafdd/src/scancode/api.py#L143
3.- and each and every output that report SPDX license ids.

We could change it in 2 with the smallest impact.

Note that this may break some summarization code and the code of downstream users who use the presence/absence of the SPDX key to know if this is a license known to SPDX or not,

[sschuberth]

only when you output an SPDX document: it does not affect the JSON output and any other output.

Thanks for the clarification. I'd argue it's potentially confusing to get different (SPDX) license findings depending on the output format you use.

users who use the presence/absence of the SPDX key to know if this is a license known to SPDX or not

That's indeed a problem.

I don't have a proposal for a solution right now, but just wanted to highlight / understand the issue. I'll give this some more thinking.

[pombredanne]

@sschuberth ping. Any update?

[sschuberth]

I'm still undecided. I'd be fine with keeping the current behavior to only use a LicenseRef-scancode- prefix in SPDX output, and for other output formats use ScanCode's license key without any prefix, as-is, although it's sub-optimal. On the other hand, if you're willing to slightly break with backwards compatibility with the 3.0 release, I'd prefer to align the license naming between output formats and also use a LicenseRef-scancode- prefix for the SPDX license key in non-SPDX output formats.

[sschuberth]

Note that this may break some summarization code and the code of downstream users who use the presence/absence of the SPDX key to know if this is a license known to SPDX or not

In fact, now that #1360 is merged we also use the presence / absence of spdx_license_key to distinguish between "core" SPDX licenses and non-SPDX licenses that we internally generate LicenseRefs for. But we can adjust out code as needed depending on in which direction this ticket goes.

[pombredanne]

LicenseRefs are now always included since the SPDX license key is now mandatory for all licenses. We use by default the scancode namespace but we can acomodate any namespace.