Make it easier to get the SPDX license expression for individual matches

[sschuberth]

Starting with output format version 3, there is detected_license_expression_spdx, but this seems to hold the combined expression for all matches in a file. However, in ORT we're interested in the SPDX expressions for individual matches that come with start / end line numbers.

Matches do have license_expression, but these use ScanCode-specific keys instead of SPDX IDs for licenses. Before output format version 3, we were able to work around this by replacing key values with spdx_license_key values in expressions, but in output format version 3 spdx_license_key is not available anymore.

TL;DR if matches would have license_expression_spdx in addition to license_expression we'd be happy 😉

[AyanSinhaMahapatra]

we're interested in the SPDX expressions for individual matches that come with start / end line numbers.

Curious why that's the case, since the detected_license_expression_spdx is not merely the accumulation of the underlying matches, but there is some processing involved to get rid of unknowns and simplify the license expression. For more info see https://scancode-toolkit.readthedocs.io/en/stable/reference/license-detection-reference.html#examples
Is this on UI/visualization side where you want to show which SPDX expressions come from which part of the file?

Before output format version 3, we were able to work around this by replacing key values with spdx_license_key values in expressions

This is still available, but at the top-level as reference with the CLI option --license-references See for more details: https://scancode-toolkit.readthedocs.io/en/stable/cli-reference/scan-options-post.html?highlight=license-refrences#license-references-option

[sschuberth]

the detected_license_expression_spdx is not merely the accumulation of the underlying matches, but there is some processing involved

Interesting. Is this documented somewhere?

Is this on UI/visualization side where you want to show which SPDX expressions come from which part of the file?

Yes. However, it's not "only" for visualization, but also for curation of false positives: In order to judge whether a given expression is correct, you need to know where it comes from. ORT's data model demands a file location for each expression finding.

This is still available, but at the top-level as reference with the CLI option --license-references

Thanks for that! However, not needing to do this replacement at all would be preferable, of course 😉

[pombredanne]

@sschuberth re:

...but also for curation of false positives:

Just a reminder that any false positive is a bug that will be fixed in ScanCode, if this is reported. How could we have a way to get these reported more easily by ORT users?

ORT's data model demands a file location for each expression finding.

Actually with the new detection there can be multiple file locations, as multiple matches from different files may be combined in a single detection. For instance "See license in file COPYING" in foo.java and an Apache 2.0 notice in COPYING would be combined together in a single detection that combines these two matches from two different files

However, not needing to do this replacement at all would be preferable, of course

This should be easy enough but be aware that you will bypass all the efforts and new code that we have added to remove false positives if you ignore the detections and use only the matches.

[sschuberth]

How could we have a way to get these reported more easily by ORT users?

Contribute a "report false positive" button e.g. to the ORT Workbench if the finding stems from ScanCode 😉

For instance "See license in file COPYING" in foo.java and an Apache 2.0 notice in COPYING would be combined together in a single detection that combines these two matches from two different files

Understood. In ORT we probably would currently model this by "resolving" the reference in foo.java by recording a finding that uses the lines of the "See license in file COPYING" string and uses "Apache-2.0" as the license for these.

you will bypass all the efforts and new code that we have added to remove false positives if you ignore the detections and use only the matches.

Uhm, so are you saying that all the files -> license_detections -> matches are as "bad" (I'm exaggerating for clarity; you know what I mean) as in older ScanCode versions, and all the "magic" that is supposed to reduce false-positives only goes into the files -> license_detections -> license_expression JSON field? If so, we have a severe problem making use of that in ORT, I believe.

[sschuberth]

If so, we have a severe problem making use of that in ORT, I believe.

At a minimum, what we'd need along with files -> license_detections -> license_expression is a "combined" start / end line and score for that global-per-file license_expression.